i saw OpenLineage was built into Airflow recently as a provider but the documentation seems really light (https://airflow.apache.org/docs/apache-airflow-providers-openlineage/stable/guides/user.html), is the documentation from openlineage the correct way I should proceed?
https://openlineage.io/docs/integrations/airflow/usage
general
Created by Julien Le Dem on Tuesday, October 20th, 2020OpenLineage spec discussion. -See also: <https://github.com/OpenLineage/OpenLineage#community> -2023 Ecosystem Survey: <http://bit.ly/ecosystem_survey|bit.ly/ecosystem_survey>
ldacey
đ1
Julien Le Dem
openlineage-airflow is the package maintained in the OpenLineage project and to be used for versions of Airflow before 2.7. You could use it with 2.7 as well but youâd be staying on the âoldâ integration.
apache-airflow-providers-openlineage is the new package, maintained in the Airflow project that can be used starting Airflow 2.7 and is the recommended package moving forward. It is compatible with the configuration of the old package described in that usage page. CC: @Maciej Obuchowski @Jakub DardziĆski It looks like this page needs improvement.
apache-airflow-providers-openlineage is the new package, maintained in the Airflow project that can be used starting Airflow 2.7 and is the recommended package moving forward. It is compatible with the configuration of the old package described in that usage page. CC: @Maciej Obuchowski @Jakub DardziĆski It looks like this page needs improvement.
Maciej Obuchowski
1
Yeah, I'll fix that
1ldacey
do I label certain raw data sources as a dataset, for example SFTP/FTP sites, 0365 emails, etc? I extract that data into a bucket for the client in a "folder" called "raw" which I know will be an OL Dataset. Would this GCS folder (after extracting the data with Airflow) be the first Dataset OL is aware of?
I then process that data into partitioned parquet datasets which would also be OL Datasets:
<gcs://client-bucket/source-system-lob/raw>I then process that data into partitioned parquet datasets which would also be OL Datasets:
<gcs://client-bucket/source-system-lob/staging><gcs://client-bucket/source-system-lob/analytics>Jakub DardziĆski
that really depends on the use case IMHO
if you consider a whole directory/folder as a dataset (meaning that each file inside folds into a larger whole) you should label dataset as directory
you might as well have directory with each file being something different - in this case it would be best to set each file separately as dataset
if you consider a whole directory/folder as a dataset (meaning that each file inside folds into a larger whole) you should label dataset as directory
you might as well have directory with each file being something different - in this case it would be best to set each file separately as dataset
Jakub DardziĆski
there was also
SymlinksDatasetFacet introduced to store alternative dataset names, might be useful: https://github.com/OpenLineage/OpenLineage/pull/936ldacey
cool, yeah in general each file is just a snapshot of data from a client (for example, daily dump). the parquet datasets are normally partitioned and might have small fragments and I definitely picture it as more of a table than individual files
đ1
Maciej Obuchowski
Agree with Jakub here - with object storage, people use different patterns, but usually some directory layer vs file is the valid abstraction level, especially if your pattern is adding files with new data inside
đ1
ldacey
I tested a dataset for each raw file versus the folder and the folder looks much cleaner (not sure if I can collapse individual datasets/files into a group?)
from 2022, this particular source had 6 raw schema changes (client controlled, no warning). what should I do to make that as obvious as possible if I track the dataset at a folder level?
from 2022, this particular source had 6 raw schema changes (client controlled, no warning). what should I do to make that as obvious as possible if I track the dataset at a folder level?
ldacey
I was thinking that I could name the dataset based on the schema_version (identified by the raw column names), so in this example I would have 6 OL datasets feeding into one "staging" dataset
ldacey
not sure what the best practice would be in this scenario though
ldacey
âą also saw the docs reference
URI = gs://{bucket name}{path} and I wondered if the path would include the filename, or if it was just the base path like I showed above
Mars Lan
Has anyone managed to get the OL Airflow integration to work on AWS MWAA? We've tried pretty much every trick but still ended up with the following error:
Broken plugin: [openlineage.airflow.plugin] No module named 'openlineage.airflow'; 'openlineage' is not a packageMaciej Obuchowski
Which version are you trying to use?
Maciej Obuchowski
Both OL and MWAA/Airflow đ
Maciej Obuchowski
suggests that something went wrong with import process, for example cycle in import path
'openlineage' is not a packagesuggests that something went wrong with import process, for example cycle in import path
Mars Lan
MWAA: 2.6.3
OL: 1.0.0
I can see from the log that OL has been successfully installed to the webserver:
This is the full stacktrace:
OL: 1.0.0
I can see from the log that OL has been successfully installed to the webserver:
Successfully installed openlineage-airflow-1.0.0 openlineage-integration-common-1.0.0 openlineage-python-1.0.0 openlineage-sql-1.0.0This is the full stacktrace:
Traceback (most recent call last):
-
-File "/usr/local/airflow/.local/lib/python3.10/site-packages/airflow/plugins_manager.py", line 229, in load_entrypoint_plugins
-plugin_class = entry_point.load()
-File "/usr/local/airflow/.local/lib/python3.10/site-packages/importlib_metadata/__init__.py", line 209, in load
-module = import_module(match.group('module'))
-File "/usr/lib/python3.10/importlib/__init__.py", line 126, in import_module
-return _bootstrap._gcd_import(name[level:], package, level)
-File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
-File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
-File "<frozen importlib._bootstrap>", line 992, in _find_and_load_unlocked
-File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
-File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
-File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
-File "<frozen importlib._bootstrap>", line 1001, in _find_and_load_unlocked
-ModuleNotFoundError: No module named 'openlineage.airflow'; 'openlineage' is not a packageJakub DardziĆski
Itâs taking long to update MWAA environment but I tested 2.6.3 version with the following
and
is there any step that might lead to some unexpected results?
requirements.txt:openlineage-airflowand
openlineage-airflow==1.0.0is there any step that might lead to some unexpected results?
Mars Lan
Yeah, it takes forever to update MWAA even for a simple change. If you open either the webserver log (in CloudWatch) or the AirFlow UI, you should see the above error message.
Jakub DardziĆski
The thing is that I donât see any error messages.
I wrote simple DAG to test too:
and it gives expected results as well
I wrote simple DAG to test too:
from __future__ import annotations
-
-from datetime import datetime
-
-from airflow.models import DAG
-
-try:
- from airflow.operators.empty import EmptyOperator
-except ModuleNotFoundError:
- from airflow.operators.dummy import DummyOperator as EmptyOperator # type: ignore
-
-from openlineage.airflow.adapter import OpenLineageAdapter
-from openlineage.client.client import OpenLineageClient
-
-from airflow.operators.python import PythonOperator
-
-DAG_ID = "example_ol"
-
-def callable():
- client = OpenLineageClient()
- adapter = OpenLineageAdapter()
- print(client, adapter)
-
-with DAG(
- dag_id=DAG_ID,
- start_date=datetime(2021, 1, 1),
- schedule="@once",
- catchup=False,
-) as dag:
- begin = EmptyOperator(task_id="begin")
-
- test = PythonOperator(task_id='print_client', python_callable=callable)and it gives expected results as well
Mars Lan
Oh how interesting. I did have a plugin that sets the endpoint & key via env var. Let me try to disable that to see if it fixes the issue. Will report back after 30 mins, or however long it takes to update MWAA đ
Jakub DardziĆski
ohh, I see
you probably followed this guide: https://aws.amazon.com/blogs/big-data/automate-data-lineage-on-amazon-mwaa-with-openlineage/?
you probably followed this guide: https://aws.amazon.com/blogs/big-data/automate-data-lineage-on-amazon-mwaa-with-openlineage/?
Mars Lan
Actually no. I'm not aware of this guide. I assume it's outdated already?
Jakub DardziĆski
tbh I donât know
Mars Lan
Actually while we're on that topic, what's the recommended way to pass the URL & API Key in MWAA?
Jakub DardziĆski
I think it's still a plugin that sets env vars
Mars Lan
Yeah based on the page you shared, secret manager + plugin seems like the way to go.
Mars Lan
Alas after disabling the plugin and restarting the cluster, I'm still getting the same error. Do you mind to share a screenshot of your cluster's settings so I can compare?
Maciej Obuchowski
Are you maybe importing some top level OpenLineage code anywhere? This error is most likely circular import
Mars Lan
Let me try removing all the dags to see if it helps.
Mars Lan
@Maciej Obuchowski you were correct! It was indeed the DAGs. The errors are gone after removing all the dags. Now just need to figure what caused the circular import since I didn't import OL directly in DAG.
Mars Lan
Could this be the issue?
How could I declare lineage manually if I can't import these classes?
from airflow.lineage.entities import File, TableHow could I declare lineage manually if I can't import these classes?
Maciej Obuchowski
@Mars Lan I'll look in more details next week, as I'm in transit now
Maciej Obuchowski
but if you could narrow down a problem to single dag that I or @Jakub DardziĆski could reproduce, ideally locally, it would help a lot
Mars Lan
Thanks. I think I understand how this works much better now. Found a few useful BQ example dags. Will give them a try and report back.
đ„2
Nitin
Hi All,
I want to capture, source and target table details as lineage information with openlineage for Amazon Redshift. Please let me know, if anyone has done it
I want to capture, source and target table details as lineage information with openlineage for Amazon Redshift. Please let me know, if anyone has done it
Jakub DardziĆski
are you using Airflow to connect to Redshift?
Nitin
Hi @Jakub DardziĆski,
Thank you for your reply.
No, we are not using Airflow.
We are using load/Unload cmd with Pyspark and also Pandas with JDBC connection
Thank you for your reply.
No, we are not using Airflow.
We are using load/Unload cmd with Pyspark and also Pandas with JDBC connection
Jakub DardziĆski
@PaweĆ LeszczyĆski might know answer if Spark<->OL integration works with Redshift. Eventually JDBC is supported with sqlparser
for Pandas I think there wasnât too much work done
for Pandas I think there wasnât too much work done
PaweĆ LeszczyĆski
@Nitin If you're using jdbc within Spark, the lineage should be obtained via sqlparser-rs library https://github.com/sqlparser-rs/sqlparser-rs. In case it's not, please try to provide some minimal SQL code (or pyspark) which leads to uncaught lineage.
Nitin
Hi @Jakub DardziĆski / @PaweĆ LeszczyĆski, thank you for taking out time to reply on my query. We need to capture only load and unload query lineage which we are running using Spark.
If you have any sample implementation for reference, it will be indeed helpful
If you have any sample implementation for reference, it will be indeed helpful
PaweĆ LeszczyĆski
I think we don't support load yet on our side: https://github.com/OpenLineage/OpenLineage/blob/main/integration/sql/impl/src/visitor.rs#L8
Nitin
Yeah! any way you can think of, we can accommodate it specially load and unload statement.
Also, we would like to capture, lineage information where our endpoints are Sagemaker and Redis
Also, we would like to capture, lineage information where our endpoints are Sagemaker and Redis
Nitin
@PaweĆ LeszczyĆski can we use this code base integration/common/openlineage/common/provider/redshift_data.py for redshift lineage capture
Jakub DardziĆski
it still expects input and output tables that are usually retrieved from sqlparser
Jakub DardziĆski
for Sagemaker there is an Airflow integration written, might be an example possibly
https://github.com/OpenLineage/OpenLineage/blob/main/integration/airflow/openlineage/airflow/extractors/sagemaker_extractors.py
https://github.com/OpenLineage/OpenLineage/blob/main/integration/airflow/openlineage/airflow/extractors/sagemaker_extractors.py
Abdallah
Approve a new release please đ
âą Fix spark integration filtering Databricks events.
âą Fix spark integration filtering Databricks events.
â11
Michael Robinson
Thank you for requesting a release @Abdallah. Three +1s from committers will authorize.
đ1
Michael Robinson
Thanks, all. The release is authorized and will be initiated within 2 business days.
Athitya Kumar
Hey folks! Do we have clear step-by-step documentation on how we can leverage the
ServiceLoader based approach for injecting specific OpenLineage customisations for tweaking the transport type with defaults / tweaking column level lineage etc?Maciej Obuchowski
For custom transport, you have to provide implementation of interface https://github.com/OpenLineage/OpenLineage/blob/4a1a5c3bf9767467b71ca0e1b6d820ba9e[âŠ]ain/java/io/openlineage/client/transports/TransportBuilder.java and point to it in
META_INF fileMaciej Obuchowski
But if I understand correctly, if you want to change behavior rather than extend, the correct way may be to either contribute it to repo - if that behavior is useful to anyone, or fork the repo
Athitya Kumar
@Maciej Obuchowski - Can you elaborate more on the "point to it in META_INF file"? Let's say we have the custom transport type built in a standalone jar by extending transport builder - what're the exact next steps to use this custom transport in the standalone jar when doing spark-submit?
Maciej Obuchowski
@Athitya Kumar your jar needs to have
META-INF/services/io.openlineage.client.transports.TransportBuilder with fully qualified class names of your custom TransportBuilders there - like openlineage-spark hasio.openlineage.client.transports.HttpTransportBuilder
-io.openlineage.client.transports.KafkaTransportBuilder
-io.openlineage.client.transports.ConsoleTransportBuilder
-io.openlineage.client.transports.FileTransportBuilder
-io.openlineage.client.transports.KinesisTransportBuilderAthitya Kumar
@Maciej Obuchowski - I think this change may be required for consumers to leverage custom transports, can you check & verify this GH comment?
https://github.com/OpenLineage/OpenLineage/issues/2007#issuecomment-1690350630
https://github.com/OpenLineage/OpenLineage/issues/2007#issuecomment-1690350630
Maciej Obuchowski
Probably, I will look at more details next week @Athitya Kumar as I'm in transit
đ1
Michael Robinson
1
@channel
We released OpenLineage 1.1.0, including:
Additions:
âą Flink: create Openlineage configuration based on Flink configuration
âą Java: add Javadocs to the Java client
âą Spark: append output dataset name to a job name
âą Spark: support Spark 3.4.1
Fixes:
âą Flink: fix a bug when getting schema for
âą Spark: fix ignored event
Plus additional bug fixes, doc changes and more.
Thanks to all the contributors, especially new contributors @pentium3 and @Abdallah!
Release: https://github.com/OpenLineage/OpenLineage/releases/tag/1.1.0
Changelog: https://github.com/OpenLineage/OpenLineage/blob/main/CHANGELOG.md
Commit history: https://github.com/OpenLineage/OpenLineage/compare/1.0.0...1.1.0
Maven: https://oss.sonatype.org/#nexus-search;quick~openlineage
PyPI: https://pypi.org/project/openlineage-python/
We released OpenLineage 1.1.0, including:
Additions:
âą Flink: create Openlineage configuration based on Flink configuration
#2033 @pawel-big-lebowskiâą Java: add Javadocs to the Java client
#2004 @julienledemâą Spark: append output dataset name to a job name
#2036 @pawel-big-lebowskiâą Spark: support Spark 3.4.1
#2057 @pawel-big-lebowskiFixes:
âą Flink: fix a bug when getting schema for
KafkaSink #2042 @pentium3âą Spark: fix ignored event
adaptive_spark_plan in Databricks #2061 @algorithmy1Plus additional bug fixes, doc changes and more.
Thanks to all the contributors, especially new contributors @pentium3 and @Abdallah!
Release: https://github.com/OpenLineage/OpenLineage/releases/tag/1.1.0
Changelog: https://github.com/OpenLineage/OpenLineage/blob/main/CHANGELOG.md
Commit history: https://github.com/OpenLineage/OpenLineage/compare/1.0.0...1.1.0
Maven: https://oss.sonatype.org/#nexus-search;quick~openlineage
PyPI: https://pypi.org/project/openlineage-python/
đ9
1Michael Robinson
@channel
Friendly reminder: our next in-person meetup is next Wednesday, August 30th in San Francisco at Astronomerâs offices in the Financial District. You can sign up and find the details on the meetup event page.
Friendly reminder: our next in-person meetup is next Wednesday, August 30th in San Francisco at Astronomerâs offices in the Financial District. You can sign up and find the details on the meetup event page.
George Polychronopoulos
hi Openlineage team , we would like to join one of your meetups(me and @Madhav Kakumani nad @Phil Rolph and we're wondering if you are hosting any meetups after the 18/9 ? We are trying to join this but air - tickets are quite expensive
Harel Shein
there will certainly be more meetups, donât worry about that!
Harel Shein
where are you located? perhaps we can try to organize a meetup closer to where you are.
George Polychronopoulos
Thanks a lot for the response, we are in London. We'd be glad to help you organise a meetup and also meet in person!
Michael Robinson
This is awesome, thanks @George Polychronopoulos. Iâll start a channel and invite you
Juan Luis Cano RodrĂguez
hi folks, I'm looking into exporting static metadata, and found that
DatasetEvent requires a eventTime, which in my mind doesn't make sense for static events. I'm setting it to None and the Python client seems to work, but wanted to ask if I'm missing something.PaweĆ LeszczyĆski
Although you emit
DatasetEvent, you still emit an event and eventTime is a valid marker.Juan Luis Cano RodrĂguez
so, should I use the current time at the moment of emitting it and that's it?
PaweĆ LeszczyĆski
1
yes, that should be it
1Juan Luis Cano RodrĂguez
and something else: I understand that Marquez does not yet support the 2.0 spec, hence it's incompatible with static metadata right? I tried to emit a list of
DatasetEvent s and got HTTPError: 422 Client Error: Unprocessable Entity for url: <http://localhost:3000/api/v1/lineage> (I'm using a FileTransport for now)PaweĆ LeszczyĆski
marquez is not capable of reflecting
DatasetEvents in DB but it should respond with Unsupported event typePaweĆ LeszczyĆski
and return 200 instead of
201 createdJuan Luis Cano RodrĂguez
I'll have a deeper look then, probably I'm doing something wrong. thanks @PaweĆ LeszczyĆski
Joshua Dotson
Hi folks. I have some pure golang jobs from which I need to emit OL events to Marquez. Is the right way to go about this to generate a Golang client from the Marquez OpenAPI spec and use that client from my go jobs?
Jakub DardziĆski
I'd rather generate them from OL spec (compliant with JSON Schema)
Joshua Dotson
I'll look into this. I take you to mean that I would use the OL spec which is available as a set of JSON schemas to create the data object and then HTTP POST it using vanilla Golang. Is that correct? Thank you for your help!
Jakub DardziĆski
Correct! Youâre also very welcome to contribute Golang client (currently we have Python & Java clients) if you manage to send events using golang đ
đ1
Michael Robinson
@channel
The agenda for the Toronto Meetup at Airflow Summit on 9/18 has been updated. This promises to be an exciting, richly productive discussion. Donât miss it if youâll be in the area!
1. Intros
2. Evolution of spec presentation/discussion (project background/history)
3. State of the community
4. Spark/Column lineage update
5. Airflow Provider update
6. Roadmap Discussion
7. Action items review/next steps
The agenda for the Toronto Meetup at Airflow Summit on 9/18 has been updated. This promises to be an exciting, richly productive discussion. Donât miss it if youâll be in the area!
1. Intros
2. Evolution of spec presentation/discussion (project background/history)
3. State of the community
4. Spark/Column lineage update
5. Airflow Provider update
6. Roadmap Discussion
7. Action items review/next steps
â€ïž3
Michael Robinson
New on the OpenLineage blog: a close look at the new OpenLineage Airflow Provider, including:
âą the critical improvements it brings to the integration
âą the high-level design
âą implementation details
âą an example operator
âą planned enhancements
âą a list of supported operators
âą more.
The post, by @Maciej Obuchowski, @Julien Le Dem and myself is live now on the OpenLineage blog.
âą the critical improvements it brings to the integration
âą the high-level design
âą implementation details
âą an example operator
âą planned enhancements
âą a list of supported operators
âą more.
The post, by @Maciej Obuchowski, @Julien Le Dem and myself is live now on the OpenLineage blog.
đ5
Sarwat Fatima
Hello, I'm currently in the process of following the instructions outlined in the provided getting started guide at https://openlineage.io/getting-started/. However, I've encountered a problem while attempting to complete Step 1 of the guide. Unfortunately, I'm encountering an internal server error at this stage. I did manage to successfully run Marquez, but it appears that there might be an issue that needs to be addressed. I have attached screen shots.
Jakub DardziĆski
is 5000 port taken by any other application? or
./docker/up.sh has some errors in logs?Sarwat Fatima
@Jakub DardziĆski 5000 port is not taken by any other application. The logs show some errors but I am not sure what is the issue here.
Maciej Obuchowski
I think Marquez is running on WSL while you're trying to connect from host computer?
Juan Luis Cano RodrĂguez
hi folks, for now I'm producing
.jsonl (or .ndjson ) files with one event per line, do you know if there's any way to validate those? would standard JSON Schema tools work?Juan Luis Cano RodrĂguez
reply by @U0544QC1DS9: yes đđŻ
đ1
ldacey
for namespaces, if my data is moving between sources (SFTP -> GCS -> Azure Blob (synapse connects to parquet datasets) then should my namespace be based on the client I am working with? my current namespace has been to refer to the bucket, but that falls apart when considering the data sources and some destinations. perhaps I should just add a field for client-name instead to have a consolidated view?
Maciej Obuchowski
> then should my namespace be based on the client I am working with?
I think each of those sources should be a different namespace?
I think each of those sources should be a different namespace?
ldacey
got it, yeah I was kind of picturing as one namespace for the client (we handle many clients but they are completely distinct entities). I was able to get it to work with multiple namespaces like you suggested and Marquez was able to plot everything correctly in the visualization
ldacey
I noticed some of my Dataset facets make more sense as Run facets, for example, the name of the specific file I processed and how many rows of data / size of the data for that schedule. that won't impact the Run facets Airflow provides right? I can still have the schedule information + my custom run facets?
Maciej Obuchowski
Yes, unless you name it the same as one of the Airflow facets đ
GitHubOpenLineageIssues
Hi, Will really appreciate if someone can guide me or provide me any pointer - if they have been able to implement authentication/authorization for access to Marquez. Have not seen much info around it. Any pointers greatly appreciated. Thanks in advance.
Julien Le Dem
1
Iâve seen people do this through the ingress controller in Kubernetes. Unfortunately I donât have documentation besides k8s specific ones you would find for the ingress controller youâre using. Youâd redirect any unauthenticated request to your identity provider
1Michael Robinson
â
1
Julien Le Dem
Iâll be there and looking forward to see @John Lukenoff âs presentation
Michael Barrientos
Can anyone let 3 people stuck downstairs into the 7th floor?
đ1
Willy Lulciuc
Sorry about that!
Yunhe
hello,everyone,i can run openLineage spark code in my notebook with python,but when use my idea to execute scala code like this:
import org.apache.spark.internal.Logging
import org.apache.spark.sql.SparkSession
import io.openlineage.client.OpenLineageClientUtils.loadOpenLineageYaml
import org.apache.spark.scheduler.{SparkListener, SparkListenerApplicationEnd, SparkListenerApplicationStart}
import sun.java2d.marlin.MarlinUtils.logInfo
object Test {
def main(args: Array[String]): Unit = {
val spark = SparkSession
.builder()
.master("local")
.appName("test")
.config("spark.jars.packages","io.openlineage:openlineage-spark:0.12.0")
.config("spark.extraListeners","io.openlineage.spark.agent.OpenLineageSparkListener")
.config("spark.openlineage.transport.type","console")
.getOrCreate()
spark.sparkContext.setLogLevel("INFO")
//spark.sparkContext.addSparkListener(new MySparkAppListener)
import spark.implicits._
val input = Seq((1, "zs", 2020), (2, "ls", 2023)).toDF("id", "name", "year")
input.select("id", "name").orderBy("id").show()
}
}
there is something wrong:
Exception in thread "spark-listener-group-shared" java.lang.NoSuchMethodError: io.openlineage.client.OpenLineageClientUtils.loadOpenLineageYaml(Ljava/io/InputStream;)Lio/openlineage/client/OpenLineageYaml;
at io.openlineage.spark.agent.ArgumentParser.extractOpenlineageConfFromSparkConf(ArgumentParser.java:114)
at io.openlineage.spark.agent.ArgumentParser.parse(ArgumentParser.java:78)
at io.openlineage.spark.agent.OpenLineageSparkListener.initializeContextFactoryIfNotInitialized(OpenLineageSparkListener.java:277)
at io.openlineage.spark.agent.OpenLineageSparkListener.onApplicationStart(OpenLineageSparkListener.java:267)
at org.apache.spark.scheduler.SparkListenerBus.doPostEvent(SparkListenerBus.scala:55)
at org.apache.spark.scheduler.SparkListenerBus.doPostEvent$(SparkListenerBus.scala:28)
at org.apache.spark.scheduler.AsyncEventQueue.doPostEvent(AsyncEventQueue.scala:37)
at org.apache.spark.scheduler.AsyncEventQueue.doPostEvent(AsyncEventQueue.scala:37)
at org.apache.spark.util.ListenerBus.postToAll(ListenerBus.scala:117)
at org.apache.spark.util.ListenerBus.postToAll$(ListenerBus.scala:101)
at org.apache.spark.scheduler.AsyncEventQueue.super$postToAll(AsyncEventQueue.scala:105)
at org.apache.spark.scheduler.AsyncEventQueue.$anonfun$dispatch$1(AsyncEventQueue.scala:105)
at scala.runtime.java8.JFunction0$mcJ$sp.apply(JFunction0$mcJ$sp.java:23)
at scala.util.DynamicVariable.withValue(DynamicVariable.scala:62)
at org.apache.spark.scheduler.AsyncEventQueue.org$apache$spark$scheduler$AsyncEventQueue$$dispatch(AsyncEventQueue.scala:100)
at org.apache.spark.scheduler.AsyncEventQueue$$anon$2.$anonfun$run$1(AsyncEventQueue.scala:96)
at org.apache.spark.util.Utils$.tryOrStopSparkContext(Utils.scala:1446)
at org.apache.spark.scheduler.AsyncEventQueue$$anon$2.run(AsyncEventQueue.scala:96)
i want to know how can i set idea scala environment correctly
import org.apache.spark.internal.Logging
import org.apache.spark.sql.SparkSession
import io.openlineage.client.OpenLineageClientUtils.loadOpenLineageYaml
import org.apache.spark.scheduler.{SparkListener, SparkListenerApplicationEnd, SparkListenerApplicationStart}
import sun.java2d.marlin.MarlinUtils.logInfo
object Test {
def main(args: Array[String]): Unit = {
val spark = SparkSession
.builder()
.master("local")
.appName("test")
.config("spark.jars.packages","io.openlineage:openlineage-spark:0.12.0")
.config("spark.extraListeners","io.openlineage.spark.agent.OpenLineageSparkListener")
.config("spark.openlineage.transport.type","console")
.getOrCreate()
spark.sparkContext.setLogLevel("INFO")
//spark.sparkContext.addSparkListener(new MySparkAppListener)
import spark.implicits._
val input = Seq((1, "zs", 2020), (2, "ls", 2023)).toDF("id", "name", "year")
input.select("id", "name").orderBy("id").show()
}
}
there is something wrong:
Exception in thread "spark-listener-group-shared" java.lang.NoSuchMethodError: io.openlineage.client.OpenLineageClientUtils.loadOpenLineageYaml(Ljava/io/InputStream;)Lio/openlineage/client/OpenLineageYaml;
at io.openlineage.spark.agent.ArgumentParser.extractOpenlineageConfFromSparkConf(ArgumentParser.java:114)
at io.openlineage.spark.agent.ArgumentParser.parse(ArgumentParser.java:78)
at io.openlineage.spark.agent.OpenLineageSparkListener.initializeContextFactoryIfNotInitialized(OpenLineageSparkListener.java:277)
at io.openlineage.spark.agent.OpenLineageSparkListener.onApplicationStart(OpenLineageSparkListener.java:267)
at org.apache.spark.scheduler.SparkListenerBus.doPostEvent(SparkListenerBus.scala:55)
at org.apache.spark.scheduler.SparkListenerBus.doPostEvent$(SparkListenerBus.scala:28)
at org.apache.spark.scheduler.AsyncEventQueue.doPostEvent(AsyncEventQueue.scala:37)
at org.apache.spark.scheduler.AsyncEventQueue.doPostEvent(AsyncEventQueue.scala:37)
at org.apache.spark.util.ListenerBus.postToAll(ListenerBus.scala:117)
at org.apache.spark.util.ListenerBus.postToAll$(ListenerBus.scala:101)
at org.apache.spark.scheduler.AsyncEventQueue.super$postToAll(AsyncEventQueue.scala:105)
at org.apache.spark.scheduler.AsyncEventQueue.$anonfun$dispatch$1(AsyncEventQueue.scala:105)
at scala.runtime.java8.JFunction0$mcJ$sp.apply(JFunction0$mcJ$sp.java:23)
at scala.util.DynamicVariable.withValue(DynamicVariable.scala:62)
at org.apache.spark.scheduler.AsyncEventQueue.org$apache$spark$scheduler$AsyncEventQueue$$dispatch(AsyncEventQueue.scala:100)
at org.apache.spark.scheduler.AsyncEventQueue$$anon$2.$anonfun$run$1(AsyncEventQueue.scala:96)
at org.apache.spark.util.Utils$.tryOrStopSparkContext(Utils.scala:1446)
at org.apache.spark.scheduler.AsyncEventQueue$$anon$2.run(AsyncEventQueue.scala:96)
i want to know how can i set idea scala environment correctly
PaweĆ LeszczyĆski
io.openlineage:openlineage-spark:0.12.0 -> could you repeat the steps with newer version?Yunhe
ok,it`s my first use thie lineage tool. first,I added dependences in my pom.xml like this:
<dependency>
<groupId>io.openlineage</groupId>
<artifactId>openlineage-java</artifactId>
<version>0.12.0</version>
</dependency>
<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-api</artifactId>
<version>2.7</version>
</dependency>
<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-core</artifactId>
<version>2.7</version>
</dependency>
<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-slf4j-impl</artifactId>
<version>2.7</version>
</dependency>
<dependency>
<groupId>io.openlineage</groupId>
<artifactId>openlineage-spark</artifactId>
<version>0.30.1</version>
</dependency>
my spark version is 3.3.1 and the version can not change
second, in file Openlineage/intergration/spark I enter command : docker-compose up and follow the steps in this doc:
https://openlineage.io/docs/integrations/spark/quickstart_local
there is no erro when i use notebook to execute pyspark for openlineage and I could get json message.
but after I enter "docker-compose up" ,I want to use my Idea tool to execute scala code like above,the erro happend like above. It seems that I does not configure the environment correctly. so how can i fix the problem .
<dependency>
<groupId>io.openlineage</groupId>
<artifactId>openlineage-java</artifactId>
<version>0.12.0</version>
</dependency>
<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-api</artifactId>
<version>2.7</version>
</dependency>
<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-core</artifactId>
<version>2.7</version>
</dependency>
<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-slf4j-impl</artifactId>
<version>2.7</version>
</dependency>
<dependency>
<groupId>io.openlineage</groupId>
<artifactId>openlineage-spark</artifactId>
<version>0.30.1</version>
</dependency>
my spark version is 3.3.1 and the version can not change
second, in file Openlineage/intergration/spark I enter command : docker-compose up and follow the steps in this doc:
https://openlineage.io/docs/integrations/spark/quickstart_local
there is no erro when i use notebook to execute pyspark for openlineage and I could get json message.
but after I enter "docker-compose up" ,I want to use my Idea tool to execute scala code like above,the erro happend like above. It seems that I does not configure the environment correctly. so how can i fix the problem .
PaweĆ LeszczyĆski
please use latest
io.openlineage:openlineage-spark:1.1.0 instead. openlineage-java is already contained in the jar, no need to add it on your own.
Sheeri Cabral (Collibra)
Will the August meeting be put up at https://wiki.lfaidata.foundation/display/OpenLineage/Monthly+TSC+meeting soon? (usually itâs up in a few days đ
Maciej Obuchowski
@Michael Robinson
Julien Le Dem
It sounds like there have been a few announcements at Google Next:
https://cloud.google.com/data-catalog/docs/how-to/open-lineage
https://cloud.google.com/dataproc/docs/guides/lineage
https://cloud.google.com/data-catalog/docs/how-to/open-lineage
https://cloud.google.com/dataproc/docs/guides/lineage
đ13
đ10
â€ïž5
Michael Robinson
![]()
1
@channel
The latest issue of OpenLineage News is out now! Please subscribe to get it directly in your inbox each month.
The latest issue of OpenLineage News is out now! Please subscribe to get it directly in your inbox each month.
đ2
Anirudh Shrinivason
Hi guys, I'd like to capture the
I was thinking that capturing this property might be useful for most databricks workloads, and whether it might make sense to auto-capture it along with other databricks variables, similar to how we capture mount points for the databricks jobs.
Does this sound okay? If so, then I can help to contribute this functionality
spark.databricks.clusterUsageTags.clusterAllTags property from databricks. However, the value of this is a list of keys, and therefore cannot be supported by custom environment facet builder.I was thinking that capturing this property might be useful for most databricks workloads, and whether it might make sense to auto-capture it along with other databricks variables, similar to how we capture mount points for the databricks jobs.
Does this sound okay? If so, then I can help to contribute this functionality
Maciej Obuchowski
Sounds good to me
Anirudh Shrinivason
Also, another small clarification is that when using
MergeIntoCommand, I'm receiving the lineage events on the backend, but I cannot seem to find any logging of the payload when I enable debug mode in openlineage. I remember there was a similar issue reported by another user in the past. May I check if it might be possible to help with this? It's making debugging quite hard for these cases. Thanks!Maciej Obuchowski
I think it only depends on log4j configuration
Maciej Obuchowski
this is what we have in
# Set everything to be logged to the console
-log4j.rootCategory=INFO, console
-log4j.appender.console=org.apache.log4j.ConsoleAppender
-log4j.appender.console.target=System.err
-log4j.appender.console.layout=org.apache.log4j.PatternLayout
-log4j.appender.console.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p %c{1}: %m%n
-
-# set the log level for the openlineage spark library
-log4j.logger.io.openlineage.spark=DEBUGthis is what we have in
log4j.properties in test environment and it worksAnirudh Shrinivason
Hmm... I can see the logs for the other commands, like createViewCommand etc. I just cannot see it for any of the delta runs
PaweĆ LeszczyĆski
that's interesting. So, logging is done here: https://github.com/OpenLineage/OpenLineage/blob/main/integration/spark/app/src/main/java/io/openlineage/spark/agent/EventEmitter.java#L63 and this code is unaware of delta.
The possible problem could be filtering delta events (which we do bcz of delta being noisy)
The possible problem could be filtering delta events (which we do bcz of delta being noisy)
PaweĆ LeszczyĆski
Recently, we've closed that https://github.com/OpenLineage/OpenLineage/issues/1982 which prevents generating events for `
createOrReplaceTempViewPaweĆ LeszczyĆski
and this is the code change: https://github.com/OpenLineage/OpenLineage/pull/1987/files
Anirudh Shrinivason
Hmm I'm a little confused here. I thought we are only filtering out events for certain specific commands, like show table etc. because its noisy right? Some important commands like MergeInto or SaveIntoDataSource used to be logged before, but I notice now that its not being logged anymore...
I'm using 0.23.0 openlineage version.
I'm using 0.23.0 openlineage version.
PaweĆ LeszczyĆski
yes, we do. it's just sometimes when doing a filter, we can remove too much. but SaveIntoDataSource and MergeInto should be fine, as we do check them within the tests
ldacey
it looks like my dynamic task mapping in Airflow has the same run ID in marquez, so even if I am processing 100 files, there is only one version of the data. is there a way to have a separate version of each dynamic task so I can track the filename etc?
Jakub DardziĆski
what version of Airflow are you using btw?
map_index should be indeed included when calculating run ID (itâs deterministic in Airflow integration)what version of Airflow are you using btw?
ldacey
2.7.0
I do see this error log in all of my dynamic tasks which might explain it:
I do see this error log in all of my dynamic tasks which might explain it:
[2023-09-05, 00:31:57 UTC] {manager.py:200} ERROR - Extractor returns non-valid metadata: None
-[2023-09-05, 00:31:57 UTC] {utils.py:401} ERROR - cannot import name 'get_operator_class' from 'airflow.providers.openlineage.utils' (/home/airflow/.local/lib/python3.11/site-packages/airflow/providers/openlineage/utils/__init__.py)
-Traceback (most recent call last):
- File "/home/airflow/.local/lib/python3.11/site-packages/airflow/providers/openlineage/utils/utils.py", line 399, in wrapper
- return f(*args, **kwargs)
- ^^^^^^^^^^^^^^^^^^
- File "/home/airflow/.local/lib/python3.11/site-packages/airflow/providers/openlineage/plugins/listener.py", line 93, in on_running
- **get_custom_facets(task_instance),
- ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
- File "/home/airflow/.local/lib/python3.11/site-packages/airflow/providers/openlineage/utils/utils.py", line 148, in get_custom_facets
- custom_facets["airflow_mappedTask"] = AirflowMappedTaskRunFacet.from_task_instance(task_instance)
- ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
- File "/home/airflow/.local/lib/python3.11/site-packages/airflow/providers/openlineage/plugins/facets.py", line 36, in from_task_instance
- from airflow.providers.openlineage.utils import get_operator_class
-ImportError: cannot import name 'get_operator_class' from 'airflow.providers.openlineage.utils' (/home/airflow/.local/lib/python3.11/site-packages/airflow/providers/openlineage/utils/__init__.py)ldacey
I only have a few custom operators with the on_complete facet so I think this is a built in one - it runs before my task custom logs for example
ldacey
and any time I messed up my custom facet, the error would be at the bottom of the logs. this is on top, probably an on_start facet?
Jakub DardziĆski
seems like some circular import
Jakub DardziĆski
I just tested it manually, itâs a bug in OL provider. let me fix that
ldacey
cool, thanks. I am glad it is just a bug, I was afraid dynamic tasks were not supported for a minute there
ldacey
how do the provider updates work? they can be released in between Airflow releases and issues for them are raised on the main Airflow repo?
Jakub DardziĆski
generally speaking anything related to OL-Airflow should be placed to Airflow repo, important changes/bug fixes would be implemented in OL repo as well
ldacey
got it, thanks
ldacey
is there a way for me to install the openlineage provider based on the commit you made to fix the circular imports?
i was going to try to install from Airflow main branch but didnt want to mess anything up
i was going to try to install from Airflow main branch but didnt want to mess anything up
ldacey
I saw it was merged to airflow main but it is not in 2.7.1 and there is no 1.0.3 provider version yet, so I wondered if I could manually install it for the time being
Jakub DardziĆski
https://github.com/apache/airflow/blob/main/BREEZE.rst#preparing-provider-packages
building the provider package on your own could be best idea probably? that depends on how you manage your Airflow instance
building the provider package on your own could be best idea probably? that depends on how you manage your Airflow instance
Jakub DardziĆski
there's 1.1.0rc1 btw
ldacey
perfect, thanks. I got started with breeze but then stopped haha
đ1
ldacey
The dynamic task mapping error is gone, I did run into this:
File "/home/airflow/.local/lib/python3.11/site-packages/airflow/providers/openlineage/extractors/base.py", line 70, in disabled_operators
operator.strip() for operator in conf.get("openlineage", "disabled_for_operators").split(";")
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.11/site-packages/airflow/configuration.py", line 1065, in get
raise AirflowConfigException(f"section/key [{section}/{key}] not found in config")
I am redeploying now with that option added to my config. I guess it did not use the default which should be ""
File "/home/airflow/.local/lib/python3.11/site-packages/airflow/providers/openlineage/extractors/base.py", line 70, in disabled_operators
operator.strip() for operator in conf.get("openlineage", "disabled_for_operators").split(";")
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.11/site-packages/airflow/configuration.py", line 1065, in get
raise AirflowConfigException(f"section/key [{section}/{key}] not found in config")
I am redeploying now with that option added to my config. I guess it did not use the default which should be ""
ldacey
added "disabled_for_operators" to my openlineage config and it worked (using Airflow helm chart - not sure if that means there is an error because the value I provided should just be the default value, not sure why I needed to explicitly specify it)
openlineage:
disabled_for_operators: ""
...
this is so much better and makes a lot more sense. most of my tasks are dynamic so I was missing a lot of metadata before the fix, thanks!
openlineage:
disabled_for_operators: ""
...
this is so much better and makes a lot more sense. most of my tasks are dynamic so I was missing a lot of metadata before the fix, thanks!
Abdallah
Hello Everyone,
I've been diving into the Marquez codebase and found a performance bottleneck in
Issue link : https://github.com/MarquezProject/marquez/issues/2608
I believe there's even more room for optimization, especially if we adjust the
Would the team be open to a PR where I share the optimized query and discuss potential further refinements? I believe these changes could significantly enhance the Marquez web UI experience.
PR link : https://github.com/MarquezProject/marquez/pull/2609
Looking forward to your feedback.
I've been diving into the Marquez codebase and found a performance bottleneck in
JobDao.java for the query related to namespaceName=MyNameSpace limit=10 and 12s with limit=25. I managed to optimize it using CTEs, and the execution times dropped dramatically to 300ms (for limit=100) and under 100ms (for limit=25 ) on the same cluster.Issue link : https://github.com/MarquezProject/marquez/issues/2608
I believe there's even more room for optimization, especially if we adjust the
job_facets_view to include the namespace_name column.Would the team be open to a PR where I share the optimized query and discuss potential further refinements? I believe these changes could significantly enhance the Marquez web UI experience.
PR link : https://github.com/MarquezProject/marquez/pull/2609
Looking forward to your feedback.
đ„4
Jakub DardziĆski
@Willy Lulciuc wdyt?
Bernat Gabor
Has there been any conversation on the extensibility of facets/concepts? E.g.:
âą how does one extends the list of run states https://openlineage.io/docs/spec/run-cycle to add a paused/resumed state?
âą how does one extend https://openlineage.io/docs/spec/facets/run-facets/nominal_time to add a created at field?
âą how does one extends the list of run states https://openlineage.io/docs/spec/run-cycle to add a paused/resumed state?
âą how does one extend https://openlineage.io/docs/spec/facets/run-facets/nominal_time to add a created at field?
Julien Le Dem
Hello Bernat,
The primary mechanism to extend the model is through facets. You can either:
âą create new standard facets in the spec: https://github.com/OpenLineage/OpenLineage/tree/main/spec/facets
âą create custom facets defined somewhere else with a prefix in their name: https://github.com/OpenLineage/OpenLineage/blob/main/spec/OpenLineage.md#custom-facet-naming
âą Update existing facets with a backward compatible change (example: adding an optional field).
The core spec can also be modified. Here is an example of adding a state
That being said I think more granular states like pause/resume are probably better suited in a run facet. There was an issue opened for that particular one a while ago: https://github.com/OpenLineage/OpenLineage/issues/9 maybe that particular discussion can continue there.
For the nominal time facet, You could open an issue describing the use case and on community agreement follow up with a PR on the facet itself: https://github.com/OpenLineage/OpenLineage/blob/main/spec/facets/NominalTimeRunFacet.json
(adding an optional field is backwards compatible)
The primary mechanism to extend the model is through facets. You can either:
âą create new standard facets in the spec: https://github.com/OpenLineage/OpenLineage/tree/main/spec/facets
âą create custom facets defined somewhere else with a prefix in their name: https://github.com/OpenLineage/OpenLineage/blob/main/spec/OpenLineage.md#custom-facet-naming
âą Update existing facets with a backward compatible change (example: adding an optional field).
The core spec can also be modified. Here is an example of adding a state
That being said I think more granular states like pause/resume are probably better suited in a run facet. There was an issue opened for that particular one a while ago: https://github.com/OpenLineage/OpenLineage/issues/9 maybe that particular discussion can continue there.
For the nominal time facet, You could open an issue describing the use case and on community agreement follow up with a PR on the facet itself: https://github.com/OpenLineage/OpenLineage/blob/main/spec/facets/NominalTimeRunFacet.json
(adding an optional field is backwards compatible)
đ1
Bernat Gabor
I see, so in general one is best copying a standard facet and maintain it under a different name. That way can be made mandatory đ and one does not need to be blocked for a long time until there's a community agreement đ€
Julien Le Dem
Yes, The goal of custom facets is to allow you to experiment and extend the spec however you want without having to wait for approval.
If the custom facet is very specific to a third party project/product then it makes sense for it to stay a custom facet.
If it is more generic then it makes sense to add it to the core facets as part of the spec.
Hopefully community agreement can be achieved relatively quickly. Unless someone is strongly against something, it can be added without too much red tape. Typically with support in at least one of the integrations to validate the model.
If the custom facet is very specific to a third party project/product then it makes sense for it to stay a custom facet.
If it is more generic then it makes sense to add it to the core facets as part of the spec.
Hopefully community agreement can be achieved relatively quickly. Unless someone is strongly against something, it can be added without too much red tape. Typically with support in at least one of the integrations to validate the model.
Michael Robinson
@channel
This monthâs TSC meeting is next Thursday the 14th at 10am PT. On the tentative agenda:
âą announcements
âą recent releases
âą demo: Spark integration tests in Databricks runtime
âą open discussion
âą more (TBA)
More info and the meeting link can be found on the website. All are welcome! Also, feel free to reply or DM me with discussion topics, agenda items, etc.
This monthâs TSC meeting is next Thursday the 14th at 10am PT. On the tentative agenda:
âą announcements
âą recent releases
âą demo: Spark integration tests in Databricks runtime
âą open discussion
âą more (TBA)
More info and the meeting link can be found on the website. All are welcome! Also, feel free to reply or DM me with discussion topics, agenda items, etc.
đ1
Michael Robinson
@channel
The first Toronto OpenLineage Meetup, featuring a presentation by recent adopter Metaphor, is just one week away. On the agenda:
1. Evolution of spec presentation/discussion (project background/history)
2. State of the community
3. Integrating OpenLineage with Metaphor (by special guests Ye & Ivan)
4. Spark/Column lineage update
5. Airflow Provider update
6. Roadmap Discussion
Find more details and RSVP here.
The first Toronto OpenLineage Meetup, featuring a presentation by recent adopter Metaphor, is just one week away. On the agenda:
1. Evolution of spec presentation/discussion (project background/history)
2. State of the community
3. Integrating OpenLineage with Metaphor (by special guests Ye & Ivan)
4. Spark/Column lineage update
5. Airflow Provider update
6. Roadmap Discussion
Find more details and RSVP here.
đ7
John Lukenoff
Iâm seeing some odd behavior with my http transport when upgrading airflow/openlineage-airflow from 2.3.2 -> 2.6.3 and 0.24.0 -> 0.28.0. Previously I had a config like this that let me provide my own auth tokens. However, after upgrading Iâm getting a 401 from the endpoint and further debugging seems to reveal that weâre not using the token provided in my TokenProvider. Does anyone know if something changed between these versions that could be causing this? (more details in 𧔠)
transport:
- type: http
- url: <https://my.fake-marquez-endpoint.com>
- auth:
- type: some.fully.qualified.classpathJohn Lukenoff
If I log this line I can tell the TokenProvider is the class instance I would expect: https://github.com/OpenLineage/OpenLineage/blob/45d94fb73b5488d34b8ca544b58317382ceb3980/client/python/openlineage/client/transport/http.py#L55
John Lukenoff
However, if I log the
token_provider here I get the origin TokenProvider: https://github.com/OpenLineage/OpenLineage/blob/45d94fb73b5488d34b8ca544b58317382ceb3980/client/python/openlineage/client/transport/http.py#L154John Lukenoff
Ah I think I see the issue. Looks like this was introduced here, we are instantiating with the base token provider here when we should be using the subclass: https://github.com/OpenLineage/OpenLineage/pull/1869/files#diff-2f8ea6f9a22b5567de8ab56c6a63da8e7adf40cb436ee5e7e6b16e70a82afe05R57
Sarwat Fatima
This particular code in docker-compose exits with code 1 because it is unable to find wait-for-it.sh, file in the container. I have checked the mounting path from the local machine, It is correct and the path on the container for Marquez is also correct i.e. /usr/src/app but it is unable to mount the wait-for-it.sh. Does anyone know why is this? This code exists in the open lineage repository as well https://github.com/OpenLineage/OpenLineage/blob/main/integration/spark/docker-compose.yml
# Marquez as an OpenLineage Client
- api:
- image: marquezproject/marquez
- container_name: marquez-api
- ports:
- - "5000:5000"
- - "5001:5001"
- volumes:
- - ./docker/wait-for-it.sh:/usr/src/app/wait-for-it.sh
- links:
- - "db:postgres"
- depends_on:
- - db
- entrypoint: [ "./wait-for-it.sh", "db:5432", "--", "./entrypoint.sh" ]Sarwat Fatima
This is the error message:
Maciej Obuchowski
no permissions?
Guntaka Jeevan Paul
I am trying to run Google Cloud Composer where i have added the openlineage-airflow pypi packagae as a dependency and have added the env OPENLINEAGE_EXTRACTORS to point to my custom extractor. I have added a folder by name dependencies and inside that i have placed my extractor file, and the path given to OPENLINEAGE_EXTRACTORS is dependencies.<file_name>.<extractor_class_name>âŠstill it fails with the exception saying No module named âdependenciesâ. Can anyone kindly help me out on correcting my mistake
Harel Shein
Hey @Guntaka Jeevan Paul, can you share some details on which versions of airflow and openlineage youâre using?
Guntaka Jeevan Paul
airflow ---> 2.5.3, openlinegae-airflow ---> 1.1.0
Guntaka Jeevan Paul
import traceback
-import uuid
-from typing import List, Optional
-
-from openlineage.airflow.extractors.base import BaseExtractor, TaskMetadata
-from openlineage.airflow.utils import get_job_name
-
-
-class BigQueryInsertJobExtractor(BaseExtractor):
- def __init__(self, operator):
- super().__init__(operator)
-
- @classmethod
- def get_operator_classnames(cls) -> List[str]:
- return ['BigQueryInsertJobOperator']
-
- def extract(self) -> Optional[TaskMetadata]:
- return None
-
- def extract_on_complete(self, task_instance) -> Optional[TaskMetadata]:
- self.log.debug(f"JEEVAN ---> extract_on_complete({task_instance})")
- random_uuid = str(uuid.uuid4())
- self.log.debug(f"JEEVAN ---> Randomly Generated UUID --> {random_uuid}")
-
- self.operator.job_id = random_uuid
-
- return TaskMetadata(
- name=get_job_name(task=self.operator)
- )Guntaka Jeevan Paul
this is the custom extractor code that im trying with
Harel Shein
thanks @Guntaka Jeevan Paul, will try to take a deeper look tomorrow
Maciej Obuchowski
This sounds like general Python problem
No module named 'dependencies'.This sounds like general Python problem
Maciej Obuchowski
basically, if you're able to import the file from your dag code, OL should be able too
Guntaka Jeevan Paul
The Problem is in the GCS Composer there is a component called Triggerer, which they say is used for deferrable operatorsâŠi have logged into that pod and i could see that the GCS Bucket is not mounted on this, but i am unable to understand why is the initialisation happening inside the triggerer pod
Guntaka Jeevan Paul
Maciej Obuchowski
> The Problem is in the GCS Composer there is a component called Triggerer, which they say is used for deferrable operatorsâŠi have logged into that pod and i could see that the GCS Bucket is not mounted on this, but i am unable to understand why is the initialisation happening inside the triggerer pod
OL integration is not running on triggerer, only on worker and scheduler pods
OL integration is not running on triggerer, only on worker and scheduler pods
Guntaka Jeevan Paul
Guntaka Jeevan Paul
As you can see in this screenshot i am seeing the logs of the triggerer and it says clearly unable to import plugin openlineage
Maciej Obuchowski
I see. There are few possible things to do here - composer could mount the user files, Airflow could not start plugins on triggerer, or we could detect we're on triggerer and not import anything there. However, does it impact OL or Airflow operation in other way than this log?
Maciej Obuchowski
Probably we'd have to do something if that really bothers you as there won't be further changes to Airflow 2.5
Guntaka Jeevan Paul
The Problem is it is actually not registering this custom extractor written by me, henceforth i am just receiving the DefaultExtractor things and my piece of extractor code is not even getting triggered
Guntaka Jeevan Paul
any suggestions to try @Maciej Obuchowski
Maciej Obuchowski
Could you share worker logs?
Maciej Obuchowski
and check if module is importable from your dag code?
Guntaka Jeevan Paul
these are the worker pod logsâŠwhere there is no log of openlineageplugin
Guntaka Jeevan Paul
https://openlineage.slack.com/archives/C01CK9T7HKR/p1694608076879469?thread_ts=1694545905.974339&cid=C01CK9T7HKR --> sure will check now on this one
Maciej Obuchowski
it doesn't see
however, is it necessary? I generally see people using imports directly from dags folder
{
- "textPayload": "Traceback (most recent call last): File \"/opt/python3.8/lib/python3.8/site-packages/openlineage/airflow/utils.py\", line 427, in import_from_string module = importlib.import_module(module_path) File \"/opt/python3.8/lib/python3.8/importlib/__init__.py\", line 127, in import_module return _bootstrap._gcd_import(name[level:], package, level) File \"<frozen importlib._bootstrap>\", line 1014, in _gcd_import File \"<frozen importlib._bootstrap>\", line 991, in _find_and_load File \"<frozen importlib._bootstrap>\", line 961, in _find_and_load_unlocked File \"<frozen importlib._bootstrap>\", line 219, in _call_with_frames_removed File \"<frozen importlib._bootstrap>\", line 1014, in _gcd_import File \"<frozen importlib._bootstrap>\", line 991, in _find_and_load File \"<frozen importlib._bootstrap>\", line 961, in _find_and_load_unlocked File \"<frozen importlib._bootstrap>\", line 219, in _call_with_frames_removed File \"<frozen importlib._bootstrap>\", line 1014, in _gcd_import File \"<frozen importlib._bootstrap>\", line 991, in _find_and_load File \"<frozen importlib._bootstrap>\", line 973, in _find_and_load_unlockedModuleNotFoundError: No module named 'airflow.gcs'",
- "insertId": "pt2eu6fl9z5vw",
- "resource": {
- "type": "cloud_composer_environment",
- "labels": {
- "environment_name": "openlineage",
- "location": "us-west1",
- "project_id": "acceldata-acm"
- }
- },
- "timestamp": "2023-09-13T06:20:44.131577764Z",
- "severity": "ERROR",
- "labels": {
- "worker_id": "airflow-worker-xttt8"
- },
- "logName": "projects/acceldata-acm/logs/airflow-worker",
- "receiveTimestamp": "2023-09-13T06:20:48.847319607Z"
- },it doesn't see
No module named 'airflow.gcs' that is part of your extractor path airflow.gcs.dags.big_query_insert_job_extractor.BigQueryInsertJobExtractorhowever, is it necessary? I generally see people using imports directly from dags folder
Guntaka Jeevan Paul
this is one of the experimentation that i have did, but then i reverted it back to keeping it to dependencies.big_query_insert_job_extractor.BigQueryInsertJobExtractorâŠwhere dependencies is a module i have created inside my dags folder
Guntaka Jeevan Paul
these are the logs of the triggerer pod specifically
Maciej Obuchowski
yeah it would be expected to have this in triggerer where it's not mounted, but will it behave the same for worker where it's mounted?
Maciej Obuchowski
maybe
___init___.py is missing for top-level dag path?Guntaka Jeevan Paul
these are the logs of the worker pod at startup, where it does not complain of the plugin like in triggerer, but when tasks are run on this workerâŠsomehow it is not picking up the extractor for the operator that i have written it for
Guntaka Jeevan Paul
https://openlineage.slack.com/archives/C01CK9T7HKR/p1694609229577469?thread_ts=1694545905.974339&cid=C01CK9T7HKR --> you mean to make the dags folder as well like a module by adding the init.py?
Maciej Obuchowski
yes, I would put whole custom code directly in dags folder, to make sure import paths are the problem
Maciej Obuchowski
and would be nice if you could set
AIRFLOW__LOGGING__LOGGING_LEVEL="DEBUG"Guntaka Jeevan Paul
Starting the process, got command: triggerer
-Initializing airflow.cfg.
-airflow.cfg initialization is done.
-[2023-09-13T13:11:46.620+0000] {settings.py:267} DEBUG - Setting up DB connection pool (PID 8)
-[2023-09-13T13:11:46.622+0000] {settings.py:372} DEBUG - settings.prepare_engine_args(): Using pool settings. pool_size=5, max_overflow=10, pool_recycle=570, pid=8
-[2023-09-13T13:11:46.742+0000] {cli_action_loggers.py:39} DEBUG - Adding <function default_action_log at 0x7ff39ca1d3a0> to pre execution callback
-[2023-09-13T13:11:47.638+0000] {cli_action_loggers.py:65} DEBUG - Calling callbacks: [<function default_action_log at 0x7ff39ca1d3a0>]
- ____________ _____________
- ____ |__( )_________ __/__ /________ __
-____ /| |_ /__ ___/_ /_ __ /_ __ \_ | /| / /
-___ ___ | / _ / _ __/ _ / / /_/ /_ |/ |/ /
- _/_/ |_/_/ /_/ /_/ /_/ \____/____/|__/
-[2023-09-13T13:11:50.527+0000] {plugins_manager.py:300} DEBUG - Loading plugins
-[2023-09-13T13:11:50.580+0000] {plugins_manager.py:244} DEBUG - Loading plugins from directory: /home/airflow/gcs/plugins
-[2023-09-13T13:11:50.581+0000] {plugins_manager.py:224} DEBUG - Loading plugins from entrypoints
-[2023-09-13T13:11:50.587+0000] {plugins_manager.py:227} DEBUG - Importing entry_point plugin OpenLineagePlugin
-[2023-09-13T13:11:50.740+0000] {utils.py:430} WARNING - No module named 'boto3'
-[2023-09-13T13:11:50.743+0000] {utils.py:430} WARNING - No module named 'botocore'
-[2023-09-13T13:11:50.833+0000] {utils.py:430} WARNING - No module named 'airflow.providers.sftp'
-[2023-09-13T13:11:51.144+0000] {utils.py:430} WARNING - No module named 'big_query_insert_job_extractor'
-[2023-09-13T13:11:51.145+0000] {plugins_manager.py:237} ERROR - Failed to import plugin OpenLineagePlugin
-Traceback (most recent call last):
- File "/opt/python3.8/lib/python3.8/site-packages/openlineage/airflow/utils.py", line 427, in import_from_string
- module = importlib.import_module(module_path)
- File "/opt/python3.8/lib/python3.8/importlib/__init__.py", line 127, in import_module
- return _bootstrap._gcd_import(name[level:], package, level)
- File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
- File "<frozen importlib._bootstrap>", line 991, in _find_and_load
- File "<frozen importlib._bootstrap>", line 973, in _find_and_load_unlocked
-ModuleNotFoundError: No module named 'big_query_insert_job_extractor'
-
-The above exception was the direct cause of the following exception:
-
-Traceback (most recent call last):
- File "/opt/python3.8/lib/python3.8/site-packages/airflow/plugins_manager.py", line 229, in load_entrypoint_plugins
- plugin_class = entry_point.load()
- File "/opt/python3.8/lib/python3.8/site-packages/setuptools/_vendor/importlib_metadata/__init__.py", line 194, in load
- module = import_module(match.group('module'))
- File "/opt/python3.8/lib/python3.8/importlib/__init__.py", line 127, in import_module
- return _bootstrap._gcd_import(name[level:], package, level)
- File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
- File "<frozen importlib._bootstrap>", line 991, in _find_and_load
- File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
- File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
- File "<frozen importlib._bootstrap_external>", line 843, in exec_module
- File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
- File "/opt/python3.8/lib/python3.8/site-packages/openlineage/airflow/plugin.py", line 32, in <module>
- from openlineage.airflow import listener
- File "/opt/python3.8/lib/python3.8/site-packages/openlineage/airflow/listener.py", line 75, in <module>
- extractor_manager = ExtractorManager()
- File "/opt/python3.8/lib/python3.8/site-packages/openlineage/airflow/extractors/manager.py", line 16, in __init__
- self.task_to_extractor = Extractors()
- File "/opt/python3.8/lib/python3.8/site-packages/openlineage/airflow/extractors/extractors.py", line 122, in __init__
- extractor = import_from_string(extractor.strip())
- File "/opt/python3.8/lib/python3.8/site-packages/openlineage/airflow/utils.py", line 431, in import_from_string
- raise ImportError(f"Failed to import {path}") from e
-ImportError: Failed to import big_query_insert_job_extractor.BigQueryInsertJobExtractor
-[2023-09-13T13:11:51.235+0000] {plugins_manager.py:227} DEBUG - Importing entry_point plugin composer_menu_plugin
-[2023-09-13T13:11:51.719+0000] {plugins_manager.py:316} DEBUG - Loading 1 plugin(s) took 1.14 seconds
-[2023-09-13T13:11:51.733+0000] {triggerer_job.py:101} INFO - Starting the triggerer
-[2023-09-13T13:11:51.734+0000] {selector_events.py:59} DEBUG - Using selector: EpollSelector
-[2023-09-13T13:11:56.118+0000] {base_job.py:240} DEBUG - [heartbeat]
-[2023-09-13T13:12:01.359+0000] {base_job.py:240} DEBUG - [heartbeat]
-[2023-09-13T13:12:06.665+0000] {base_job.py:240} DEBUG - [heartbeat]
-[2023-09-13T13:12:11.880+0000] {base_job.py:240} DEBUG - [heartbeat]
-[2023-09-13T13:12:17.098+0000] {base_job.py:240} DEBUG - [heartbeat]
-[2023-09-13T13:12:22.323+0000] {base_job.py:240} DEBUG - [heartbeat]
-[2023-09-13T13:12:27.597+0000] {base_job.py:240} DEBUG - [heartbeat]
-[2023-09-13T13:12:32.826+0000] {base_job.py:240} DEBUG - [heartbeat]
-[2023-09-13T13:12:38.049+0000] {base_job.py:240} DEBUG - [heartbeat]
-[2023-09-13T13:12:43.275+0000] {base_job.py:240} DEBUG - [heartbeat]
-[2023-09-13T13:12:48.509+0000] {base_job.py:240} DEBUG - [heartbeat]
-[2023-09-13T13:12:53.867+0000] {base_job.py:240} DEBUG - [heartbeat]
-[2023-09-13T13:12:59.087+0000] {base_job.py:240} DEBUG - [heartbeat]
-[2023-09-13T13:13:04.300+0000] {base_job.py:240} DEBUG - [heartbeat]
-[2023-09-13T13:13:09.539+0000] {base_job.py:240} DEBUG - [heartbeat]
-[2023-09-13T13:13:14.785+0000] {base_job.py:240} DEBUG - [heartbeat]
-[2023-09-13T13:13:20.007+0000] {base_job.py:240} DEBUG - [heartbeat]
-[2023-09-13T13:13:25.274+0000] {base_job.py:240} DEBUG - [heartbeat]
-[2023-09-13T13:13:30.510+0000] {base_job.py:240} DEBUG - [heartbeat]
-[2023-09-13T13:13:35.729+0000] {base_job.py:240} DEBUG - [heartbeat]
-[2023-09-13T13:13:40.960+0000] {base_job.py:240} DEBUG - [heartbeat]
-[2023-09-13T13:13:46.444+0000] {base_job.py:240} DEBUG - [heartbeat]
-[2023-09-13T13:13:51.751+0000] {base_job.py:240} DEBUG - [heartbeat]
-[2023-09-13T13:13:57.084+0000] {base_job.py:240} DEBUG - [heartbeat]
-[2023-09-13T13:14:02.310+0000] {base_job.py:240} DEBUG - [heartbeat]
-[2023-09-13T13:14:07.535+0000] {base_job.py:240} DEBUG - [heartbeat]
-[2023-09-13T13:14:12.754+0000] {base_job.py:240} DEBUG - [heartbeat]
-[2023-09-13T13:14:17.967+0000] {base_job.py:240} DEBUG - [heartbeat]
-[2023-09-13T13:14:23.185+0000] {base_job.py:240} DEBUG - [heartbeat]
-[2023-09-13T13:14:28.406+0000] {base_job.py:240} DEBUG - [heartbeat]
-[2023-09-13T13:14:33.661+0000] {base_job.py:240} DEBUG - [heartbeat]
-[2023-09-13T13:14:38.883+0000] {base_job.py:240} DEBUG - [heartbeat]
-[2023-09-13T13:14:44.247+0000] {base_job.py:240} DEBUG - [heartbeat]Guntaka Jeevan Paul
still the same error in the triggerer pod
Guntaka Jeevan Paul
have changed the dags folder where i have added the init file as you suggested and then have updated the OPENLINEAGE_EXTRACTORS to big_query_insert_job_extractor.BigQueryInsertJobExtractorâŠstill the same thing
Maciej Obuchowski
> still the same error in the triggerer pod
it won't change, we're not trying to fix the triggerer import but worker, and should look only at worker pod at this point
it won't change, we're not trying to fix the triggerer import but worker, and should look only at worker pod at this point
Guntaka Jeevan Paul
extractor for <class 'airflow.providers.google.cloud.operators.bigquery.BigQueryInsertJobOperator'> is <class 'big_query_insert_job_extractor.BigQueryInsertJobExtractor'
-
-Using extractor BigQueryInsertJobExtractor task_type=BigQueryInsertJobOperator airflow_dag_id=data_analytics_dag task_id=join_bq_datasets.bq_join_holidays_weather_data_2021 airflow_run_id=manual__2023-09-13T13:24:08.946947+00:00
-
-fatal: not a git repository (or any parent up to mount point /home/airflow)
-Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).
-fatal: not a git repository (or any parent up to mount point /home/airflow)
-Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).Guntaka Jeevan Paul
able to see these logs in the worker podâŠso what you said is right that it is able to get the extractor but i get the below error immediately where it says not a git repository
Guntaka Jeevan Paul
seems like we are almost there nearbyâŠam i missing something obvious
Maciej Obuchowski
>
hm, this could be the actual bug?
fatal: not a git repository (or any parent up to mount point /home/airflow)
-> Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).
-> fatal: not a git repository (or any parent up to mount point /home/airflow)
-> Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).hm, this could be the actual bug?
Jakub DardziĆski
thatâs casual log in composer
Jakub DardziĆski
thatâs actually class from your custom module, right?
extractor for <class 'airflow.providers.google.cloud.operators.bigquery.BigQueryInsertJobOperator'> is <class 'big_query_insert_job_extractor.BigQueryInsertJobExtractor'thatâs actually class from your custom module, right?
Jakub DardziĆski
Iâve done experiment, thatâs how gcs looks like
Jakub DardziĆski
and env vars
Jakub DardziĆski
I have this extractor detected as expected
Jakub DardziĆski
seens as
<class 'dependencies.bq.BigQueryInsertJobExtractor'>Jakub DardziĆski
no
__init__.py in base dags folderJakub DardziĆski
I also checked that triggerer pod indeed has no gcsfuse set up, tbh no idea why, maybe some kind of optimization
the only effect is that when loading plugins in triggerer it throws some errors in logs, we donât do anything at the moment there
the only effect is that when loading plugins in triggerer it throws some errors in logs, we donât do anything at the moment there
Guntaka Jeevan Paul
okkâŠgot it @Jakub DardziĆskiâŠso the init at the top level of dags is as well not reqd, got it. Just one more doubt, there is a requirement where i want to change the operators property in the extractor inside the extract function, will that be taken into account and the operatorâs execute be called with the property that i have populated in my extractor?
Guntaka Jeevan Paul
for example i want to add a custom job_id to the BigQueryInsertJobOperator, so wheneerv someone uses the BigQueryInsertJobOperator operator i want to intercept that and add this job_id property to the operatorâŠwill that work?
Jakub DardziĆski
Iâm not sure if using OL for such thing is best choice. Wouldnât it be better to subclass the operator?
Jakub DardziĆski
but the answer is: it dependes on the airflow version, in 2.3+ Iâm pretty sure the changed property stays in execute method
Guntaka Jeevan Paul
yeah ideally that is how we should have done this but the problem is our client is having around 1000+ Dagâs in different google cloud projects, which are owned by multiple teamsâŠso they are not willing to change anything in their dag. Thankfully they are using airflow 2.4.3
Maciej Obuchowski
task_policy might be better tool for that: https://airflow.apache.org/docs/apache-airflow/2.6.0/administration-and-deployment/cluster-policies.html
â1
Jakub DardziĆski
btw I double-checked - execute method is in different process so this would not change taskâs attribute there
Guntaka Jeevan Paul
@Jakub DardziĆski any idea how can we achieve this one. ---> https://openlineage.slack.com/archives/C01CK9T7HKR/p1694849427228709
Guntaka Jeevan Paul
@here has anyone succeded in getting a custom extractor to work in GCP Cloud Composer or AWS MWAA, seems like there is no way
Mars Lan
I'm getting quite close with MWAA. See https://openlineage.slack.com/archives/C01CK9T7HKR/p1692743745585879.
Suraj Gupta
I am exploring Spark - OpenLineage integration (using the latest PySpark and OL versions). I tested a simple pipeline which:
âą Reads JSON data into PySpark DataFrame
âą Apply data transformations
âą Write transformed data to MySQL database
Observed that we receive 4 events (2
Team, can you please confirm if this behaviour is expected? Seems to be different from the Airflow integration where we relate jobs to Parent Jobs.
âą Reads JSON data into PySpark DataFrame
âą Apply data transformations
âą Write transformed data to MySQL database
Observed that we receive 4 events (2
START and 2 COMPLETE) for the same job name. The events are almost identical with a small diff in the facets. All the events share the same runId, and we don't get any parentRunId.Team, can you please confirm if this behaviour is expected? Seems to be different from the Airflow integration where we relate jobs to Parent Jobs.
Damien Hawes
The Spark integration requires that two parameters are passed to it, namely:
You can find the list of parameters here:
https://github.com/OpenLineage/OpenLineage/blob/main/integration/spark/README.md
spark.openlineage.parentJobName
-spark.openlineage.parentRunIdYou can find the list of parameters here:
https://github.com/OpenLineage/OpenLineage/blob/main/integration/spark/README.md
Suraj Gupta
Thanks, will check this out
Damien Hawes
As for double accounting of events - that's a bit harder to diagnose.
Maciej Obuchowski
Can you share the the job and events?
Also @PaweĆ LeszczyĆski
Also @PaweĆ LeszczyĆski
PaweĆ LeszczyĆski
1
Hi @Suraj Gupta,
Thanks for providing such a detailed description of the problem.
It is not expected behaviour, it's an issue. The events correspond to the same logical plan which for some reason lead to sending two OL events. Is it reproducible aka. does it occur each time? If yes, we please feel free to raise an issue for that.
We have added in recent months several tests to verify amount of OL events being generated but we haven't tested it that way with JDBC. BTW. will the same happen if you write your data
Thanks for providing such a detailed description of the problem.
It is not expected behaviour, it's an issue. The events correspond to the same logical plan which for some reason lead to sending two OL events. Is it reproducible aka. does it occur each time? If yes, we please feel free to raise an issue for that.
We have added in recent months several tests to verify amount of OL events being generated but we haven't tested it that way with JDBC. BTW. will the same happen if you write your data
df_transformed to a file (like parquet file) ?1Suraj Gupta
Thanks @PaweĆ LeszczyĆski, will confirm about writing to file and get back.
Suraj Gupta
And yes, the issue is reproducible. Will raise an issue for this.
PaweĆ LeszczyĆski
even if you write onto a file?
Suraj Gupta
Yes, even when I write to a parquet file.
PaweĆ LeszczyĆski
ok. i think i was able to reproduce it locally with https://github.com/OpenLineage/OpenLineage/pull/2103/files
Suraj Gupta
@PaweĆ LeszczyĆski I see that the PR is work in progress. Any rough estimate on when we can expect this fix to be released?
PaweĆ LeszczyĆski
@Suraj Gupta put a comment within your issue. it's a bug we need to solve but I cannot bring any estimates today.
Suraj Gupta
Thanks for update @PaweĆ LeszczyĆski, also please look into this comment. It might related and I'm not sure if expected behaviour.
Michael Robinson
@channel
This monthâs TSC meeting, open to all, is tomorrow: https://openlineage.slack.com/archives/C01CK9T7HKR/p1694113940400549
This monthâs TSC meeting, open to all, is tomorrow: https://openlineage.slack.com/archives/C01CK9T7HKR/p1694113940400549
â
1
Damien Hawes
Context:
We use Spark with YARN, running on Hadoop 2.x (I can't remember the exact minor version) with Hive support.
Problem:
I'm noticed that
For reference:
Running this:
This is an output:
We use Spark with YARN, running on Hadoop 2.x (I can't remember the exact minor version) with Hive support.
Problem:
I'm noticed that
CreateDataSourceAsSelectCommand objects are always transformed to an OutputDataset with a namespace value set to file - which is curious, because the inputs always have a (correct) namespace of hdfs://<name-node> - is this a known issue? A flaw with Apache Spark? A bug in the resolution logic?For reference:
public class CreateDataSourceTableCommandVisitor
- extends QueryPlanVisitor<CreateDataSourceTableCommand, OpenLineage.OutputDataset> {
-
- public CreateDataSourceTableCommandVisitor(OpenLineageContext context) {
- super(context);
- }
-
- @Override
- public List<OpenLineage.OutputDataset> apply(LogicalPlan x) {
- CreateDataSourceTableCommand command = (CreateDataSourceTableCommand) x;
- CatalogTable catalogTable = command.table();
-
- return Collections.singletonList(
- outputDataset()
- .getDataset(
- PathUtils.fromCatalogTable(catalogTable),
- catalogTable.schema(),
- OpenLineage.LifecycleStateChangeDatasetFacet.LifecycleStateChange.CREATE));
- }
-}Running this:
cat events.log | jq '{eventTime: .eventTime, eventType: .eventType, runId: .run.runId, jobNamespace: .job.namespace, jobName: .job.name, outputs: .outputs[] | {namespace: .namespace, name: .name}, inputs: .inputs[] | {namespace: .namespace, name: .name}}'This is an output:
{
- "eventTime": "2023-09-13T16:01:27.059Z",
- "eventType": "START",
- "runId": "bbbb5763-3615-46c0-95ca-1fc398c91d5d",
- "jobNamespace": "spark.cluster-1",
- "jobName": "ol_hadoop_test.execute_create_data_source_table_as_select_command.dhawes_db_ol_test_hadoop_tgt",
- "outputs": {
- "namespace": "file",
- "name": "/user/hive/warehouse/dhawes.db/ol_test_hadoop_tgt"
- },
- "inputs": {
- "namespace": "<hdfs://nn1>",
- "name": "/user/hive/warehouse/dhawes.db/ol_test_hadoop_src"
- }
-}đ1
PaweĆ LeszczyĆski
Seems like an issue on our side. Do you know how the source is read? What LogicalPlan leaf is used to read src? Would love to find how is this done differently
Damien Hawes
Hmm, I'll have to do explain plan to see what exactly it is.
However my sample job uses
which itself is created using
However my sample job uses
spark.sql("SELECT * FROM dhawes.ol_test_hadoop_src")which itself is created using
spark.sql("SELECT 1 AS id").write.format("orc").mode("overwrite").saveAsTable("dhawes.ol_test_hadoop_src")Damien Hawes
>>> spark.sql("SELECT * FROM dhawes.ol_test_hadoop_src").explain(True)
-== Parsed Logical Plan ==
-'Project [*]
-+- 'UnresolvedRelation `dhawes`.`ol_test_hadoop_src`
-
-== Analyzed Logical Plan ==
-id: int
-Project [id#3]
-+- SubqueryAlias `dhawes`.`ol_test_hadoop_src`
- +- Relation[id#3] orc
-
-== Optimized Logical Plan ==
-Relation[id#3] orc
-
-== Physical Plan ==
-*(1) FileScan orc dhawes.ol_test_hadoop_src[id#3] Batched: true, Format: ORC, Location: InMemoryFileIndex[<hdfs://nn1/user/hive/warehouse/dhawes.db/ol_test_hadoop_src>], PartitionFilters: [], PushedFilters: [], ReadSchema: struct<id:int>
tati
Hey everyone,
Any chance we could have a openlineage-integration-common 1.1.1 release with the following changes..?
âą https://github.com/OpenLineage/OpenLineage/pull/2106
âą https://github.com/OpenLineage/OpenLineage/pull/2108
Any chance we could have a openlineage-integration-common 1.1.1 release with the following changes..?
âą https://github.com/OpenLineage/OpenLineage/pull/2106
âą https://github.com/OpenLineage/OpenLineage/pull/2108
â6
tati
Specially the first PR is affecting users of the astronomer-cosmos library: https://github.com/astronomer/astronomer-cosmos/issues/533
Michael Robinson
1
Thanks @tati for requesting your first OpenLineage release! Three +1s from committers will authorize
1Michael Robinson
The release is authorized and will be initiated within two business days.
đ1
tati
Thanks a lot, @Michael Robinson!
Julien Le Dem
Per discussion in the OpenLineage sync today here is a very early strawman proposal for an OpenLineage registry that producers and consumers could be registered in.
Feedback or alternate proposals welcome
https://docs.google.com/document/d/1zIxKST59q3I6ws896M4GkUn7IsueLw8ejct5E-TR0vY/edit
Once this is sufficiently fleshed out, Iâll create an actual proposal on github
Feedback or alternate proposals welcome
https://docs.google.com/document/d/1zIxKST59q3I6ws896M4GkUn7IsueLw8ejct5E-TR0vY/edit
Once this is sufficiently fleshed out, Iâll create an actual proposal on github
đ1
Julien Le Dem
I have cleaned up the registry proposal.
https://docs.google.com/document/d/1zIxKST59q3I6ws896M4GkUn7IsueLw8ejct5E-TR0vY/edit
In particular:
âą I clarified that option 2 is preferred at this point.
âą I moved discussion notes to the bottom. they will go away at some point
âą Once it is stable, Iâll create a proposal with the preferred option.
âą we need a good proposal for the core facets prefix. My suggestion is to move core facets to
https://docs.google.com/document/d/1zIxKST59q3I6ws896M4GkUn7IsueLw8ejct5E-TR0vY/edit
In particular:
âą I clarified that option 2 is preferred at this point.
âą I moved discussion notes to the bottom. they will go away at some point
âą Once it is stable, Iâll create a proposal with the preferred option.
âą we need a good proposal for the core facets prefix. My suggestion is to move core facets to
core in the registry. The drawback is prefix would be inconsistent.Julien Le Dem
I have created a ticket to make this easier to find. Once I get more feedback Iâll turn it into a md file in the repo: https://docs.google.com/document/d/1zIxKST59q3I6ws896M4GkUn7IsueLw8ejct5E-TR0vY/edit#heading=h.enpbmvu7n8gu
https://github.com/OpenLineage/OpenLineage/issues/2161
https://github.com/OpenLineage/OpenLineage/issues/2161
Michael Robinson
@channel
Friendly reminder: the next OpenLineage meetup, our first in Toronto, is happening this coming Monday at 5 PM ET https://openlineage.slack.com/archives/C01CK9T7HKR/p1694441261486759
Friendly reminder: the next OpenLineage meetup, our first in Toronto, is happening this coming Monday at 5 PM ET https://openlineage.slack.com/archives/C01CK9T7HKR/p1694441261486759
đ1
Guntaka Jeevan Paul
@here we have dataproc operator getting called from a dag which submits a spark job, we wanted to maintain that continuity of parent job in the spark job so according to the documentation we can acheive that by using a macro called lineage_run_id that requires task and task_instance as the parameters. The problem we are facing is that our clientâs have 1000's of dags, so asking them to change this everywhere it is used is not feasible, so we thought of using the task_policy feature in the airflowâŠbut the problem is that task_policy gives you access to only the task/operator, but we donât have the access to the task instance..that is required as a parameter to the lineage_run_id function. Can anyone kindly help us on how should we go about this one
t1 = DataProcPySparkOperator(
- task_id=job_name,
- #required pyspark configuration,
- job_name=job_name,
- dataproc_pyspark_properties={
- 'spark.driver.extraJavaOptions':
- f"-javaagent:{jar}={os.environ.get('OPENLINEAGE_URL')}/api/v1/namespaces/{os.getenv('OPENLINEAGE_NAMESPACE', 'default')}/jobs/{job_name}/runs/{{{{macros.OpenLineagePlugin.lineage_run_id(task, task_instance)}}}}?api_key={os.environ.get('OPENLINEAGE_API_KEY')}"
- dag=dag)â1
Jakub DardziĆski
you don't need actual task instance to do that. you only should set additional argument as jinja template, same as above
Jakub DardziĆski
task_instance in this case is just part of string which is evaluated when jinja render happens
Guntaka Jeevan Paul
ohhâŠthen we could use the same example as above inside the task_policy to intercept the Operator and add the openlineage specific additions properties?
Jakub DardziĆski
correct, just remember not to override all properties, just add ol specific
Guntaka Jeevan Paul
yeah sureâŠthank you so much @Jakub DardziĆski, will try this out and keep you posted
đ1
Maciej Obuchowski
We want to automate setting those options at some point inside the operator itself
â1
Guntaka Jeevan Paul
@here is there a way by which we could add custom headers to openlineage client in airflow, i see that provision is there for spark integration via these properties spark.openlineage.transport.headers.xyz --> abcdef
Jakub DardziĆski
thereâs no out-of-the-box possibility to do that yet, youâre very welcome to create an issue in GitHub and maybe contribute as well! đ
Mars Lan
It doesn't seem like there's a way to override the OL endpoint from the default (
/api/v1/lineage) in Airflow? I tried setting the OPENLINEAGE_ENDPOINT environment to no avail. Based on this statement, it seems that only OPENLINEAGE_URL was used to construct HttpConfig ?Jakub DardziĆski
Thatâs correct. For now thereâs no way to configure the endpoint via env var. You can do that by using config file
Mars Lan
How do you do that in Airflow? Any particular reason for excluding endpoint override via env var? Happy to create a PR to fix that.
Jakub DardziĆski
historical I guess? go for the PR, of course đ
Terese Larsson
Hi! I'm in need of help with wrapping my head around OpenLineage. My team have the goal of collecting metadata from the Airflow operators GreatExpectationsOperator, PythonOperator, MsSqlOperator and BashOperator (for dbt). Where can I see the sourcecode for what is collected for each operator, and is there support for these in the new provider apache-airflow-providers-openlineage? I am super confused and feel lost in the docs. đ€Ż We are using MSSQL/ODBC to connect to our db, and this data does not seem to appear as datasets in Marquez, do I need to configure this? If so, HOW and WHERE? đ„Č
Happy for any help, big or small! đ
Happy for any help, big or small! đ
Jakub DardziĆski
thereâs no actual single source of what integrations are currently implemented in openlineage Airflow provider. Thatâs something we should work on so itâs more visible
Jakub DardziĆski
answering this quickly - GE & MS SQL are not currently implemented yet in the provider
Jakub DardziĆski
but I also invite you to contribute if youâre interested! đ
sarathch
Hi I need help in extracting OpenLineage for PostgresOperator in json format.
any suggestions or comments would be greatly appreciated
any suggestions or comments would be greatly appreciated
Maciej Obuchowski
If you're using Airflow 2.7, take a look at https://airflow.apache.org/docs/apache-airflow-providers-openlineage/stable/guides/user.html
â€ïž1
Maciej Obuchowski
If you use one of the lower versions, take a look here https://openlineage.io/docs/integrations/airflow/usage
sarathch
Maciej,
Thanks for sharing the link https://airflow.apache.org/docs/apache-airflow-providers-openlineage/stable/guides/user.html
this should address the issue
Thanks for sharing the link https://airflow.apache.org/docs/apache-airflow-providers-openlineage/stable/guides/user.html
this should address the issue
Juan Luis Cano RodrĂguez
congrats folks đ„ł https://lfaidata.foundation/blog/2023/09/20/lf-ai-data-foundation-announces-graduation-of-openlineage-project
đ11
đ1
â€ïž1
Michael Robinson
@channel
We released OpenLineage 1.2.2!
Added
âą Spark: publish the
âą Spark: capture and emit
Fixed
âą Common: support parsing dbt_project.yml without target-path
âą Proxy: fix Proxy chart
âą Python: fix serde filtering
âą Python: use non-deprecated
âą Spark: Improve RDDs on S3 integration.
âą Flink: prevent sending
âą Spark & Flink: Unify dataset naming from URI objects
âą Spark: Databricks improvements
Removed
âą SQL: remove sqlparser dependency from iface-java and iface-py
Thanks to all the contributors, including new contributors @tati, @xli-1026, and @d-m-h!
Release: https://github.com/OpenLineage/OpenLineage/releases/tag/1.2.2
Changelog: https://github.com/OpenLineage/OpenLineage/blob/main/CHANGELOG.md
Commit history: https://github.com/OpenLineage/OpenLineage/compare/1.1.0...1.2.2
Maven: https://oss.sonatype.org/#nexus-search;quick~openlineage
PyPI: https://pypi.org/project/openlineage-python/
We released OpenLineage 1.2.2!
Added
âą Spark: publish the
ProcessingEngineRunFacet as part of the normal operation of the OpenLineageSparkEventListener #2089 @d-m-hâą Spark: capture and emit
spark.databricks.clusterUsageTags.clusterAllTags variable from databricks environment #2099 @Anirudh181001Fixed
âą Common: support parsing dbt_project.yml without target-path
#2106 @tatianaâą Proxy: fix Proxy chart
#2091 @harelsâą Python: fix serde filtering
#2044 @xli-1026âą Python: use non-deprecated
apiKey if loading it from env variables @2029 @mobuchowskiâą Spark: Improve RDDs on S3 integration.
#2039 @pawel-big-lebowskiâą Flink: prevent sending
running events after job completes #2075 @pawel-big-lebowskiâą Spark & Flink: Unify dataset naming from URI objects
#2083 @pawel-big-lebowskiâą Spark: Databricks improvements
#2076 @pawel-big-lebowskiRemoved
âą SQL: remove sqlparser dependency from iface-java and iface-py
#2090 @JDarDagranThanks to all the contributors, including new contributors @tati, @xli-1026, and @d-m-h!
Release: https://github.com/OpenLineage/OpenLineage/releases/tag/1.2.2
Changelog: https://github.com/OpenLineage/OpenLineage/blob/main/CHANGELOG.md
Commit history: https://github.com/OpenLineage/OpenLineage/compare/1.1.0...1.2.2
Maven: https://oss.sonatype.org/#nexus-search;quick~openlineage
PyPI: https://pypi.org/project/openlineage-python/
đ„3
đ3
Michael Robinson
Hi Yevhenii, thank you for using OpenLineage. The DataHub integration is new to us, but perhaps the experts on Spark and Airflow know more. @PaweĆ LeszczyĆski @Maciej Obuchowski @Jakub DardziĆski
Maciej Obuchowski
@U05ST398BHT at Airflow Summit, Shirshanka Das from DataHub mentioned this as upcoming feature.
đ1
đŻ1
Suraj Gupta
Hi, we're using Custom Operators in airflow(2.5) and are planning to expose lineage via default extractors: https://openlineage.io/docs/integrations/airflow/default-extractors/
Question: Now if we upgrade our Airflow version to 2.7 in the future, would our code be backward compatible?
Since OpenLineage has now moved inside airflow and I think there is no concept of extractors in the latest version.
Question: Now if we upgrade our Airflow version to 2.7 in the future, would our code be backward compatible?
Since OpenLineage has now moved inside airflow and I think there is no concept of extractors in the latest version.
Suraj Gupta
Also, do we have any docs on how OL works with the latest airflow version? Few questions:
âą How is it replacing the concept of custom extractors and Manually Annotated Lineage in the latest version?
âą Do we have any examples of setting up the integration to emit input/output datasets for non supported Operators like PythonOperator?
âą How is it replacing the concept of custom extractors and Manually Annotated Lineage in the latest version?
âą Do we have any examples of setting up the integration to emit input/output datasets for non supported Operators like PythonOperator?
Jakub DardziĆski
> Question: Now if we upgrade our Airflow version to 2.7 in the future, would our code be backward compatible?
It will be compatible, âdefault extractorsâ is generally the same concept as weâre using in the 2.7 integration.
One thing that might be good to update is import paths, from
> âą Do we have any code samples/docs of setting up the integration to emit input/output datasets for non supported Operators like PythonOperator?
Our experience with that is currently lacking - this means, it works like in bare airflow, if you annotate your PythonOperator tasks with old Airflow lineage like in this doc.
We want to make this experience better - by doing few things
âą instrumenting hooks, then collecting lineage from them
âą integration with AIP-48 datasets
âą allowing to emit lineage collected inside Airflow task by other means, by providing core Airflow API for that
All those things require changing core Airflow in a couple of ways:
âą tracking which hooks were used during PythonOperator execution
âą just being able to emit datasets (airflow inlets/outlets) from inside of a task - they are now a static thing, so if you try that it does not work
âą providing better API for emitting that lineage, preferably based on OpenLineage itself rather than us having to convert that later.
As this requires core Airflow changes, it wonât be live until Airflow 2.8 at the earliest.
thanks to @Maciej Obuchowski for this response
It will be compatible, âdefault extractorsâ is generally the same concept as weâre using in the 2.7 integration.
One thing that might be good to update is import paths, from
openlineage.airflow to airflow.providers.openlineage but should work both ways> âą Do we have any code samples/docs of setting up the integration to emit input/output datasets for non supported Operators like PythonOperator?
Our experience with that is currently lacking - this means, it works like in bare airflow, if you annotate your PythonOperator tasks with old Airflow lineage like in this doc.
We want to make this experience better - by doing few things
âą instrumenting hooks, then collecting lineage from them
âą integration with AIP-48 datasets
âą allowing to emit lineage collected inside Airflow task by other means, by providing core Airflow API for that
All those things require changing core Airflow in a couple of ways:
âą tracking which hooks were used during PythonOperator execution
âą just being able to emit datasets (airflow inlets/outlets) from inside of a task - they are now a static thing, so if you try that it does not work
âą providing better API for emitting that lineage, preferably based on OpenLineage itself rather than us having to convert that later.
As this requires core Airflow changes, it wonât be live until Airflow 2.8 at the earliest.
thanks to @Maciej Obuchowski for this response
Jason Yip
I am using this accelerator that leverages OpenLineage on Databricks to publish lineage info to Purview, but it's using a rather old version of OpenLineage aka 0.18, anybody has tried it on a newer version of OpenLineage? I am facing some issues with the inputs and outputs for the same object is having different json
https://github.com/microsoft/Purview-ADB-Lineage-Solution-Accelerator/
https://github.com/microsoft/Purview-ADB-Lineage-Solution-Accelerator/
â
1
Jason Yip
I installed 1.2.2 on Databricks, followed the below init script: https://github.com/OpenLineage/OpenLineage/blob/main/integration/spark/databricks/open-lineage-init-script.sh
my cluster config looks like this:
spark.openlineage.version v1
spark.openlineage.namespace adb-5445974573286168.8#default
spark.openlineage.endpoint v1/lineage
spark.openlineage.url.param.code 8kZl0bo2TJfnbpFxBv-R2v7xBDj-PgWMol3yUm5iP1vaAzFu9kIZGg==
spark.openlineage.url https://f77b-50-35-69-138.ngrok-free.app
But it is not calling the API, it works fine with 0.18 version
my cluster config looks like this:
spark.openlineage.version v1
spark.openlineage.namespace adb-5445974573286168.8#default
spark.openlineage.endpoint v1/lineage
spark.openlineage.url.param.code 8kZl0bo2TJfnbpFxBv-R2v7xBDj-PgWMol3yUm5iP1vaAzFu9kIZGg==
spark.openlineage.url https://f77b-50-35-69-138.ngrok-free.app
But it is not calling the API, it works fine with 0.18 version
â
1
Jason Yip
I am attaching the log4j, there is no openlineagecontext
â
1
Jason Yip
this issue is resolved, solution can be found here: https://openlineage.slack.com/archives/C01CK9T7HKR/p1691592987038929
Harel Shein
We were all out at Airflow Summit last week, so apologies for the delayed response. Glad you were able to resolve the issue!
Sangeeta Mishra
@here I'm presently addressing a particular scenario that pertains to Openlineage authentication, specifically involving the use of an access key and secret.
I've implemented a custom token provider called AccessKeySecretKeyTokenProvider, which extends the TokenProvider class. This token provider communicates with another service, obtaining a token and an expiration time based on the provided access key, secret, and client ID.
My goal is to retain this token in a cache prior to its expiration, thereby eliminating the need for network calls to the third-party service. Is it possible without relying on an external caching system.
I've implemented a custom token provider called AccessKeySecretKeyTokenProvider, which extends the TokenProvider class. This token provider communicates with another service, obtaining a token and an expiration time based on the provided access key, secret, and client ID.
My goal is to retain this token in a cache prior to its expiration, thereby eliminating the need for network calls to the third-party service. Is it possible without relying on an external caching system.
Harel Shein
Hey @Sangeeta Mishra, Iâm not sure that I fully understand your question here. What do you mean by OpenLineage authentication?
What are you using to generate OL events? Whatâs your OL receiving backend?
What are you using to generate OL events? Whatâs your OL receiving backend?
Sangeeta Mishra
Hey @Harel Shein,
I wanted to clarify the previous message. I apologize for any confusion. When I mentioned "OpenLineage authentication," I was actually referring to the authentication process for the OpenLineage backend, specifically using HTTP transport. This involves using my custom token provider, which utilizes access keys and secrets for authentication. The OL backend is http based backend . I hope this clears things up!
I wanted to clarify the previous message. I apologize for any confusion. When I mentioned "OpenLineage authentication," I was actually referring to the authentication process for the OpenLineage backend, specifically using HTTP transport. This involves using my custom token provider, which utilizes access keys and secrets for authentication. The OL backend is http based backend . I hope this clears things up!
Harel Shein
Are you using Marquez?
Sangeeta Mishra
We are trying to leverage our own backend here.
Harel Shein
I see.. Iâm not sure the OpenLineage community could help here. Which webserver framework are you using?
Sangeeta Mishra
KTOR framework
Sangeeta Mishra
Our backend authentication operates based on either a pair of keys or a single bearer token, with a limited time of expiry. Hence, wanted to cache this information inside the token provider.
Sangeeta Mishra
Thank you
PaweĆ LeszczyĆski
@Sangeeta Mishra which openlineage client are you using: java or python?
Sangeeta Mishra
@PaweĆ LeszczyĆski I am using python client
Suraj Gupta
I'm using the Spark OpenLineage integration. In the
The Job performs a SQL insert into a MySQL table and I'm receiving the
I'm not sure what the size means here. Does this mean number of bytes inserted/updated?
Also, do we have any documentation for Spark specific Job and Run facets?
outputStatistics output dataset facet we receive rowCount and size.The Job performs a SQL insert into a MySQL table and I'm receiving the
size as 0.{
- "outputStatistics":
- {
- "_producer": "<https://github.com/OpenLineage/OpenLineage/tree/1.1.0/integration/spark>",
- "_schemaURL": "<https://openlineage.io/spec/facets/1-0-0/OutputStatisticsOutputDatasetFacet.json#/$defs/OutputStatisticsOutputDatasetFacet>",
- "rowCount": 1,
- "size": 0
- }
-}I'm not sure what the size means here. Does this mean number of bytes inserted/updated?
Also, do we have any documentation for Spark specific Job and Run facets?
PaweĆ LeszczyĆski
I am not sure it's stated in the doc. Here's the list of spark facets schemas: https://github.com/OpenLineage/OpenLineage/tree/main/integration/spark/shared/facets/spark/v1
Guntaka Jeevan Paul
@here In Airflow Integration we send across a lineage Event for Dag start and complete, but that is not the case with spark integrationâŠwe donât receive any event for the application start and complete in sparkâŠis this expected behaviour or am i missing something?
â1
PaweĆ LeszczyĆski
For spark we do send
start and complete for each spark action being run (single operation that causes spark processing being run). However, it is difficult for us to know if we're dealing with the last action within spark job or a spark script.PaweĆ LeszczyĆski
I think we need to look deeper into that as there is reoccuring need to capture such information
PaweĆ LeszczyĆski
and spark listener event has methods like
onApplicationStart and onApplicationEndGuntaka Jeevan Paul
We are using the SparkListener, which has a function called OnApplicationStart which gets called whenever a spark application starts, so i was thinking why cant we send one at start and simlarly at end as well
PaweĆ LeszczyĆski
additionally, we would like to have a concept of a parent run for a spark job which aggregates all actions run within a single spark job context
Guntaka Jeevan Paul
yeah exactly. the way that it works with airflow integration
PaweĆ LeszczyĆski
we do have an issue for that https://github.com/OpenLineage/OpenLineage/issues/2105
PaweĆ LeszczyĆski
what you can is: come to our monthly Openlineage open meetings and raise that issue and convince the community about its importance
Guntaka Jeevan Paul
yeah sure would love to do thatâŠhow can i join them, will that be posted here in this slack channel?
Michael Robinson
Meetup recap: Toronto Meetup @ Airflow Summit, September 18, 2023
It was great to see so many members of our community at this event! I counted 32 total attendees, with all but a handful being first-timers.
Topics included:
âą Presentation on the history, architecture and roadmap of the project by @Julien Le Dem and @Harel Shein
âą Discussion of OpenLineage support in Marquez by @Willy Lulciuc
âą Presentation by Ye Liu and Ivan Perepelitca from Metaphor, the social platform for data, about their integration
âą Presentation by @PaweĆ LeszczyĆski about the Spark integration
âą Presentation by @Maciej Obuchowski about the Apache Airflow Provider
Thanks to all the presenters and attendees with a shout out to @Harel Shein for the help with organizing and day-of logistics, @Jakub DardziĆski for the help with set up/clean up, and @Sheeri Cabral (Collibra) for the crucial assist with the signup sheet.
This was our first meetup in Toronto, and we learned some valuable lessons about planning events in new cities â the first and foremost being to ask for a pic of the building! đ But it seemed like folks were undeterred, and the space itself lived up to expectations.
For a recording and clips from the meetup, head over to our YouTube channel.
Upcoming events:
âą October 5th in San Francisco: Marquez Meetup @ Astronomer (sign up here)
âą November: Warsaw meetup (details, date TBA)
âą January: London meetup (details, date TBA)
Are you interested in hosting or co-hosting an OpenLineage or Marquez meetup? DM me!
It was great to see so many members of our community at this event! I counted 32 total attendees, with all but a handful being first-timers.
Topics included:
âą Presentation on the history, architecture and roadmap of the project by @Julien Le Dem and @Harel Shein
âą Discussion of OpenLineage support in Marquez by @Willy Lulciuc
âą Presentation by Ye Liu and Ivan Perepelitca from Metaphor, the social platform for data, about their integration
âą Presentation by @PaweĆ LeszczyĆski about the Spark integration
âą Presentation by @Maciej Obuchowski about the Apache Airflow Provider
Thanks to all the presenters and attendees with a shout out to @Harel Shein for the help with organizing and day-of logistics, @Jakub DardziĆski for the help with set up/clean up, and @Sheeri Cabral (Collibra) for the crucial assist with the signup sheet.
This was our first meetup in Toronto, and we learned some valuable lessons about planning events in new cities â the first and foremost being to ask for a pic of the building! đ But it seemed like folks were undeterred, and the space itself lived up to expectations.
For a recording and clips from the meetup, head over to our YouTube channel.
Upcoming events:
âą October 5th in San Francisco: Marquez Meetup @ Astronomer (sign up here)
âą November: Warsaw meetup (details, date TBA)
âą January: London meetup (details, date TBA)
Are you interested in hosting or co-hosting an OpenLineage or Marquez meetup? DM me!
đ3
â€ïž6
đ2
đ
1
â
1
Michael Robinson
A few more pics:
Damien Hawes
Hi folks, am I correct in my observations that the Spark integration does not generate inputs and outputs for Kafka-to-Kafka pipelines?
EDIT: Removed the crazy wall of text. Relevant GitHub issue is here.
EDIT: Removed the crazy wall of text. Relevant GitHub issue is here.
đ1
PaweĆ LeszczyĆski
responded within the issue
Erik Alfthan
Hello community
First time poster - bear with me :)
I am looking to make a minor PR on the airflow integration (fixing github #2130), and the code change is easy enough, but I fail to install the python environment. I have tried the simple ones
or
but they both fail on
(which I think is an unreleased version in the git project)
How would I go about to install the requirements?
//Erik
PS. Sorry for posting this in general if there is a specific integration or contribution channel - I didnt find a better channel
First time poster - bear with me :)
I am looking to make a minor PR on the airflow integration (fixing github #2130), and the code change is easy enough, but I fail to install the python environment. I have tried the simple ones
OpenLineage/integration/airflow > pip install -e .or
OpenLineage/integration/airflow > pip install -r dev-requirements.txtbut they both fail on
ERROR: No matching distribution found for openlineage-sql==1.3.0(which I think is an unreleased version in the git project)
How would I go about to install the requirements?
//Erik
PS. Sorry for posting this in general if there is a specific integration or contribution channel - I didnt find a better channel
PaweĆ LeszczyĆski
Hi @Erik Alfthan, the channel is totally OK. I am not airflow integration expert, but what it looks to me, you're missing openlineage-sql library, which is a rust library used to extract lineage from sql queries. This is how we do that in circle ci:
https://app.circleci.com/pipelines/github/OpenLineage/OpenLineage/8080/workflows/aba53369-836c-48f5-a2dd-51bc0740a31c/jobs/140113
and subproject page with build instructions: https://github.com/OpenLineage/OpenLineage/tree/main/integration/sql
https://app.circleci.com/pipelines/github/OpenLineage/OpenLineage/8080/workflows/aba53369-836c-48f5-a2dd-51bc0740a31c/jobs/140113
and subproject page with build instructions: https://github.com/OpenLineage/OpenLineage/tree/main/integration/sql
Erik Alfthan
Ok, so I go and "manually" build the internal dependency so that it becomes available in the pip cache?
I was hoping for something more automagical, but that should work
I was hoping for something more automagical, but that should work
PaweĆ LeszczyĆski
I think so. @Jakub DardziĆski am I right?
Jakub DardziĆski
https://openlineage.io/docs/development/developing/python/setup
thereâs a guide how to setup the dev environment
> Typically, you first need to build
This might be somewhat exposed more in GitHub repository README as well
thereâs a guide how to setup the dev environment
> Typically, you first need to build
openlineage-sql locally (see README). After each release you have to repeat this step in order to bump local version of the package.This might be somewhat exposed more in GitHub repository README as well
Erik Alfthan
It didnt find the wheel in the cache, but if I used the line in the sql/README.md
It is installed and thus skipped/passed when pip later checks if it needs to be installed.
Now I have a second issue because it is expecting me to have mysqlclient-2.2.0 which seems to need a binary
and
I am on Ubuntu 22.04 in WSL2. Should I go to apt and grab me a mysql client?
pip install openlineage-sql --no-index --find-links ../target/wheels --force-reinstallIt is installed and thus skipped/passed when pip later checks if it needs to be installed.
Now I have a second issue because it is expecting me to have mysqlclient-2.2.0 which seems to need a binary
Command 'pkg-config --exists mysqlclient' returned non-zero exit status 127and
Command 'pkg-config --exists mariadb' returned non-zero exit status 127I am on Ubuntu 22.04 in WSL2. Should I go to apt and grab me a mysql client?
Jakub DardziĆski
> It didnt find the wheel in the cache, but if I used the line in the sql/README.md
>
> It is installed and thus skipped/passed when pip later checks if it needs to be installed.
Thatâs actually expected. You should build new wheel locally and then install it.
> Now I have a second issue because it is expecting me to have mysqlclient-2.2.0 which seems to need a binary
>
> and
>
> I am on Ubuntu 22.04 in WSL2. Should I go to apt and grab me a mysql client?
Weâve left some system specific configuration, e.g. mysqlclient, to users as itâs a bit aside from OpenLineage and more of general development task.
probably
should work
>
pip install openlineage-sql --no-index --find-links ../target/wheels --force-reinstall> It is installed and thus skipped/passed when pip later checks if it needs to be installed.
Thatâs actually expected. You should build new wheel locally and then install it.
> Now I have a second issue because it is expecting me to have mysqlclient-2.2.0 which seems to need a binary
>
Command 'pkg-config --exists mysqlclient' returned non-zero exit status 127> and
>
Command 'pkg-config --exists mariadb' returned non-zero exit status 127> I am on Ubuntu 22.04 in WSL2. Should I go to apt and grab me a mysql client?
Weâve left some system specific configuration, e.g. mysqlclient, to users as itâs a bit aside from OpenLineage and more of general development task.
probably
sudo apt-get install python3-dev default-libmysqlclient-dev build-essential should work
Erik Alfthan
I just realized that I should probably skip setting up my wsl and just run the tests in the docker setup you prepared
Jakub DardziĆski
You could do that as well but if you want to test your changes vs many Airflow versions that wouldnât be possible I think (run them with tox btw)
Erik Alfthan
This is starting to feel like a rabbit hole đ
When I run tox, I get a lot of build errors
âą client needs to be built
âą sql needs to be built to a different target than its readme says
âą a lot of builds fail on cython_sources
When I run tox, I get a lot of build errors
âą client needs to be built
âą sql needs to be built to a different target than its readme says
âą a lot of builds fail on cython_sources
Jakub DardziĆski
would you like to share some exact log lines? Iâve never seen such errors, they probably are system specific
Erik Alfthan
Getting requirements to build wheel did not run successfully.â exit code: 1â°â> [62 lines of output]/tmp/pip-build-env-q1pay0xo/overlay/lib/python3.10/site-packages/setuptools/config/setupcfg.py:293: _DeprecatedConfig: Deprecated config in setup.cfg`
- !!
-
- **
- The licensefile parameter is deprecated, use licensefiles instead.
-
- By 2023-Oct-30, you need to update your project and remove deprecated calls
- or your builds will no longer be supported.
-
- See https://setuptools.pypa.io/en/latest/userguide/declarative_config.html for details.
- **
-
- !!
- parsed = self.parsers.get(option_name, lambda x: x)(value)
- running egg_info
- writing lib3/PyYAML.egg-info/PKG-INFO
- writing dependency_links to lib3/PyYAML.egg-info/dependency_links.txt
- writing top-level names to lib3/PyYAML.egg-info/top_level.txt
- Traceback (most recent call last):
- File "/home/obr_erikal/projects/OpenLineage/integration/airflow/.tox/py3-airflow-2.1.4/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
- main()
- File "/home/obr_erikal/projects/OpenLineage/integration/airflow/.tox/py3-airflow-2.1.4/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
- json_out['return_val'] = hook(**hook_input['kwargs'])
- File "/home/obr_erikal/projects/OpenLineage/integration/airflow/.tox/py3-airflow-2.1.4/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 118, in get_requires_for_build_wheel
- return hook(config_settings)
- File "/tmp/pip-build-env-q1pay0xo/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 355, in get_requires_for_build_wheel
- return self._get_build_requires(config_settings, requirements=['wheel'])
- File "/tmp/pip-build-env-q1pay0xo/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 325, in _get_build_requires
- self.run_setup()
- File "/tmp/pip-build-env-q1pay0xo/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 341, in run_setup
- exec(code, locals())
- File "<string>", line 271, in <module>
- File "/tmp/pip-build-env-q1pay0xo/overlay/lib/python3.10/site-packages/setuptools/__init.py", line 103, in setup
- return distutils.core.setup(**attrs)
- File "/tmp/pip-build-env-q1pay0xo/overlay/lib/python3.10/site-packages/setuptools/distutils/core.py", line 185, in setup
- return runcommands(dist)
- File "/tmp/pip-build-env-q1pay0xo/overlay/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 201, in run_commands
- dist.run_commands()
- File "/tmp/pip-build-env-q1pay0xo/overlay/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 969, in run_commands
- self.run_command(cmd)
- File "/tmp/pip-build-env-q1pay0xo/overlay/lib/python3.10/site-packages/setuptools/dist.py", line 989, in run_command
- super().run_command(command)
- File "/tmp/pip-build-env-q1pay0xo/overlay/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
- cmd_obj.run()
- File "/tmp/pip-build-env-q1pay0xo/overlay/lib/python3.10/site-packages/setuptools/command/egg_info.py", line 318, in run
- self.find_sources()
- File "/tmp/pip-build-env-q1pay0xo/overlay/lib/python3.10/site-packages/setuptools/command/egg_info.py", line 326, in find_sources
- mm.run()
- File "/tmp/pip-build-env-q1pay0xo/overlay/lib/python3.10/site-packages/setuptools/command/egg_info.py", line 548, in run
- self.add_defaults()
- File "/tmp/pip-build-env-q1pay0xo/overlay/lib/python3.10/site-packages/setuptools/command/egg_info.py", line 586, in add_defaults
- sdist.add_defaults(self)
- File "/tmp/pip-build-env-q1pay0xo/overlay/lib/python3.10/site-packages/setuptools/command/sdist.py", line 113, in add_defaults
- super().add_defaults()
- File "/tmp/pip-build-env-q1pay0xo/overlay/lib/python3.10/site-packages/setuptools/_distutils/command/sdist.py", line 251, in add_defaults
- self._add_defaults_ext()
- File "/tmp/pip-build-env-q1pay0xo/overlay/lib/python3.10/site-packages/setuptools/_distutils/command/sdist.py", line 336, in _add_defaults_ext
- self.filelist.extend(build_ext.get_source_files())
- File "<string>", line 201, in get_source_files
- File "/tmp/pip-build-env-q1pay0xo/overlay/lib/python3.10/site-packages/setuptools/_distutils/cmd.py", line 107, in __getattr
- raise AttributeError(attr)
- AttributeError: cython_sources
- [end of output]
-
-note: This error originates from a subprocess, and is likely not a problem with pip.
-py3-airflow-2.1.4: exit 1 (7.85 seconds) /home/obr_erikal/projects/OpenLineage/integration/airflow> python -m pip install --find-links target/wheels/ --find-links ../sql/iface-py/target/wheels --use-deprecated=legacy-resolver --constraint=https://raw.githubusercontent.com/apache/airflow/constraints-2.1.4/constraints-3.8.txt apache-airflow==2.1.4 'mypy>=0.9.6' pytest pytest-mock -r dev-requirements.txt pid=368621
-py3-airflow-2.1.4: FAIL â in 7.92 seconds`Erik Alfthan
Then, for the actual error in my PR: Evidently you are not using isort, so what linter/fixer should I use for imports?
Jakub DardziĆski
for the error - I think thereâs a mistake in the docs. Could you please run
maturin build --out target/wheels as a temp solution?đ1
Jakub DardziĆski
weâre using
ruff , tox runs it as one of commandsErik Alfthan
Not in the airflow folder?
OpenLineage/integration/airflow$ maturin build --out target/wheels :boom: maturin failedCaused by: pyproject.toml at /home/obr_erikal/projects/OpenLineage/integration/airflow/pyproject.toml is invalidCaused by: TOML parse error at line 1, column 1|1 | [tool.ruff]| ^missing field build-system``Jakub DardziĆski
I meant change here https://github.com/OpenLineage/OpenLineage/blob/main/integration/sql/README.md
so
becomes
tox runs
but it should be
actually and Iâm posting PR to fix that
so
cd iface-py
-python -m pip install maturin
-maturin build --out ../target/wheelsbecomes
cd iface-py
-python -m pip install maturin
-maturin build --out target/wheelstox runs
install_command = python -m pip install {opts} --find-links target/wheels/ \
- --find-links ../sql/iface-py/target/wheelsbut it should be
install_command = python -m pip install {opts} --find-links target/wheels/ \
- --find-links ../sql/target/wheelsactually and Iâm posting PR to fix that
Erik Alfthan
yes, that part I actually worked out myself, but the cython_sources error I fail to understand cause. I have python3-dev installed on WSL Ubuntu with python version 3.10.12 in a virtualenv. Anything in that that could cause issues?
Jakub DardziĆski
looks like it has something to do with latest release of Cython?
pip install "Cython<3" maybe solves the issue?Erik Alfthan
I didnt have any cython before the install. Also no change. Could it be some update to setuptools itself? seems like the depreciation notice and the error is coming from inside setuptools
Erik Alfthan
(I.e. I tried the
pip install "Cython<3" command without any change in the output )Erik Alfthan
Applying ruff lint on the converter.py file fixed the issue on the PR though so unless you have any feedback on the change itself, I will set it up on my own computer later instead (right now doing changes on behalf of a client on the clients computer)
If the issue persists on my own computer, I'll dig a bit further
If the issue persists on my own computer, I'll dig a bit further
Jakub DardziĆski
Itâs a bit hard for me to find the root cause as I cannot reproduce this locally and CI works fine as well
Erik Alfthan
Yeah, I am thinking that if I run into the same problem "at home", I might find it worthwhile to understand the issue. Right now, the client only wants the fix.
đ1
Erik Alfthan
Is there an official release cycle?
or more specific, given that the PRs are approved, how soon can they reach openlineage-dbt and apache-airflow-providers-openlineage ?
or more specific, given that the PRs are approved, how soon can they reach openlineage-dbt and apache-airflow-providers-openlineage ?
Jakub DardziĆski
we need to differentiate some things:
1. OpenLineage repository:
a. dbt integration - this is the only place where it is maintained
b. Airflow integration - here we only keep backwards compatibility but generally speaking starting from Airflow 2.7+ we would like to do all the job in Airflow repo as OL Airflow provider
2. Airflow repository - thereâs only Airflow Openlineage provider compatible (and works best) with Airflow 2.7+
we have control over releases (obviously) in OL repo - itâs monthly cycle so beginning next week that should happen. Thereâs also a possibility to ask for ad-hoc release in #general slack channel and with approvals of committers the new version is also released
For Airflow providers - the cycle is monthly as well
1. OpenLineage repository:
a. dbt integration - this is the only place where it is maintained
b. Airflow integration - here we only keep backwards compatibility but generally speaking starting from Airflow 2.7+ we would like to do all the job in Airflow repo as OL Airflow provider
2. Airflow repository - thereâs only Airflow Openlineage provider compatible (and works best) with Airflow 2.7+
we have control over releases (obviously) in OL repo - itâs monthly cycle so beginning next week that should happen. Thereâs also a possibility to ask for ad-hoc release in #general slack channel and with approvals of committers the new version is also released
For Airflow providers - the cycle is monthly as well
Jakub DardziĆski
itâs a bit complex for this split but needed temporarily
Erik Alfthan
oh, I did the fix in the wrong place! The client is on airflow 2.7 and is using the provider. Is it syncing?
Jakub DardziĆski
itâs not, two separate places and we havenât even added the whole thing with converting old lineage objects to OL specific
editing, thatâs not true
editing, thatâs not true
Jakub DardziĆski
Jakub DardziĆski
sorry I did not mention this earlier. we definitely need to add some guidance how to proceed with contributions to OL and Airflow OL provider
Erik Alfthan
anyway, the dbt fix is the blocking issue, so if that parts comes next week, there is no real urgency in getting the columns. It is a nice to have for our ingest parquet files.
Jakub DardziĆski
may I ask if you use some custom operator / python operator there?
Erik Alfthan
yeah, taskflow with inlets/outlets
Erik Alfthan
so we extract from sources and use pyarrow to create parquet files in storage that an mssql-server can use as external tables
Jakub DardziĆski
awesome đ
we have plans to integrate more with Python operator as well but not earlier than in Airflow 2.8
we have plans to integrate more with Python operator as well but not earlier than in Airflow 2.8
Erik Alfthan
I guess writing a generic extractor for the python operator is quite hard, but if you could support some inlet/outlet type for tabular fileformat / their python libraries like pyarrow or maybe even pandas and document it, I think a lot of people would understand how to use them
â1
Michael Robinson
Are you located in the Brussels area or within commutable distance? Interested in attending a meetup between October 16-20? If so, please DM @Sheeri Cabral (Collibra) or myself. TIA
â€ïž1
Michael Robinson
@channel
Hello all, Iâd like to open a vote to release OpenLineage 1.3.0, including:
âą support for Spark 3.5 in the Spark integration
âą scheme preservation bug fix in the Spark integration
âą find-links path in tox bug in the Airflow integration fix
âą more graceful logging when no OL provider is installed in the Airflow integration
âą columns as schema facet for airflow.lineage.Table addition
âą SQLSERVER to supported dbt profile types addition
Three +1s from committers will authorize. Thanks in advance.
Hello all, Iâd like to open a vote to release OpenLineage 1.3.0, including:
âą support for Spark 3.5 in the Spark integration
âą scheme preservation bug fix in the Spark integration
âą find-links path in tox bug in the Airflow integration fix
âą more graceful logging when no OL provider is installed in the Airflow integration
âą columns as schema facet for airflow.lineage.Table addition
âą SQLSERVER to supported dbt profile types addition
Three +1s from committers will authorize. Thanks in advance.
đ3
đ2
â4
Michael Robinson
Thanks all. The release is authorized and will be initiated within 2 business days.
Jason Yip
looking forward to that, I am seeing inconsistent results in Databricks for Spark 3.4+, sometimes there's no inputs / outputs, hope that is fixed?
Harel Shein
@Jason Yip if it isnât fixed for you, would love it if you could open up an issue that will allow us to reproduce and fix
đ1
Jason Yip
@Harel Shein the issue still exists -> Spark 3.4 and above, including 3.5, saveAsTable and create table won't have inputs and outputs in Databricks
Jason Yip
and of course this issue still exists
Harel Shein
thanks for posting, weâll continue looking into this.. if you find any clues that might help, please let us know.
Jason Yip
is there any instructions on how to hook up a debugger to OL?
Harel Shein
@PaweĆ LeszczyĆski has been working on adding a debug facet, but more suggestions are more than welcome!
Jason Yip
@PaweĆ LeszczyĆski do you have a build for the PR? Appreciated!
Harel Shein
weâll ask for a release once itâs reviewed and merged
Michael Robinson
@channel
The September issue of OpenLineage News is here! This issue covers the big news about OpenLineage coming out of Airflow Summit, progress on the Airflow Provider, highlights from our meetup in Toronto, and much more.
To get the newsletter directly in your inbox each month, sign up here.
The September issue of OpenLineage News is here! This issue covers the big news about OpenLineage coming out of Airflow Summit, progress on the Airflow Provider, highlights from our meetup in Toronto, and much more.
To get the newsletter directly in your inbox each month, sign up here.
đŠ2
đ„3
Damien Hawes
Hi folks - I'm wondering if its just me, but does
io.openlineage:openlineage-sql-java:1.2.2 ship with the arm64.dylib binary? When i try and run code that uses the Java package on an Apple M1, the binary isn't found, The workaround is to checkout 1.2.2 and then build and publish it locally.PaweĆ LeszczyĆski
Not sure if I follow your question. Whenever OL is released, there is a script new-version.sh - https://github.com/OpenLineage/OpenLineage/blob/main/new-version.sh being run and modify the codebase.
So, If you pull the code, it contains OL version that has not been released yet and in case of dependencies, one need to build them on their own.
For example, here https://github.com/OpenLineage/OpenLineage/tree/main/integration/spark#preparation Preparation section describes how to build openlineage-java and openlineage-sql in order to build openlineage-spark.
So, If you pull the code, it contains OL version that has not been released yet and in case of dependencies, one need to build them on their own.
For example, here https://github.com/OpenLineage/OpenLineage/tree/main/integration/spark#preparation Preparation section describes how to build openlineage-java and openlineage-sql in order to build openlineage-spark.
Damien Hawes
Hmm. Let's elaborate my use case a bit.
We run Apache Hive on-premise. Hive provides query execution hooks for pre-query, post-query, and I think failed query.
Any way, as part of the hook, you're given the query string.
So I, naturally, tried to pass the query string into
I was using
I then stepped through the code and noticed that it was looking for the
We run Apache Hive on-premise. Hive provides query execution hooks for pre-query, post-query, and I think failed query.
Any way, as part of the hook, you're given the query string.
So I, naturally, tried to pass the query string into
OpenLineageSql.parse(Collections.singletonList(hookContext.getQueryPlan().getQueryStr()), "hive") in order to test this out.I was using
openlineage-sql-java:1.2.2 at that time, and no matter what query string I gave it, nothing was returned.I then stepped through the code and noticed that it was looking for the
arm64 lib, and I noticed that that package (downloaded from maven central) lacked that particular native binary.Damien Hawes
I hope that helps.
đ1
PaweĆ LeszczyĆski
I get in now. In Circle CI we do have 3 build steps:
but no mac m1. I think at that time circle CI did not have a proper resource class in free plan. Additionally, @Maciej Obuchowski would prefer to migrate this to github actions as he claims this can be achieved there in a cleaner way (https://github.com/OpenLineage/OpenLineage/issues/1624).
Feel free to create an issue for this. Others would be able to upvote it in case they have similar experience.
- build-integration-sql-x86
- - build-integration-sql-arm
- - build-integration-sql-macosbut no mac m1. I think at that time circle CI did not have a proper resource class in free plan. Additionally, @Maciej Obuchowski would prefer to migrate this to github actions as he claims this can be achieved there in a cleaner way (https://github.com/OpenLineage/OpenLineage/issues/1624).
Feel free to create an issue for this. Others would be able to upvote it in case they have similar experience.
Maciej Obuchowski
It doesn't have the free resource class still đ
We're blocked on that unfortunately. Other solution would be to migrate to GH actions, where most of our solution could be replaced by something like that https://github.com/PyO3/maturin-action
We're blocked on that unfortunately. Other solution would be to migrate to GH actions, where most of our solution could be replaced by something like that https://github.com/PyO3/maturin-action
Michael Robinson
@channel
We released OpenLineage 1.3.1!
Added:
âą Airflow: add some basic stats to the Airflow integration
âą Airflow: add columns as schema facet for
âą DBT: add SQLSERVER to supported dbt profile types
âą Spark: support for latest 3.5
Fixed:
âą Airflow: fix find-links path in tox
âą Airflow: add more graceful logging when no OpenLineage provider installed
âą Spark: fix bug in PathUtilsâ
Thanks to all the contributors, including new contributor @Erik Alfthan!
Release: https://github.com/OpenLineage/OpenLineage/releases/tag/1.3.1
Changelog: https://github.com/OpenLineage/OpenLineage/blob/main/CHANGELOG.md
Commit history: https://github.com/OpenLineage/OpenLineage/compare/1.2.2...1.3.1
Maven: https://oss.sonatype.org/#nexus-search;quick~openlineage
PyPI: https://pypi.org/project/openlineage-python/
We released OpenLineage 1.3.1!
Added:
âą Airflow: add some basic stats to the Airflow integration
#1845 @harelsâą Airflow: add columns as schema facet for
airflow.lineage.Table (if defined) #2138 @erikalfthanâą DBT: add SQLSERVER to supported dbt profile types
#2136 @erikalfthanâą Spark: support for latest 3.5
#2118 @pawel-big-lebowskiFixed:
âą Airflow: fix find-links path in tox
#2139 @JDarDagranâą Airflow: add more graceful logging when no OpenLineage provider installed
#2141 @JDarDagranâą Spark: fix bug in PathUtilsâ
prepareDatasetIdentifierFromDefaultTablePath (CatalogTable) to correctly preserve scheme from CatalogTableâs location #2142 @d-m-hThanks to all the contributors, including new contributor @Erik Alfthan!
Release: https://github.com/OpenLineage/OpenLineage/releases/tag/1.3.1
Changelog: https://github.com/OpenLineage/OpenLineage/blob/main/CHANGELOG.md
Commit history: https://github.com/OpenLineage/OpenLineage/compare/1.2.2...1.3.1
Maven: https://oss.sonatype.org/#nexus-search;quick~openlineage
PyPI: https://pypi.org/project/openlineage-python/
đ4
đ1
Mars Lan
Any chance we can do a 1.3.2 soonish to include https://github.com/OpenLineage/OpenLineage/pull/2151 instead of waiting for the next monthly release?
Matthew Paras
Hey everyone - does anyone have a good mechanism for alerting on issues with open lineage? For example, maybe alerting when an event times out - perhaps to prometheus or some other kind of generic endpoint? Not sure the best approach here (if the meta inf extension would be able to achieve it)
PaweĆ LeszczyĆski
That's a great usecase for OpenLineage. Unfortunately, we don't have any doc or recomendation on that.
I would try using FluentD proxy we have (https://github.com/OpenLineage/OpenLineage/tree/main/proxy/fluentd) to copy event stream (alerting is just one of usecases for lineage events) and write fluentd plugin to send it asynchronously further to alerting service like PagerDuty.
It looks cool to me but I never had enough time to test this approach.
I would try using FluentD proxy we have (https://github.com/OpenLineage/OpenLineage/tree/main/proxy/fluentd) to copy event stream (alerting is just one of usecases for lineage events) and write fluentd plugin to send it asynchronously further to alerting service like PagerDuty.
It looks cool to me but I never had enough time to test this approach.
đ1
Julien Le Dem
I have cleaned up the registry proposal.
https://docs.google.com/document/d/1zIxKST59q3I6ws896M4GkUn7IsueLw8ejct5E-TR0vY/edit
In particular:
âą I clarified that option 2 is preferred at this point.
âą I moved discussion notes to the bottom. they will go away at some point
âą Once it is stable, Iâll create a proposal with the preferred option.
âą we need a good proposal for the core facets prefix. My suggestion is to move core facets to
https://docs.google.com/document/d/1zIxKST59q3I6ws896M4GkUn7IsueLw8ejct5E-TR0vY/edit
In particular:
âą I clarified that option 2 is preferred at this point.
âą I moved discussion notes to the bottom. they will go away at some point
âą Once it is stable, Iâll create a proposal with the preferred option.
âą we need a good proposal for the core facets prefix. My suggestion is to move core facets to
core in the registry. The drawback is prefix would be inconsistent.Michael Robinson
@channel
This monthâs TSC meeting is next Thursday the 12th at 10am PT. On the tentative agenda:
âą announcements
âą recent releases
âą Airflow Summit recap
âą tutorial: migrating to the Airflow Provider
âą discussion topic: observability for OpenLineage/Marquez
âą open discussion
âą more (TBA)
More info and the meeting link can be found on the website. All are welcome! Do you have a discussion topic, use case or integration youâd like to demo? DM me to be added to the agenda.
This monthâs TSC meeting is next Thursday the 12th at 10am PT. On the tentative agenda:
âą announcements
âą recent releases
âą Airflow Summit recap
âą tutorial: migrating to the Airflow Provider
âą discussion topic: observability for OpenLineage/Marquez
âą open discussion
âą more (TBA)
More info and the meeting link can be found on the website. All are welcome! Do you have a discussion topic, use case or integration youâd like to demo? DM me to be added to the agenda.
đ3
Julien Le Dem
I have created a ticket to make this easier to find. Once I get more feedback Iâll turn it into a md file in the repo: https://docs.google.com/document/d/1zIxKST59q3I6ws896M4GkUn7IsueLw8ejct5E-TR0vY/edit#heading=h.enpbmvu7n8gu
https://github.com/OpenLineage/OpenLineage/issues/2161
https://github.com/OpenLineage/OpenLineage/issues/2161
Julien Le Dem
The Marquez meetup in San Francisco is happening right now!
https://www.meetup.com/meetup-group-bnfqymxe/events/295444209/?utm_medium=referral&utm_campaign=share-btn_savedevents_share_modal&utm_source=link
https://www.meetup.com/meetup-group-bnfqymxe/events/295444209/?utm_medium=referral&utm_campaign=share-btn_savedevents_share_modal&utm_source=link
đ2
Mars Lan
@Michael Robinson can we cut a new release to include this change?
âą https://github.com/OpenLineage/OpenLineage/pull/2151
âą https://github.com/OpenLineage/OpenLineage/pull/2151
â5
Michael Robinson
Thanks for requesting a release, @Mars Lan. It has been approved and will be initiated within 2 business days of next Monday.
đ1
Guntaka Jeevan Paul
@here I am trying out the openlineage integration of spark on databricks. There is no event getting emitted from Openlineage, I see logs saying OpenLineage Event Skipped. I am attaching the Notebook that i am trying to run and the cluster logs. Kindly can someone help me on this
Jason Yip
from my experience, it will only work on Spark 3.3.x or below, aka Runtime 12.2 or below. Anything above the events will show up once in a blue moon
Guntaka Jeevan Paul
ohh, thanks for the information @Jason Yip, I am trying out with 13.3 Databricks Version and Spark 3.4.1, will try using a below version as you suggested. Any issue tracking this bug @Jason Yip
Guntaka Jeevan Paul
tried with databricks 12.2 --> spark 3.3.2, still the same behaviour no event getting emitted
Jason Yip
you can do 11.3, its the most stable one I know
Guntaka Jeevan Paul
sure, let me try that out
Guntaka Jeevan Paul
still the same problemâŠthe jar that i am using is the latest openlineage-spark-1.3.1.jar, do you think that can be the problem
Guntaka Jeevan Paul
tried with openlineage-spark-1.2.2.jar, still the same issue, seems like they are skipping some events
Jason Yip
Probably not all events will be captured, I have only tested create tables and jobs
PaweĆ LeszczyĆski
Hi @Guntaka Jeevan Paul, how did you configure openlineage and what is your job doing?
We do have a bunch of integration tests on Databricks platform available here and they're passing on databricks runtime
Could you also try running code same as our test does (this one)? If you run it and see OL events, this will make us sure your config is OK and we can continue further debug.
Looking at your spark script: could you save your dataset and see if you still don't see any events?
We do have a bunch of integration tests on Databricks platform available here and they're passing on databricks runtime
13.0.x-scala2.12.Could you also try running code same as our test does (this one)? If you run it and see OL events, this will make us sure your config is OK and we can continue further debug.
Looking at your spark script: could you save your dataset and see if you still don't see any events?
Guntaka Jeevan Paul
babynames = spark.read.format("csv").option("header", "true").option("inferSchema", "true").load("dbfs:/FileStore/babynames.csv")
-babynames.createOrReplaceTempView("babynames_table")
-years = spark.sql("select distinct(Year) from babynames_table").rdd.map(lambda row : row[0]).collect()
-years.sort()
-dbutils.widgets.dropdown("year", "2014", [str(x) for x in years])
-display(babynames.filter(babynames.Year == dbutils.widgets.get("year")))Guntaka Jeevan Paul
this is the script that i am running @PaweĆ LeszczyĆskiâŠkindly let me know if iâm doing any mistake. I have added the init script at the cluster level and from the logs i could see that openlineage is configured as i see a log statement
PaweĆ LeszczyĆski
there's nothing wrong in that script. It's just we decided to limit amount of OL events for jobs that don't write their data anywhere and just do
collect operationPaweĆ LeszczyĆski
this is also a potential reason why can't you see any events
Guntaka Jeevan Paul
ohhâŠokk, will try out the test script that you have mentioned above. Kindly correct me if my understanding is correct, so if there are a few transformatiosna nd finally writing somewhere that is where the OL events are expected to be emitted?
PaweĆ LeszczyĆski
yes. main purpose of the lineage is to track dependencies between the datasets, when a job reads from dataset A and writes to dataset B. In case of databricks notebook, that do
show or collect and print some query result on the screen, there may be no reason to track it in the sense of lineage.Michael Robinson
@channel
We released OpenLineage 1.4.1!
Additions:
âą Client: allow setting clientâs endpoint via environment variable 2151 @Mars Lan
âą Flink: expand Iceberg source types 2149 @U05QA2D1XNV
âą Spark: add debug facet 2147 @PaweĆ LeszczyĆski
âą Spark: enable Nessie REST catalog 2165 @julwin
Thanks to all the contributors, especially new contributors @U05QA2D1XNV and @julwin!
Release: https://github.com/OpenLineage/OpenLineage/releases/tag/1.4.1
Changelog: https://github.com/OpenLineage/OpenLineage/blob/main/CHANGELOG.md
Commit history: https://github.com/OpenLineage/OpenLineage/compare/1.3.1...1.4.1
Maven: https://oss.sonatype.org/#nexus-search;quick~openlineage
PyPI: https://pypi.org/project/openlineage-python/
We released OpenLineage 1.4.1!
Additions:
âą Client: allow setting clientâs endpoint via environment variable 2151 @Mars Lan
âą Flink: expand Iceberg source types 2149 @U05QA2D1XNV
âą Spark: add debug facet 2147 @PaweĆ LeszczyĆski
âą Spark: enable Nessie REST catalog 2165 @julwin
Thanks to all the contributors, especially new contributors @U05QA2D1XNV and @julwin!
Release: https://github.com/OpenLineage/OpenLineage/releases/tag/1.4.1
Changelog: https://github.com/OpenLineage/OpenLineage/blob/main/CHANGELOG.md
Commit history: https://github.com/OpenLineage/OpenLineage/compare/1.3.1...1.4.1
Maven: https://oss.sonatype.org/#nexus-search;quick~openlineage
PyPI: https://pypi.org/project/openlineage-python/
đ5
Drew Bittenbender
Hello. I am getting started with OL and Marquez with dbt. I am using dbt-ol. The namespace of the dataset showing up in Marquez is not the namespace I provide using OPENLINEAGE_NAMESPACE. It happens to be the same as the source in Marquez which is the snowflake account uri. It's obviously picking up the other env variable OPENLINEAGE_URL so i am pretty sure its not the environment. Is this expected?
Michael Robinson
Hi Drew, thank you for using OpenLineage! I donât know the details of your use case, but I believe this is expected, yes. In general, the dataset namespace is different. Jobs are namespaced separately from datasets, which are namespaced by their containing datasources. This is the case so datasets have the same name regardless of the job writing to them, as datasets are sometimes shared by jobs in different namespaces.
đ1
Jason Yip
Any idea why "environment-properties" is gone in Spark 3.4+ in StartEvent?
Jason Yip
example:
{"environment-properties":{"spark.databricks.clusterUsageTags.clusterName":"<mailto:jason.yip@tredence.com|jason.yip@tredence.com>'s Cluster","spark.databricks.job.runId":"","spark.databricks.job.type":"","spark.databricks.clusterUsageTags.azureSubscriptionId":"a4f54399-8db8-4849-adcc-a42aed1fb97f","spark.databricks.notebook.path":"/Repos/jason.yip@tredence.com/segmentation/01_Data Prep","spark.databricks.clusterUsageTags.clusterOwnerOrgId":"4679476628690204","MountPoints":[{"MountPoint":"/databricks-datasets","Source":"databricks-datasets"},{"MountPoint":"/Volumes","Source":"UnityCatalogVolumes"},{"MountPoint":"/databricks/mlflow-tracking","Source":"databricks/mlflow-tracking"},{"MountPoint":"/databricks-results","Source":"databricks-results"},{"MountPoint":"/databricks/mlflow-registry","Source":"databricks/mlflow-registry"},{"MountPoint":"/Volume","Source":"DbfsReserved"},{"MountPoint":"/volumes","Source":"DbfsReserved"},{"MountPoint":"/","Source":"DatabricksRoot"},{"MountPoint":"/volume","Source":"DbfsReserved"}],"User":"<mailto:jason.yip@tredence.com|jason.yip@tredence.com>","UserId":"4768657035718622","OrgId":"4679476628690204"}}PaweĆ LeszczyĆski
Is this related to any OL version? In OL 1.2.2. we've added extra variable
I think we're facing some issues on recent databricks runtime versions. Here is an issue for this: https://github.com/OpenLineage/OpenLineage/issues/2131
Is the problem you describe specific to some databricks runtime versions?
spark.databricks.clusterUsageTags.clusterAllTags to be captured, but this should not break things.I think we're facing some issues on recent databricks runtime versions. Here is an issue for this: https://github.com/OpenLineage/OpenLineage/issues/2131
Is the problem you describe specific to some databricks runtime versions?
Jason Yip
yes, exactly Spark 3.4+
Jason Yip
Btw I don't understand the code flow entirely, if we are talking about a different classpath only, I see there's Unity Catalog handler in the code and it says it works the same as Delta, but I am not seeing it subclassing Delta. I suppose it will work the same.
I am happy to jump on a call to show you if needed
I am happy to jump on a call to show you if needed
Jason Yip
@PaweĆ LeszczyĆski do you think in Spark 3.4+ only one event would happen?
/**
* We get exact copies of OL events for org.apache.spark.scheduler.SparkListenerJobStart and
* org.apache.spark.sql.execution.ui.SparkListenerSQLExecutionStart. The same happens for end
* events.
*
* @return
*/
private boolean isOnJobStartOrEnd(SparkListenerEvent event) {
return event instanceof SparkListenerJobStart || event instanceof SparkListenerJobEnd;
}
/**
* We get exact copies of OL events for org.apache.spark.scheduler.SparkListenerJobStart and
* org.apache.spark.sql.execution.ui.SparkListenerSQLExecutionStart. The same happens for end
* events.
*
* @return
*/
private boolean isOnJobStartOrEnd(SparkListenerEvent event) {
return event instanceof SparkListenerJobStart || event instanceof SparkListenerJobEnd;
}
Guntaka Jeevan Paul
@here i am trying out the databricks spark integration and in one of the events i am getting a openlineage event where the output dataset is having a facet called
Can someone kindly let me know what this
symlinks , the statement that generated this event is this sqlCREATE TABLE IF NOT EXISTS covid_research.covid_data
-USING CSV
-LOCATION '<abfss://oltptestdata@jeevanacceldata.dfs.core.windows.net/testdata/johns-hopkins-covid-19-daily-dashboard-cases-by-states.csv>'
-OPTIONS (header "true", inferSchema "true");Can someone kindly let me know what this
symlinks facet is. i tried seeing the spec but did not get it completelyJason Yip
I use it to get the table with database name
Guntaka Jeevan Paul
so can i think it like if there is a synlink, then that table is kind of a reference to the original dataset
Jason Yip
yes
đ1
Guntaka Jeevan Paul
@here When i am running this sql as part of a databricks notebook, i am recieving an OL event where i see only an output dataset and there is no input dataset or a symlink facet inside the dataset to map it to the underlying azure storage object. Can anyone kindly help on this
spark.sql(f"CREATE TABLE IF NOT EXISTS covid_research.uscoviddata USING delta LOCATION '<abfss://oltptestdata@jeevanacceldata.dfs.core.windows.net/testdata/modified-delta>'")
-{
- "eventTime": "2023-10-11T10:47:36.296Z",
- "producer": "<https://github.com/OpenLineage/OpenLineage/tree/1.2.2/integration/spark>",
- "schemaURL": "<https://openlineage.io/spec/2-0-2/OpenLineage.json#/$defs/RunEvent>",
- "eventType": "COMPLETE",
- "run": {
- "runId": "d0f40be9-b921-4c84-ac9f-f14a86c29ff7",
- "facets": {
- "spark.logicalPlan": {
- "_producer": "<https://github.com/OpenLineage/OpenLineage/tree/1.2.2/integration/spark>",
- "_schemaURL": "<https://openlineage.io/spec/2-0-2/OpenLineage.json#/$defs/RunFacet>",
- "plan": [
- {
- "class": "org.apache.spark.sql.catalyst.plans.logical.CreateTable",
- "num-children": 1,
- "name": 0,
- "tableSchema": [],
- "partitioning": [],
- "tableSpec": null,
- "ignoreIfExists": true
- },
- {
- "class": "org.apache.spark.sql.catalyst.analysis.ResolvedIdentifier",
- "num-children": 0,
- "catalog": null,
- "identifier": null
- }
- ]
- },
- "spark_version": {
- "_producer": "<https://github.com/OpenLineage/OpenLineage/tree/1.2.2/integration/spark>",
- "_schemaURL": "<https://openlineage.io/spec/2-0-2/OpenLineage.json#/$defs/RunFacet>",
- "spark-version": "3.3.0",
- "openlineage-spark-version": "1.2.2"
- },
- "processing_engine": {
- "_producer": "<https://github.com/OpenLineage/OpenLineage/tree/1.2.2/integration/spark>",
- "_schemaURL": "<https://openlineage.io/spec/facets/1-1-0/ProcessingEngineRunFacet.json#/$defs/ProcessingEngineRunFacet>",
- "version": "3.3.0",
- "name": "spark",
- "openlineageAdapterVersion": "1.2.2"
- }
- }
- },
- "job": {
- "namespace": "default",
- "name": "adb-3942203504488904.4.azuredatabricks.net.create_table.covid_research_db_uscoviddata",
- "facets": {}
- },
- "inputs": [],
- "outputs": [
- {
- "namespace": "dbfs",
- "name": "/user/hive/warehouse/covid_research.db/uscoviddata",
- "facets": {
- "dataSource": {
- "_producer": "<https://github.com/OpenLineage/OpenLineage/tree/1.2.2/integration/spark>",
- "_schemaURL": "<https://openlineage.io/spec/facets/1-0-0/DatasourceDatasetFacet.json#/$defs/DatasourceDatasetFacet>",
- "name": "dbfs",
- "uri": "dbfs"
- },
- "schema": {
- "_producer": "<https://github.com/OpenLineage/OpenLineage/tree/1.2.2/integration/spark>",
- "_schemaURL": "<https://openlineage.io/spec/facets/1-0-0/SchemaDatasetFacet.json#/$defs/SchemaDatasetFacet>",
- "fields": []
- },
- "storage": {
- "_producer": "<https://github.com/OpenLineage/OpenLineage/tree/1.2.2/integration/spark>",
- "_schemaURL": "<https://openlineage.io/spec/facets/1-0-0/StorageDatasetFacet.json#/$defs/StorageDatasetFacet>",
- "storageLayer": "unity",
- "fileFormat": "parquet"
- },
- "symlinks": {
- "_producer": "<https://github.com/OpenLineage/OpenLineage/tree/1.2.2/integration/spark>",
- "_schemaURL": "<https://openlineage.io/spec/facets/1-0-0/SymlinksDatasetFacet.json#/$defs/SymlinksDatasetFacet>",
- "identifiers": [
- {
- "namespace": "/user/hive/warehouse/covid_research.db",
- "name": "covid_research.uscoviddata",
- "type": "TABLE"
- }
- ]
- },
- "lifecycleStateChange": {
- "_producer": "<https://github.com/OpenLineage/OpenLineage/tree/1.2.2/integration/spark>",
- "_schemaURL": "<https://openlineage.io/spec/facets/1-0-0/LifecycleStateChangeDatasetFacet.json#/$defs/LifecycleStateChangeDatasetFacet>",
- "lifecycleStateChange": "CREATE"
- }
- },
- "outputFacets": {}
- }
- ]
-}Damien Hawes
Hey Guntaka - can I ask you a favour? Can you please stop using
@here or @channel - please keep in mind, you're pinging over 1000 people when you use that mention. Its incredibly distracting to have Slack notify me of a message that isn't pertinent to me.Guntaka Jeevan Paul
sure noted @Damien Hawes
Damien Hawes
Thank you!
Madhav Kakumani
Hi @there, I am trying to make API call to get column-lineage information could you please let me know the url construct to retrieve the same? As per the API documentation I am passing the following url to GET column-lineage: http://localhost:5000/api/v1/column-lineage but getting error code:400. Thanks
Willy Lulciuc
Make sure to provide a dataset field
You can view the API docs for column lineage here!
nodeId as a query param in your request. If youâve seeded Marquez with test metadata, you can use:curl -XGET "<http://localhost:5002/api/v1/column-lineage?nodeId=datasetField%3Afood_delivery%3Apublic.delivery_7_days%3Acustomer_email>"You can view the API docs for column lineage here!
Madhav Kakumani
Thanks Willy. The documentation says 'name space' so i constructed API Like this:
'http://marquez-web:3000/api/v1/column-lineage/nodeId=datasetField:file:/home/jovyan/Downloads/event_attribute.csv:eventType'
but it is still not working đ
'http://marquez-web:3000/api/v1/column-lineage/nodeId=datasetField:file:/home/jovyan/Downloads/event_attribute.csv:eventType'
but it is still not working đ
Madhav Kakumani
nodeId is constructed like this: datasetField:<namespace>:<dataset>:<field name>
Michael Robinson
@channel
Friendly reminder: this monthâs TSC meeting, open to all, is tomorrow at 10 am PT: https://openlineage.slack.com/archives/C01CK9T7HKR/p1696531454431629
Friendly reminder: this monthâs TSC meeting, open to all, is tomorrow at 10 am PT: https://openlineage.slack.com/archives/C01CK9T7HKR/p1696531454431629
Michael Robinson
Newly added discussion topics:
âą a proposal to add a Registry of Consumers and Producers
âą a dbt issue to add OpenLineage Dataset names to the Manifest
âą a proposal to add Dataset support in Spark LogicalPlan Nodes
âą a proposal to institute a certification process for new integrations
âą a proposal to add a Registry of Consumers and Producers
âą a dbt issue to add OpenLineage Dataset names to the Manifest
âą a proposal to add Dataset support in Spark LogicalPlan Nodes
âą a proposal to institute a certification process for new integrations
Jason Yip
This might be a dumb question, I guess I need to setup local Spark in order for the Spark tests to run successfully?
PaweĆ LeszczyĆski
just follow these instructions: https://github.com/OpenLineage/OpenLineage/tree/main/integration/spark#build
Guntaka Jeevan Paul
when trying to install openlineage-java in local via this command --> cd ../../client/java/ && ./gradlew publishToMavenLocal, i am receiving this error
> Task :signMavenJavaPublication FAILED
-
-FAILURE: Build failed with an exception.
-
-* What went wrong:
-Execution failed for task ':signMavenJavaPublication'.
-> Cannot perform signing task ':signMavenJavaPublication' because it has no configured signatoryJason Yip
@PaweĆ LeszczyĆski this is what I am getting
Jason Yip
attaching the html
PaweĆ LeszczyĆski
which java are you using? what is your operation system (is it windows?)?
Jason Yip
yes it is Windows, i downloaded java 8 but I can try to build it with Linux subsystem or Mac
Guntaka Jeevan Paul
In my case it is Mac
Jason Yip
* Where:
Build file '/mnt/c/Users/jason/Downloads/github/OpenLineage/integration/spark/build.gradle' line: 9
* What went wrong:
An exception occurred applying plugin request [id: 'com.adarshr.test-logger', version: '3.2.0']
> Failed to apply plugin [id 'com.adarshr.test-logger']
> Could not generate a proxy class for class com.adarshr.gradle.testlogger.TestLoggerExtension.
* Try:
Build file '/mnt/c/Users/jason/Downloads/github/OpenLineage/integration/spark/build.gradle' line: 9
* What went wrong:
An exception occurred applying plugin request [id: 'com.adarshr.test-logger', version: '3.2.0']
> Failed to apply plugin [id 'com.adarshr.test-logger']
> Could not generate a proxy class for class com.adarshr.gradle.testlogger.TestLoggerExtension.
* Try:
Jason Yip
tried with Linux subsystem
PaweĆ LeszczyĆski
we don't have any restrictions for windows builds, however it is something we don't test regularly. 2h ago we did have a successful build on circle CI https://app.circleci.com/pipelines/github/OpenLineage/OpenLineage/8271/workflows/0ec521ae-cd21-444a-bfec-554d101770ea
Jason Yip
... 111 more
Caused by: java.lang.ClassNotFoundException: org.gradle.api.provider.HasMultipleValues
... 117 more
Caused by: java.lang.ClassNotFoundException: org.gradle.api.provider.HasMultipleValues
... 117 more
Jason Yip
@PaweĆ LeszczyĆski now I am doing gradlew instead of gradle on windows coz Linux one doesn't work. The doc didn't mention about setting up Spark / Hadoop and that's my original question -- do I need to setup local Spark? Now it's throwing an error on Hadoop: java.io.FileNotFoundException: java.io.FileNotFoundException: HADOOP_HOME and hadoop.home.dir are unset.
Jason Yip
Got it working with Mac, couldn't get it working with Windows / Linux subsystem
Jason Yip
Now getting class not found despite build and test succeeded
Jason Yip
I uploaded the wrong jar.. there are so many jars, only the jar in the spark folder works, not subfolder
Anirudh Shrinivason
Hi team, I am running the following pyspark code in a cell:
I am running with debug log levels. I actually don't see any of the events being logged for
print("SELECTING 100 RECORDS FROM METADATA TABLE")
-df = spark.sql("""select * from <table> limit 100""")
-
-print("WRITING (1) 100 RECORDS FROM METADATA TABLE")
-df.write.mode("overwrite").format('delta').save("<s3 location 1>")
-df.createOrReplaceTempView("temp_metadata")
-
-print("WRITING (2) 100 RECORDS FROM METADATA TABLE")
-df.write.mode("overwrite").format("delta").save("<s3 location 2>")
-
-print("READING (1) 100 RECORDS FROM METADATA TABLE")
-df_read = spark.read.format('delta').load("<s3 location 3>")
-df_read.createOrReplaceTempView("metadata_1")
-
-print("DOING THE MERGE INTO SQL STEP!")
-df_new = spark.sql("""
- MERGE INTO metadata_1
- USING <table>
- ON metadata_1.id = temp_metadata.id
- WHEN MATCHED THEN UPDATE SET
- metadata_1.id = temp_metadata.id,
- metadata_1.aspect = temp_metadata.aspect
- WHEN NOT MATCHED THEN INSERT (id, aspect)
- VALUES (temp_metadata.id, temp_metadata.aspect)
-""")I am running with debug log levels. I actually don't see any of the events being logged for
SaveIntoDataSourceCommand or the MergeIntoCommand, but OL is in fact emitting events to the backend. It seems like the events are just not being logged... I actually observe this for all delta table related spark sql queries...Anirudh Shrinivason
Hi @PaweĆ LeszczyĆski is this expected? CMIIW but we should expect to see the events being logged when running with debug log level right?
Damien Hawes
It's impossible to know without seeing how you've configured the listener.
Can you show this configuration?
Can you show this configuration?
Anirudh Shrinivason
These are my spark configs... I'm setting log level to debug with
spark.openlineage.transport.url <url>
-spark.openlineage.transport.endpoint /<endpoint>
-spark.openlineage.transport.type http
-spark.extraListeners io.openlineage.spark.agent.OpenLineageSparkListener
-spark.openlineage.facets.custom_environment_variables [BUNCH_OF_VARIABLES;]
-spark.openlineage.facets.disabled [spark_unknown\;spark.logicalPlan]These are my spark configs... I'm setting log level to debug with
sc.setLogLevel("DEBUG")Damien Hawes
Two things:
1. If you want debug logs, you're going to have to provide a
2. You're telling the listener to emit to a URL. If you want to see the events emitted to the console, then set
Do either (1) or (2).
1. If you want debug logs, you're going to have to provide a
log4j.properties file or log4j2.properties file depending on the version of spark you're running. In that file, you will need to configure the logging levels. If I am not mistaken, the sc.setLogLevel controls ONLY the log levels of Spark namespaced components (i.e., org.apache.spark)2. You're telling the listener to emit to a URL. If you want to see the events emitted to the console, then set
spark.openlineage.transport.type=console, and remove the other spark.openlineage.transport.* configurations.Do either (1) or (2).
Anirudh Shrinivason
@Damien Hawes Hi, sflr.
1. So enabling
2. I do want to emit events to the url. But, I would like to just see what exactly are the events being emitted for some specific jobs, since I see that the lineage is incorrect for some MergeInto cases
1. So enabling
sc.setLogLevel does actually enable debug logs from Openlineage. I can see the events and everyting being logged if I save it as a parquet format instead of delta. 2. I do want to emit events to the url. But, I would like to just see what exactly are the events being emitted for some specific jobs, since I see that the lineage is incorrect for some MergeInto cases
Anirudh Shrinivason
Hi @Damien Hawes would like to check again on whether you'd have any thoughts about this... Thanks! đ
Rodrigo Maia
Hello All đ!
We are currently trying to work the the spark integration for OpenLineage in our Databricks instance. The general setup is done and working with a few hicups here and there.
But one thing we are still struggling is how to link all spark jobs events with a Databricks job or a notebook run.
WeÂŽve recently noticed that some of the events produced by OL have the "environment-properties" attribute with information (for our context) regarding notebook path (if it is a notebook run), or the the job run ID (if its a databricks job run). But the thing is that these attributes are not always present.
I ran some samples yesterday for a job with 4 notebook tasks. From all 20 json payload sent by the OL listener, only 3 presented the "environment-properties" attribute. Its not only happening with Databricks jobs. When i run single notebooks and each cell has its onw set of spark jobs, not all json events presented that property either.
So my question is what is the criteria to have this attributes present or not in the event json file? Or maybe this in an issue? @Jason Yip did you find out anything about this?
âïž Spark 3.4 / OL-Spark 1.4.1
We are currently trying to work the the spark integration for OpenLineage in our Databricks instance. The general setup is done and working with a few hicups here and there.
But one thing we are still struggling is how to link all spark jobs events with a Databricks job or a notebook run.
WeÂŽve recently noticed that some of the events produced by OL have the "environment-properties" attribute with information (for our context) regarding notebook path (if it is a notebook run), or the the job run ID (if its a databricks job run). But the thing is that these attributes are not always present.
I ran some samples yesterday for a job with 4 notebook tasks. From all 20 json payload sent by the OL listener, only 3 presented the "environment-properties" attribute. Its not only happening with Databricks jobs. When i run single notebooks and each cell has its onw set of spark jobs, not all json events presented that property either.
So my question is what is the criteria to have this attributes present or not in the event json file? Or maybe this in an issue? @Jason Yip did you find out anything about this?
âïž Spark 3.4 / OL-Spark 1.4.1
PaweĆ LeszczyĆski
In general, we assume that OL events per run are cumulative. So, if you have 20 events with the same
Are you able to use job
runId , then even if a single event contains some facet, we consider this is OK and let the backend combine it together. That's what we do in Marquez project (a reference backend architecture for OL) and that's why it is worth to use in Marquez as a rest API.Are you able to use job
namespace to aggregate all the Spark actions run within the databricks notebook? This is something that should serve this purpose.Jason Yip
@Rodrigo Maia for Spark 3.4 I don't see the environment-properties showing up at all, but if you run the code as it is, register a listener on SparkListenerJobStart and get the properties, all of those properties will show up. There's an event filter that filters out the SparkListenerJobStart, I suspect that filtered out the "unneccessary" events.. was trying to do a custom build to do that, but still trying to setup Hadoop and Spark on my local
Rodrigo Maia
@PaweĆ LeszczyĆski you are right. This is what we are doing as well, combining events with the same runId to process the information on our backend. But even so, there are several runIds without this information. I went through these events to have a better view of what was happening. As you can see from 7 runIds, only 3 were showing the "environment-properties" attribute. Some condition is not being met here, or maybe it is what @Jason Yip suspects and there's some sort of filtering of unnecessary events
PaweĆ LeszczyĆski
@Rodrigo Maia, If you are able to provide a small Spark script such that none of the OL events contain the environment-properties, but at least one should, please raise an issue for this.
PaweĆ LeszczyĆski
It's extremely helpful when community open issues that are not only described well, but also contain small piece of code needed to reproduce this.
Rodrigo Maia
I know. that's the goal. that is why I wanted to understand in the first place if there was any condition preventing this from happening, but now i get that this is not expected behaviour.
đ1
Jason Yip
@PaweĆ LeszczyĆski @Rodrigo Maia I am referring to this: https://github.com/OpenLineage/OpenLineage/blob/main/integration/spark/shared/src/main/java/io/openlineage/spark/agent/filters/DeltaEventFilter.java#L51
Jason Yip
Please note that I am getting the same behavior, no code is needed, Spark 3.4+ won't be generating no matter what. I have been testing the same code for 2 months from this issue: https://github.com/OpenLineage/OpenLineage/issues/2124
I tried the code without OL and it worked perfectly, so it is OL filtering out the event for sure. I will try posting the code I use to collect the properties.
I tried the code without OL and it worked perfectly, so it is OL filtering out the event for sure. I will try posting the code I use to collect the properties.
Jason Yip
this code proves that the prosperities are still there, somehow got filtered out by OL:
%scala
-import org.apache.spark.scheduler._
-
-class JobStartListener extends SparkListener {
- override def onJobStart(jobStart: SparkListenerJobStart): Unit = {
- // Extract properties here
- val jobId = jobStart.jobId
- val stageInfos = jobStart.stageInfos
- val properties = jobStart.properties
-
- // You can print properties or save them somewhere
- println(s"JobId: $jobId, Stages: ${stageInfos.size}, Properties: $properties")
- }
-}
-
-val listener = new JobStartListener()
-spark.sparkContext.addSparkListener(listener)
-
-val df = spark.range(1000).repartition(10)
-df.count()Jason Yip
of course feel free to test this logic as well, it still works -- if not the filtering:
https://github.com/OpenLineage/OpenLineage/blob/main/integration/spark/shared/src/[âŠ]ark/agent/facets/builder/DatabricksEnvironmentFacetBuilder.java
https://github.com/OpenLineage/OpenLineage/blob/main/integration/spark/shared/src/[âŠ]ark/agent/facets/builder/DatabricksEnvironmentFacetBuilder.java
Rodrigo Maia
Any ideas on how could i test it?
ankit jain
Hello All, I am completely new for Openlineage, I have to setup the lab to conduct POC on various aspects like Lineage, metadata management , etc. As per openlineage site, i tried downloading Ubuntu, docker and binary files for Marquez. But I am lost somewhere and unable to configure whole setup. Can someone please assist in steps to start from scratch so that i can delve into the Openlineage capabilities. Many thanks
Jakub DardziĆski
hey, did you try to follow one of these guides?
https://openlineage.io/docs/guides/about
https://openlineage.io/docs/guides/about
Michael Robinson
Which guide were you using, and what errors/issues are you encountering?
ankit jain
Thanks Jakub for the response.
ankit jain
In docker, marquez-api image is not running and exiting with the exit code 127.
Michael Robinson
@ankit jain thanks. I don't recognize 127, but 9 times out of 10 if the API or DB container fails the reason is a port conflict. Have you checked if port 5000 is available?
Jakub DardziĆski
could you please check whatâs the output of
or
?
git config --get core.autocrlfor
git config --global --get core.autocrlf?
ankit jain
@Michael Robinson thanks , I checked the port 5000 is not available.
I tried deleting docker images and recreating them, but still the same issue persist stating
/Usr/bin/env bash/r not found.
Gradle build is successful.
I tried deleting docker images and recreating them, but still the same issue persist stating
/Usr/bin/env bash/r not found.
Gradle build is successful.
ankit jain
@Jakub DardziĆski thanks, first command resulted as true and second command has no response
Jakub DardziĆski
are you running docker and git in Windows or Mac OS before 10.0?
Matthew Paras
Hey all - we've been noticing that some events go unreported by openlineage (spark) when the
Also as a note, the query plans from this job are pretty big - could the listener just be choking up? Happy to open a github issue as well if we suspect that it could be the listener itself having issues
AsyncEventQueue fills up and starts dropping events. Wondering if anyone has experienced this before, and knows why it is happening? We've expanded the event queue capacity and thrown more hardware at the problem but no diceAlso as a note, the query plans from this job are pretty big - could the listener just be choking up? Happy to open a github issue as well if we suspect that it could be the listener itself having issues
Anirudh Shrinivason
Hi, just checking, are you excluding the sparkPlan from the events? Or is it sending the spark plan too
Maciej Obuchowski
yeah - setting
spark.openlineage.facets.disabled to [spark_unknown;spark.logicalPlan] should helpMatthew Paras
sorry for the late reply - turns out this job is just whack đ we were going in circles trying to figure it out, we end up dropping events without open lineage enabled at all. But good to know that disabling the logical plan should speed us up if we run into this again
savan
@GitHubOpenLineageIssues
I am trying to contribute to Integration tests which is listed here as good first issue
the CONTRIBUTING.md mentions that i can trigger CI for integration tests from forked branch.
using this tool.
but i am unable to do so, is there a way to trigger CI from forked brach or do i have to get permission from someone to run the CI?
i am getting this error when i run this command
>
i have tried to configure ssh key
also tried to trigger CI from another brach,
and tried all of this after fetching the latest upstream
cc: @Athitya Kumar @Maciej Obuchowski @U05HD9G5T17
I am trying to contribute to Integration tests which is listed here as good first issue
the CONTRIBUTING.md mentions that i can trigger CI for integration tests from forked branch.
using this tool.
but i am unable to do so, is there a way to trigger CI from forked brach or do i have to get permission from someone to run the CI?
i am getting this error when i run this command
sudo git-push-fork-to-upstream-branch upstream savannavalgi:hacktober>
Username for '<https://github.com>': savannavalgi
-> Password for '<https://savannavalgi@github.com>':
-> remote: Permission to OpenLineage/OpenLineage.git denied to savannavalgi.
-> fatal: unable to access '<https://github.com/OpenLineage/OpenLineage.git/>': The requested URL returned error: 403i have tried to configure ssh key
also tried to trigger CI from another brach,
and tried all of this after fetching the latest upstream
cc: @Athitya Kumar @Maciej Obuchowski @U05HD9G5T17
praveen kanamarlapudi
Hi,
We are using openlineage spark connector. We have used spark 3.2 and scala 2.12 so far. We have triggered a new job with Spark 3.4 and scala 2.13 and faced below exception.
We are using openlineage spark connector. We have used spark 3.2 and scala 2.12 so far. We have triggered a new job with Spark 3.4 and scala 2.13 and faced below exception.
java.lang.NoSuchMethodError: 'scala.collection.Seq org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.map(scala.Function1)'
- at io.openlineage.spark.agent.lifecycle.OpenLineageRunEventBuilder.lambda$buildInputDatasets$6(OpenLineageRunEventBuilder.java:341)
- at java.base/java.util.Optional.map(Optional.java:265)
- at io.openlineage.spark.agent.lifecycle.OpenLineageRunEventBuilder.buildInputDatasets(OpenLineageRunEventBuilder.java:339)
- at io.openlineage.spark.agent.lifecycle.OpenLineageRunEventBuilder.populateRun(OpenLineageRunEventBuilder.java:295)
- at io.openlineage.spark.agent.lifecycle.OpenLineageRunEventBuilder.buildRun(OpenLineageRunEventBuilder.java:279)
- at io.openlineage.spark.agent.lifecycle.OpenLineageRunEventBuilder.buildRun(OpenLineageRunEventBuilder.java:222)
- at io.openlineage.spark.agent.lifecycle.SparkSQLExecutionContext.start(SparkSQLExecutionContext.java:72)
- at io.openlineage.spark.agent.OpenLineageSparkListener.lambda$sparkSQLExecStart$0(OpenLineageSparkListener.java:91)PaweĆ LeszczyĆski
Hmy, that is interesting. Did it occur on databricks runtime? Could you give it a try with Scala 2.12? I think we don't test scala 2.13.
praveen kanamarlapudi
I believe our Scala 2.12 jobs are working fine. It's not databricks runtime. We run Spark on Kube.
PaweĆ LeszczyĆski
Ok. I think You can raise an issue to support Scala 2.13 for latest Spark versions.
priya narayana
Hi I want to customise the events which comes from Openlineage spark . Can some one give some information
PaweĆ LeszczyĆski
Hi @priya narayana, please get familiar with
Extending section on our docs: https://github.com/OpenLineage/OpenLineage/tree/main/integration/spark#extendingpriya narayana
Okay thank you. Just checking any other docs or git code which also can help me
harsh loomba
Hello Team
harsh loomba
Im upgrading the version from openlineage-airflow==0.24.0 to openlineage-airflow 1.4.1 but im seeing the following error, any help is appreciated
harsh loomba
@Jakub DardziĆski any thoughts?
Jakub DardziĆski
what version of Airflow are you using?
harsh loomba
2.6.3 that satisfies the requirement
Jakub DardziĆski
is it possible you have some custom operator?
harsh loomba
i think its the base operator causing the issue
harsh loomba
so no i believe
Jakub DardziĆski
BaseOperator is parent class for any other operators, it defines how to do deepcopy
harsh loomba
yeah so its controlled by Airflow itself, I didnt customize it
Jakub DardziĆski
uhm, maybe it's possible you could share dag code? you may hide sensitive data
harsh loomba
let me try with lower versions of openlineage, what's say
harsh loomba
its a big jump from 0.24.0 to 1.4.1
harsh loomba
but i will help here to investigate this issue
Jakub DardziĆski
for me it seems that within dag or task you're defining some object that is not easy to copy
harsh loomba
possible, but with 0.24.0 that issue is not occurring, so worry is that the version upgrade could potentially break things
Jakub DardziĆski
0.24.0 is not that old đ€
harsh loomba
i see the issue with 0.24.0 I see it as warning
but with 1.4.1 its stopped processing any further and threw error
[airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - File "/usr/lib64/python3.8/threading.py", line 932, in _bootstrap_inner
-[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - self.run()
-[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - File "/usr/lib64/python3.8/threading.py", line 870, in run
-[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - self._target(*self._args, **self._kwargs)
-[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - File "/home/upgrade/.local/lib/python3.8/site-packages/openlineage/airflow/listener.py", line 89, in on_running
-[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - task_instance_copy = copy.deepcopy(task_instance)
-[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - File "/usr/lib64/python3.8/copy.py", line 172, in deepcopy
-[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - y = _reconstruct(x, memo, *rv)
-[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - File "/usr/lib64/python3.8/copy.py", line 270, in _reconstruct
-[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - state = deepcopy(state, memo)
-[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - File "/usr/lib64/python3.8/copy.py", line 146, in deepcopy
-[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - y = copier(x, memo)
-[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - File "/usr/lib64/python3.8/copy.py", line 230, in _deepcopy_dict
-[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - y[deepcopy(key, memo)] = deepcopy(value, memo)
-[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - File "/usr/lib64/python3.8/copy.py", line 172, in deepcopy
-[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - y = _reconstruct(x, memo, *rv)
-[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - File "/usr/lib64/python3.8/copy.py", line 270, in _reconstruct
-[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - state = deepcopy(state, memo)
-[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - File "/usr/lib64/python3.8/copy.py", line 146, in deepcopy
-[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - y = copier(x, memo)
-[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - File "/usr/lib64/python3.8/copy.py", line 230, in _deepcopy_dict
-[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - y[deepcopy(key, memo)] = deepcopy(value, memo)
-[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - File "/usr/lib64/python3.8/copy.py", line 153, in deepcopy
-[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - y = copier(memo)
-[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - File "/home/upgrade/.local/lib/python3.8/site-packages/airflow/models/dag.py", line 2162, in __deepcopy__
-[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - setattr(result, k, copy.deepcopy(v, memo))
-[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - File "/usr/lib64/python3.8/copy.py", line 146, in deepcopy
-[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - y = copier(x, memo)
-[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - File "/usr/lib64/python3.8/copy.py", line 230, in _deepcopy_dict
-[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - y[deepcopy(key, memo)] = deepcopy(value, memo)
-[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - File "/usr/lib64/python3.8/copy.py", line 153, in deepcopy
-[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - y = copier(memo)
-[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - File "/home/upgrade/.local/lib/python3.8/site-packages/airflow/models/baseoperator.py", line 1224, in __deepcopy__
-[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - setattr(result, k, copy.deepcopy(v, memo))
-[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - File "/usr/lib64/python3.8/copy.py", line 172, in deepcopy
-[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - y = _reconstruct(x, memo, *rv)
-[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - File "/usr/lib64/python3.8/copy.py", line 270, in _reconstruct
-[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - state = deepcopy(state, memo)
-[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - File "/usr/lib64/python3.8/copy.py", line 146, in deepcopy
-[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - y = copier(x, memo)
-[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - File "/usr/lib64/python3.8/copy.py", line 230, in _deepcopy_dict
-[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - y[deepcopy(key, memo)] = deepcopy(value, memo)
-[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - File "/usr/lib64/python3.8/copy.py", line 146, in deepcopy
-[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - y = copier(x, memo)
-[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - File "/usr/lib64/python3.8/copy.py", line 230, in _deepcopy_dict
-[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - y[deepcopy(key, memo)] = deepcopy(value, memo)
-[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - File "/usr/lib64/python3.8/copy.py", line 153, in deepcopy
-[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - y = copier(memo)
-[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - File "/home/upgrade/.local/lib/python3.8/site-packages/airflow/models/baseoperator.py", line 1224, in __deepcopy__
-[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - setattr(result, k, copy.deepcopy(v, memo))
-[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - File "/usr/lib64/python3.8/copy.py", line 146, in deepcopy
-[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - y = copier(x, memo)
-[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - File "/usr/lib64/python3.8/copy.py", line 230, in _deepcopy_dict
-[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - y[deepcopy(key, memo)] = deepcopy(value, memo)
-[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - File "/usr/lib64/python3.8/copy.py", line 161, in deepcopy
-[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - rv = reductor(4)
-[2023-10-26, 17:40:50 UTC] [airflow/utils/log/logging_mixin.py::_propagate_log()::150] WARNING - TypeError: cannot pickle 'module' objectbut with 1.4.1 its stopped processing any further and threw error
harsh loomba
I see the difference of calling in these 2 versions, current versions checks if Airflow is >2.6 then directly runs on_running but earlier version was running on separate thread. IS this what's raising this exception?
harsh loomba
this is the issue - https://github.com/OpenLineage/OpenLineage/blob/c343835c1664eda94d5c315897ae6702854c81bd/integration/airflow/openlineage/airflow/listener.py#L89 while copying the task
harsh loomba
since we are directly running if version>2.6.0 therefore its throwing error in main processing
harsh loomba
may i know which Airflow version we tested this process?
harsh loomba
im on 2.6.3
Jakub DardziĆski
2.1.4, 2.2.4, 2.3.4, 2.4.3, 2.5.2, 2.6.1
usually there are not too many changes between minor versions
I still believe it might be some code you might improve and probably is also an antipattern in airflow
usually there are not too many changes between minor versions
I still believe it might be some code you might improve and probably is also an antipattern in airflow
harsh loomba
hummm...that's a valid observation but I dont write DAGS, other teams do, so imagine if many people wrote such DAGS I can't ask everyone to change their patterns right? If something is running on current openlineage version with warning that should still be running on upgraded version isn't it?
harsh loomba
however I see ur point
harsh loomba
So that specific task has 570 line of query and pretty bulky query, let me split into smaller units
harsh loomba
that should help right? @Jakub DardziĆski
Jakub DardziĆski
query length shouldnât be the issue, rather any python code
Jakub DardziĆski
I get your point too, we might figure out some mechanism to skip irrelevant parts of task instance so that it doesnât fail then
harsh loomba
actually its failing on that task itself
harsh loomba
let me try it will be pretty quick
harsh loomba
@Jakub DardziĆski but ur right we have to fix this at Openlineage side as well. Because ideally Openlineage shouldn't be causing any issue to the main DAG processing
Jakub DardziĆski
it doesnât break any airflow functionality, execution is wrapped into try/except block, only exception traceback is logged as you can see
Maciej Obuchowski
Can you migrate to Airflow 2.7 and use
apache-airflow-providers-openlineage? Ideally we wouldn't make meaningful changes to openlineage-airflowharsh loomba
yup thats what im planning to do
harsh loomba
referencing to this conversation - what it takes to move to openlineage provider package from openlineage-airflow. Im updating Airflow to 2.7.2 but moving off of openlineage-airflow to provider package Im trying to estimate the amount of work it takes, any thoughts? reading change_logs I dont think its too much of a change but please share your thoughts and if somewhere its drafted please do share that as well
Maciej Obuchowski
Generally not much - I would maybe think of a operator coverage. For example, for BigQuery old
openlineage-airflow supports BigQueryExecuteQueryOperator. However, new apache-airflow-providers-openlineage supports BigQueryInsertJobOperator - because it's intended replacement for BigQueryExecuteQueryOperator and Airflow community does not want to accept contributions to deprecated operators.đ1
harsh loomba
one question if someone is around - when im keeping both
any thoughts?
openlineage-airflow and apache-airflow-providers-openlineage in my requirement file, i see the following error - from openlineage.airflow.extractors import Extractors
-ModuleNotFoundError: No module named 'openlineage.airflow'any thoughts?
John Lukenoff
I would usually do a
pip freeze | grep openlineage as a sanity check to validate that the module is actually installed. Not sure how the provider and the module play together thoughharsh loomba
yeah so @John Lukenoff im not getting how i can use the specific extractor when i run my operator. Say for example, I have custom datawarehouseOperator and i want to override get_openlineage_facets_on_start and get_openlineage_facets_on_complete using the redshift extractor then how would i do that?
Rodrigo Maia
Spark Integration Logs
Hey There
Are these events skipped because it's not supported or it's configured somewhere?
Hey There
Are these events skipped because it's not supported or it's configured somewhere?
23/10/27 08:25:58 INFO SparkSQLExecutionContext: OpenLineage received Spark event that is configured to be skipped: SparkListenerSQLExecutionStart23/10/27 08:25:58 INFO SparkSQLExecutionContext: OpenLineage received Spark event that is configured to be skipped: SparkListenerSQLExecutionEnd
Hitesh
Hi People, actually I want to intercept the OpenLineage spark events right after the job ends and before they are emitted, so that I can add some extra information to the events or remove some information that I don't want.
Is there any way of doing this? Can someone please help me
Is there any way of doing this? Can someone please help me
Michael Robinson
It general, I think this kind of use case is probably best served by facets, but what do you think @PaweĆ LeszczyĆski?
harsh loomba
referencing to this conversation - what it takes to move to openlineage provider package from openlineage-airflow. Im updating Airflow to 2.7.2 but moving off of openlineage-airflow to provider package Im trying to estimate the amount of work it takes, any thoughts? reading change_logs I dont think its too much of a change but please share your thoughts and if somewhere its drafted please do share that as well
Kavitha
Hello, has anyone run into similar error as posted in this github open issues[https://github.com/MarquezProject/marquez/issues/2468] while setting up marquez on an EC2 Instance, would appreciate any help to get past the errors
Willy Lulciuc
More specifically, the AWS RDS section. How are you deploying Marquez on Ec2?
Kavitha
we were primarily referencing this document on git - https://github.com/MarquezProject/marquez
Kavitha
leveraged docker and docker-compose
Willy Lulciuc
hmm so youâre running docker-compose up on an Ec2 instance youâve sshâd into? (just trying to understand your setup better)
Kavitha
yes, thats correct
Willy Lulciuc
Iâve only used docker compose for local dev or integration tests. but, ok youâre probably in the PoC phase. Can you run the docker cmd on you local machine successfully? What OS is stalled on the Ec2 instance?
Kavitha
yes, i can run and the OS is Ubuntu 20.04.6 LTS
Kavitha
we initiallly ran into a permission denied error related to postgressql.conf file and we had to update file permissions to 777 and after which we started to see below errors
Kavitha
marquez-db | 2023-10-27 20:35:52.512 GMT [35] FATAL: no pg_hba.conf entry for host "172.18.0.5", user "marquez", database "marquez", no encryption
marquez-db | 2023-10-27 20:35:52.529 GMT [36] FATAL: no pg_hba.conf entry for host "172.18.0.5", user "marquez", database "marquez", no encryption
marquez-db | 2023-10-27 20:35:52.529 GMT [36] FATAL: no pg_hba.conf entry for host "172.18.0.5", user "marquez", database "marquez", no encryption
Kavitha
we then manually updated pg_hba.conf file to include host user and db details
Willy Lulciuc
Did you also update the
marquez.yml with the db user / password?Kavitha
after which we started to see the errors posted in the github open issues page
Willy Lulciuc
hmm are you using an external database or are you spinning up the entire Marquez stack with docker compose?
Kavitha
we are spinning up the entire Marquez stack with docker compose
Kavitha
we did not change anything in the marquez.yml, i think we did not find that file in the github repo that we cloned into our local instance
Willy Lulciuc
can you grab all the docker compose logs and share them? itâs hard to debug otherwise
Willy Lulciuc
I would first suggest to remove the
--build flag since you are specifying a version of Marquez to use via --tagWilly Lulciuc
no the issue per se, but will help clear up some of the logs
Kavitha
for sure thanks. we could get the logs without the --build portion, we tried with that option just once
Kavitha
the errors were the same with/without --build option
Kavitha
marquez-api | ERROR [2023-10-27 21:34:58,019] org.apache.tomcat.jdbc.pool.ConnectionPool: Unable to create initial connections of pool.
marquez-api | ! org.postgresql.util.PSQLException: FATAL: password authentication failed for user "marquez"
marquez-api | ! at org.postgresql.core.v3.ConnectionFactoryImpl.doAuthentication(ConnectionFactoryImpl.java:693)
marquez-api | ! at org.postgresql.core.v3.ConnectionFactoryImpl.tryConnect(ConnectionFactoryImpl.java:203)
marquez-api | ! at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:258)
marquez-api | ! at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:54)
marquez-api | ! at org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:253)
marquez-api | ! at org.postgresql.Driver.makeConnection(Driver.java:434)
marquez-api | ! at org.postgresql.Driver.connect(Driver.java:291)
marquez-api | ! at org.apache.tomcat.jdbc.pool.PooledConnection.connectUsingDriver(PooledConnection.java:346)
marquez-api | ! at org.apache.tomcat.jdbc.pool.PooledConnection.connect(PooledConnection.java:227)
marquez-api | ! at org.apache.tomcat.jdbc.pool.ConnectionPool.createConnection(ConnectionPool.java:768)
marquez-api | ! at org.apache.tomcat.jdbc.pool.ConnectionPool.borrowConnection(ConnectionPool.java:696)
marquez-api | ! at org.apache.tomcat.jdbc.pool.ConnectionPool.init(ConnectionPool.java:495)
marquez-api | ! at org.apache.tomcat.jdbc.pool.ConnectionPool.<init>(ConnectionPool.java:153)
marquez-api | ! at org.apache.tomcat.jdbc.pool.DataSourceProxy.pCreatePool(DataSourceProxy.java:118)
marquez-api | ! at org.apache.tomcat.jdbc.pool.DataSourceProxy.createPool(DataSourceProxy.java:107)
marquez-api | ! at org.apache.tomcat.jdbc.pool.DataSourceProxy.getConnection(DataSourceProxy.java:131)
marquez-api | ! at org.flywaydb.core.internal.jdbc.JdbcUtils.openConnection(JdbcUtils.java:48)
marquez-api | ! at org.flywaydb.core.internal.jdbc.JdbcConnectionFactory.<init>(JdbcConnectionFactory.java:75)
marquez-api | ! at org.flywaydb.core.FlywayExecutor.execute(FlywayExecutor.java:147)
marquez-api | ! at org.flywaydb.core.Flyway.info(Flyway.java:190)
marquez-api | ! at marquez.db.DbMigration.hasPendingDbMigrations(DbMigration.java:73)
marquez-api | ! at marquez.db.DbMigration.migrateDbOrError(DbMigration.java:27)
marquez-api | ! at marquez.MarquezApp.run(MarquezApp.java:105)
marquez-api | ! at marquez.MarquezApp.run(MarquezApp.java:48)
marquez-api | ! at io.dropwizard.cli.EnvironmentCommand.run(EnvironmentCommand.java:67)
marquez-api | ! at io.dropwizard.cli.ConfiguredCommand.run(ConfiguredCommand.java:98)
marquez-api | ! at io.dropwizard.cli.Cli.run(Cli.java:78)
marquez-api | ! at io.dropwizard.Application.run(Application.java:94)
marquez-api | ! at marquez.MarquezApp.main(MarquezApp.java:60)
marquez-api | INFO [2023-10-27 21:34:58,024] marquez.MarquezApp: Stopping app...
marquez-api | ! org.postgresql.util.PSQLException: FATAL: password authentication failed for user "marquez"
marquez-api | ! at org.postgresql.core.v3.ConnectionFactoryImpl.doAuthentication(ConnectionFactoryImpl.java:693)
marquez-api | ! at org.postgresql.core.v3.ConnectionFactoryImpl.tryConnect(ConnectionFactoryImpl.java:203)
marquez-api | ! at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:258)
marquez-api | ! at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:54)
marquez-api | ! at org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:253)
marquez-api | ! at org.postgresql.Driver.makeConnection(Driver.java:434)
marquez-api | ! at org.postgresql.Driver.connect(Driver.java:291)
marquez-api | ! at org.apache.tomcat.jdbc.pool.PooledConnection.connectUsingDriver(PooledConnection.java:346)
marquez-api | ! at org.apache.tomcat.jdbc.pool.PooledConnection.connect(PooledConnection.java:227)
marquez-api | ! at org.apache.tomcat.jdbc.pool.ConnectionPool.createConnection(ConnectionPool.java:768)
marquez-api | ! at org.apache.tomcat.jdbc.pool.ConnectionPool.borrowConnection(ConnectionPool.java:696)
marquez-api | ! at org.apache.tomcat.jdbc.pool.ConnectionPool.init(ConnectionPool.java:495)
marquez-api | ! at org.apache.tomcat.jdbc.pool.ConnectionPool.<init>(ConnectionPool.java:153)
marquez-api | ! at org.apache.tomcat.jdbc.pool.DataSourceProxy.pCreatePool(DataSourceProxy.java:118)
marquez-api | ! at org.apache.tomcat.jdbc.pool.DataSourceProxy.createPool(DataSourceProxy.java:107)
marquez-api | ! at org.apache.tomcat.jdbc.pool.DataSourceProxy.getConnection(DataSourceProxy.java:131)
marquez-api | ! at org.flywaydb.core.internal.jdbc.JdbcUtils.openConnection(JdbcUtils.java:48)
marquez-api | ! at org.flywaydb.core.internal.jdbc.JdbcConnectionFactory.<init>(JdbcConnectionFactory.java:75)
marquez-api | ! at org.flywaydb.core.FlywayExecutor.execute(FlywayExecutor.java:147)
marquez-api | ! at org.flywaydb.core.Flyway.info(Flyway.java:190)
marquez-api | ! at marquez.db.DbMigration.hasPendingDbMigrations(DbMigration.java:73)
marquez-api | ! at marquez.db.DbMigration.migrateDbOrError(DbMigration.java:27)
marquez-api | ! at marquez.MarquezApp.run(MarquezApp.java:105)
marquez-api | ! at marquez.MarquezApp.run(MarquezApp.java:48)
marquez-api | ! at io.dropwizard.cli.EnvironmentCommand.run(EnvironmentCommand.java:67)
marquez-api | ! at io.dropwizard.cli.ConfiguredCommand.run(ConfiguredCommand.java:98)
marquez-api | ! at io.dropwizard.cli.Cli.run(Cli.java:78)
marquez-api | ! at io.dropwizard.Application.run(Application.java:94)
marquez-api | ! at marquez.MarquezApp.main(MarquezApp.java:60)
marquez-api | INFO [2023-10-27 21:34:58,024] marquez.MarquezApp: Stopping app...
Willy Lulciuc
debugging docker issues like this is so difficult
Willy Lulciuc
it could be a number of things, but you are connected to the database itâs just that the
marquez user hasnât been createdWilly Lulciuc
so itâs possible that the script isnât running for whatever reason on your Ec2 instance
Willy Lulciuc
do you have other services running on that Ec2 instance? Like, other than Marquez
Willy Lulciuc
is there a postgres process running outside of docker?
Kavitha
no other services except marquez on this EC2 instance
Kavitha
this was a new Ec2 instance that was spun up to install and use marquez
Kavitha
n we can confirm that no postgres process runs outside of docker
Jason Yip
I realize in Spark 3.4+, some job ids don't have a start event. What part of the code is responsible for triggering the START and COMPLETE event
PaweĆ LeszczyĆski
hi @Jason Yip could you provide an example of such a job?
Jason Yip
@PaweĆ LeszczyĆski same old:
# delete the old table if needed
_ = spark.sql('DROP TABLE IF EXISTS transactions')
# expected structure of the file
transactions_schema = StructType([
StructField('household_id', IntegerType()),
StructField('basket_id', LongType()),
StructField('day', IntegerType()),
StructField('product_id', IntegerType()),
StructField('quantity', IntegerType()),
StructField('sales_amount', FloatType()),
StructField('store_id', IntegerType()),
StructField('discount_amount', FloatType()),
StructField('transaction_time', IntegerType()),
StructField('week_no', IntegerType()),
StructField('coupon_discount', FloatType()),
StructField('coupon_discount_match', FloatType())
])
# read data to dataframe
df = (spark
.read
.csv(
adlsRootPath + '/examples/data/csv/completejourney/transaction_data.csv',
header=True,
schema=transactions_schema))
df.write\
.format('delta')\
.mode('overwrite')\
.option('overwriteSchema', 'true')\
.option('path', adlsRootPath + '/examples/data/csv/completejourney/silver/transactions')\
.saveAsTable('transactions')
df.count()
# # create table object to make delta lake queryable
# _ = spark.sql(f'''
# CREATE TABLE transactions
# USING DELTA
# LOCATION '{adlsRootPath}/examples/data/csv/completejourney/silver/transactions'
# ''')
# show data
display(
spark.table('transactions')
)
# delete the old table if needed
_ = spark.sql('DROP TABLE IF EXISTS transactions')
# expected structure of the file
transactions_schema = StructType([
StructField('household_id', IntegerType()),
StructField('basket_id', LongType()),
StructField('day', IntegerType()),
StructField('product_id', IntegerType()),
StructField('quantity', IntegerType()),
StructField('sales_amount', FloatType()),
StructField('store_id', IntegerType()),
StructField('discount_amount', FloatType()),
StructField('transaction_time', IntegerType()),
StructField('week_no', IntegerType()),
StructField('coupon_discount', FloatType()),
StructField('coupon_discount_match', FloatType())
])
# read data to dataframe
df = (spark
.read
.csv(
adlsRootPath + '/examples/data/csv/completejourney/transaction_data.csv',
header=True,
schema=transactions_schema))
df.write\
.format('delta')\
.mode('overwrite')\
.option('overwriteSchema', 'true')\
.option('path', adlsRootPath + '/examples/data/csv/completejourney/silver/transactions')\
.saveAsTable('transactions')
df.count()
# # create table object to make delta lake queryable
# _ = spark.sql(f'''
# CREATE TABLE transactions
# USING DELTA
# LOCATION '{adlsRootPath}/examples/data/csv/completejourney/silver/transactions'
# ''')
# show data
display(
spark.table('transactions')
)
John Lukenoff
đ Hi team, cross-posting from the Marquez Channel in case anyone here has a better idea of the spec
> For most of our lineage extractors in airflow, we are using the rust sql parser from openlineage-sql to extract table lineage via sql statements. When errors occur we are adding an
>
> I would still expect to see the other lineage we sent for the run but am instead just seeing the
> For most of our lineage extractors in airflow, we are using the rust sql parser from openlineage-sql to extract table lineage via sql statements. When errors occur we are adding an
extractionError run facet similar to what is being done here. Iâm finding in the case that multiple statements were extracted but one failed to parse while many others were successful, the lineage for these runs doesnât appear as expected in Marquez. Is there any logic around the extractionError run facet that could be causing this? It seems reasonable to assume that we might take this to mean the entire run event is invalid if we have any extraction errors. >
> I would still expect to see the other lineage we sent for the run but am instead just seeing the
extractionError in the marquez UI, in the database, runs with an extractionError facet donât seem to make it to the job_versions_io_mapping tableMaciej Obuchowski
Can you show the actual event? Should be in the events tab in Marquez
Kavitha
@John Lukenoff, would you mind posting the link to Marquez teams slack channel?
John Lukenoff
yep here is the link: https://marquezproject.slack.com/archives/C01E8MQGJP7/p1698702140709439
This is the full event, sanitized of internal info:
This is the full event, sanitized of internal info:
{
- "job": {
- "name": "some_dag.some_task",
- "facets": {},
- "namespace": "default"
- },
- "run": {
- "runId": "a9565df2-f1a1-3ee3-b202-7626f8c4b92d",
- "facets": {
- "extractionError": {
- "errors": [
- {
- "task": "ALTER SESSION UNSET QUERY_TAG;",
- "_producer": "<https://github.com/OpenLineage/OpenLineage/tree/0.24.0/client/python>",
- "_schemaURL": "<https://raw.githubusercontent.com/OpenLineage/OpenLineage/main/spec/OpenLineage.json#/definitions/BaseFacet>",
- "taskNumber": 0,
- "errorMessage": "Expected one of TABLE or INDEX, found: SESSION"
- }
- ],
- "_producer": "<https://github.com/OpenLineage/OpenLineage/tree/0.24.0/client/python>",
- "_schemaURL": "<https://raw.githubusercontent.com/OpenLineage/OpenLineage/main/spec/OpenLineage.json#/definitions/ExtractionErrorRunFacet>",
- "totalTasks": 1,
- "failedTasks": 1
- }
- }
- },
- "inputs": [
- {
- "name": "foo.bar",
- "facets": {},
- "namespace": "snowflake"
- },
- {
- "name": "fizz.buzz",
- "facets": {},
- "namespace": "snowflake"
- }
- ],
- "outputs": [
- { "name": "foo1.bar2", "facets": {}, "namespace": "snowflake" },
- {
- "name": "fizz1.buzz2",
- "facets": {},
- "namespace": "snowflake"
- }
- ],
- "producer": "<https://github.com/MyCompany/repo/blob/next-master/company/data/pipelines/airflow_utils/openlineage_utils/client.py>",
- "eventTime": "2023-10-30T02:46:13.367274Z",
- "eventType": "COMPLETE"
-}Kavitha
thank you!
Kavitha
@John Lukenoff, sorry to trouble again, is the slack channel still active? for whatever reason i cant get to this workspace
John Lukenoff
yep itâs still active, maybe you need to join the workspace first? https://join.slack.com/t/marquezproject/shared_invite/zt-266fdhg9g-TE7e0p~EHK50GJMMqNH4tg
Kavitha
that was a good call. the link you just shared worked! thank you!
Maciej Obuchowski
yeah from OL perspective this looks good - the inputs and outputs are there, the extraction error facet looks like it should
Maciej Obuchowski
must be some Marquez hiccup đ
đ1
John Lukenoff
Makes sense, Iâll tail my marquez logs today to see if I can find anything
John Lukenoff
Somehow this started working after we switched from our beta to prod infrastructure. I suspect something was failing due to constraints on the size of our db and the load of poor quality data it was under after months of testing against it
harsh loomba
one question if someone is around - when im keeping both
any thoughts?
openlineage-airflow and apache-airflow-providers-openlineage in my requirement file, i see the following error - from openlineage.airflow.extractors import Extractors
-ModuleNotFoundError: No module named 'openlineage.airflow'any thoughts?
Michael Robinson
@channel
Iâm opening a vote to release OpenLineage 1.5.0, including:
âą support for Cassandra Connectors lineage in the Flink integration
âą support for Databricks Runtime 13.3 in the Spark integration
âą support for
âą lowered requirements for attrs and requests packages in the Airflow integration
âą lazy rendering of yaml configs in the dbt integration
âą bug fixes, tests, infra fixes, doc changes, and more.
Three +1s from committers will authorize an immediate release.
Iâm opening a vote to release OpenLineage 1.5.0, including:
âą support for Cassandra Connectors lineage in the Flink integration
âą support for Databricks Runtime 13.3 in the Spark integration
âą support for
rdd and toDF operations from the Spark Scala API in Sparkâą lowered requirements for attrs and requests packages in the Airflow integration
âą lazy rendering of yaml configs in the dbt integration
âą bug fixes, tests, infra fixes, doc changes, and more.
Three +1s from committers will authorize an immediate release.
â6
đ1
đ2
Michael Robinson
Thanks, all. The release is authorized and will be initiated within 2 business days.
Michael Robinson
@channel
The October 2023 issue of OpenLineage News is available now! Sign up to get in directly in your inbox each month.
The October 2023 issue of OpenLineage News is available now! Sign up to get in directly in your inbox each month.
đ2
đ1
John Lukenoff
Hi team đ , weâre finding that for our Spark jobs we are almost always getting some junk characters in our dataset names. Weâve pushed the regex filter to its limits and would like to extend the logic of deriving the dataset name in openlineage-spark (currently on
1.4.1). I seem to recall hearing we could do this by implementing our own LogicalPlanVisitor or something along those lines? Is that still the recommended approach and if so would this be possible to implement in Scala vs. Java (scala noob here :simple_smile:)PaweĆ LeszczyĆski
Hi John, we're always happy to help with the contribution.
One of the possible solutions to this would be to do that just in
âą introduce config entry like
âą modify
âą additionally, you would need to add config entry in
âą document this in the doc
So, no Scala nor custom logical plan visitors required.
One of the possible solutions to this would be to do that just in
openlineage-java client:âą introduce config entry like
normalizeDatasetNameToAscii : enabled/disabledâą modify
DatasetIdentifier class to contain static member boolean normalizeDatasetNameToAscii and normalize dataset name according to this settingâą additionally, you would need to add config entry in
io.openlineage.client.OpenLineageYaml and make sure both loadOpenLineageYaml methods set DatasetIdentifier.normalizeDatasetNameToAscii based on the configâą document this in the doc
So, no Scala nor custom logical plan visitors required.
Mike Fang
I am looking to send OpenLineage events to an AWS API Gateway endpoint from an AWS MWAA instance. The problem is that all requests to AWS services need to be signed with SigV4, and using API Gateway with IAM authentication would require requests to API Gateway be signed with SigV4. Would the best way to do so be to just modify the python client HTTP transport to include a new config option for signing emitted OpenLineage events with SigV4? Are there any alternatives?
Jakub DardziĆski
thereâs actually an issue for that:
https://github.com/OpenLineage/OpenLineage/issues/2189
but the way to do this is imho to create new custom transport (it might inherit from HTTP transport) and register it in transport factory
https://github.com/OpenLineage/OpenLineage/issues/2189
but the way to do this is imho to create new custom transport (it might inherit from HTTP transport) and register it in transport factory
Mike Fang
I am thinking of just modifying the HTTP transport and using requests.auth.AuthBase to create different auth methods instead of a TokenProvider class
Classes which subclass requests.auth.AuthBase can also just directly be given to the requests call in the auth parameter
Classes which subclass requests.auth.AuthBase can also just directly be given to the requests call in the auth parameter
đ1
Jakub DardziĆski
would you like to contribute? đ
Mike Fang
I was about to contribute, but I actually just realized that there is an existing way to provide a custom transport that would solve form y use case. My only question is how do I register this custom transport in my MWAA environment? Can I provide the custom transport as an Airflow plugin and then specify the class in the Openlineage.yml config? Will it automatically pick it up?
Jakub DardziĆski
although I did not test this in MWAA but locally only: Iâve created Airflow plugin that in
setting
__init__.py has defined (or imported) following code:from openlineage.client.transport import register_transport, Transport, Config
-
-
-@register_transport
-class FakeTransport(Transport):
- kind = "fake"
- config = Config
-
- def __init__(self, config: Config) -> None:
- print(config)
-
- def emit(self, event) -> None:
- print(event)setting
AIRFLOW__OPENLINEAGE__TRANSPORT='{"type": "fake"}' does take effect and I can see output in Airflow logsJakub DardziĆski
in
setup.py itâs: ...,
- entry_points={
- 'airflow.plugins': [
- 'custom_transport = custom_transport:CustomTransportPlugin',
- ],
- },
- install_requires=["openlineage-python"]
-)Mike Fang
ok great thanks for following up on this, super helpful
Michael Robinson
@channel
We released OpenLineage 1.5.0, including:
âą support for Cassandra Connectors lineage in the Flink integration by @U05QA2D1XNV
âą support for Databricks Runtime 13.3 in the Spark integration by @PaweĆ LeszczyĆski
âą support for
âą lowered requirements for attrs and requests packages in the Airflow integration by @Jakub DardziĆski
âą lazy rendering of yaml configs in the dbt integration by @Jakub DardziĆski
âą bug fixes, tests, infra fixes, doc changes, and more.
Thanks to all the contributors, including new contributor @U05VDHJJ9T7!
Release: https://github.com/OpenLineage/OpenLineage/releases/tag/1.5.0
Changelog: https://github.com/OpenLineage/OpenLineage/blob/main/CHANGELOG.md
Commit history: https://github.com/OpenLineage/OpenLineage/compare/1.4.1...1.5.0
Maven: https://oss.sonatype.org/#nexus-search;quick~openlineage
PyPI: https://pypi.org/project/openlineage-python/
We released OpenLineage 1.5.0, including:
âą support for Cassandra Connectors lineage in the Flink integration by @U05QA2D1XNV
âą support for Databricks Runtime 13.3 in the Spark integration by @PaweĆ LeszczyĆski
âą support for
rdd and toDF operations from the Spark Scala API in Spark by @PaweĆ LeszczyĆski âą lowered requirements for attrs and requests packages in the Airflow integration by @Jakub DardziĆski
âą lazy rendering of yaml configs in the dbt integration by @Jakub DardziĆski
âą bug fixes, tests, infra fixes, doc changes, and more.
Thanks to all the contributors, including new contributor @U05VDHJJ9T7!
Release: https://github.com/OpenLineage/OpenLineage/releases/tag/1.5.0
Changelog: https://github.com/OpenLineage/OpenLineage/blob/main/CHANGELOG.md
Commit history: https://github.com/OpenLineage/OpenLineage/compare/1.4.1...1.5.0
Maven: https://oss.sonatype.org/#nexus-search;quick~openlineage
PyPI: https://pypi.org/project/openlineage-python/
đ5
đ1
Jason Yip
@PaweĆ LeszczyĆski I tested 1.5.0, it works great now, but the environment facets is gone in START... which I very much want it.. any thoughts?
Jason Yip
actually, it shows up in one of the RUNNING now... behavior is consistent between 11.3 and 13.3, thanks for fixing this issue
đ1
Jason Yip
@PaweĆ LeszczyĆski looks like I need to bring bad news.. 13.3 is fixed for specific scenarios, but 11.3 is still reading output as dbfs.. there are scenarios that it's not producing input and output like:
create table table using delta as
location 'abfss://....'
Select * from parquet.`abfss://....'
create table table using delta as
location 'abfss://....'
Select * from parquet.`abfss://....'
Jason Yip
Will test more and ope issues
Rodrigo Maia
@Jason Yiphow did you manage the get the environment attribute. it's not showing up to me at all. I've tried databricks abut also tried a local instance of spark.
Jason Yip
@Rodrigo Maia its showing up in one of the RUNNING events, not in the START event anymore
Rodrigo Maia
I never had a running event :melting_face: Am I filtering something?
Jason Yip
Umm.. ok show me your code, will try on my end
Jason Yip
@PaweĆ LeszczyĆski @Rodrigo Maia actually if you are using UC-enabled cluster, you won't get any RUNNING events
Michael Robinson
@channel
This monthâs TSC meeting (open to all) is next Thursday the 9th at 10am PT. On the agenda:
âą announcements
âą recent releases
âą recent additions to the Flink integration by @U05QA2D1XNV
âą recent additions to the Spark integration by @PaweĆ LeszczyĆski
âą updates on proposals by @Julien Le Dem
âą discussion topics
âą open discussion
More info and the meeting link can be found on the website. All are welcome! Do you have a discussion topic, use case or integration youâd like to demo? DM me to be added to the agenda.
This monthâs TSC meeting (open to all) is next Thursday the 9th at 10am PT. On the agenda:
âą announcements
âą recent releases
âą recent additions to the Flink integration by @U05QA2D1XNV
âą recent additions to the Spark integration by @PaweĆ LeszczyĆski
âą updates on proposals by @Julien Le Dem
âą discussion topics
âą open discussion
More info and the meeting link can be found on the website. All are welcome! Do you have a discussion topic, use case or integration youâd like to demo? DM me to be added to the agenda.
đ1
priya narayana
Hi Team , we are trying to customize the events by writing custom lineage listener extending OpenLineageSparkListener, but would need some direction how to capture the events
Jakub DardziĆski
https://openlineage.slack.com/archives/C01CK9T7HKR/p1698315220142929
Do you need some more guidance than that?
Do you need some more guidance than that?
priya narayana
yes
Jakub DardziĆski
It seems pretty extensively described, what kind of help do you need?
priya narayana
io.openlineage.spark.api.OpenLineageEventHandlerFactory if i use this how will i pass custom listener to my spark submitpriya narayana
I would like to know how will i customize my events using this . For example: - In "input" Facet i want only symlinks name i am not intereseted in anything else
priya narayana
can you please provide some guidance
priya narayana
@Jakub DardziĆski this is the doubt i have
priya narayana
Some one who did spark integration throw some light
Jakub DardziĆski
it's weekend for most of us so you probably need to wait until Monday for precise answers
David Goss
đ I raised a PR https://github.com/OpenLineage/OpenLineage/pull/2223 off the back of some Marquez conversations a while back to try and clarify how names of Snowflake objects should be expressed in OL events. I used Snowflakeâs OL view as a guide, but also I appreciate there are other OL producers that involve Snowflake too (Airflow? dbt?). Any feedback on this would be appreciated!
David Goss
Thanks for merging this @Maciej Obuchowski!
đ1
Athitya Kumar
Hey team! đ
We're trying to use openlineage-flink, and would like provide the
For example, in spark, the changes mostly were delegated to the spark-submit command like
And the
TIA! đ
We're trying to use openlineage-flink, and would like provide the
openlineage.transport.type=http and configure other transport configs, but we're not able to find sufficient docs (tried this doc) on where/how these configs can be provided.For example, in spark, the changes mostly were delegated to the spark-submit command like
spark-submit --conf "spark.extraListeners=io.openlineage.spark.agent.OpenLineageSparkListener" \
- --packages "io.openlineage:openlineage-spark:<spark-openlineage-version>" \
- --conf "spark.openlineage.transport.url=http://{openlineage.client.host}/api/v1/namespaces/spark_integration/" \
- --class com.mycompany.MySparkApp my_application.jarAnd the
OpenLineageSparkListener has a method to retrieve the provided spark confs as an object in the ArgumentParser. Similarly, looking for some pointers on how the openlineage.transport configs can be provided to OpenLineageFlinkJobListener & how the flink listener parses/uses these configsTIA! đ
Maciej Obuchowski
similarly to spark config, you can use flink config
Athitya Kumar
@Maciej Obuchowski - Got it. Our use-case is that we're trying to build a wrapper on top of openlineage-flink for productionising for our flink jobs.
We're trying to have a wrapper class that extends OpenLineageFlinkJobListener class, and overwrites the HTTP transport endpoint/url to a constant value (say, example.com and /api/v1/flink). But we see that the OpenLineageFlinkJobListener constructor is defined as a private constructor - just wanted to check with the team whether it was just a default scope, or intended to be private. If it was just a default scope, can we contribute a PR to make it public, to make it friendly for teams trying to adopt & extend openlineage?
And also, we wanted to understand better on where we're reading the HTTP transport endpoint/url configs in OpenLineageFlinkJobListener and what'd be the best place to override it to the constant endpoint/url for our use-case
We're trying to have a wrapper class that extends OpenLineageFlinkJobListener class, and overwrites the HTTP transport endpoint/url to a constant value (say, example.com and /api/v1/flink). But we see that the OpenLineageFlinkJobListener constructor is defined as a private constructor - just wanted to check with the team whether it was just a default scope, or intended to be private. If it was just a default scope, can we contribute a PR to make it public, to make it friendly for teams trying to adopt & extend openlineage?
And also, we wanted to understand better on where we're reading the HTTP transport endpoint/url configs in OpenLineageFlinkJobListener and what'd be the best place to override it to the constant endpoint/url for our use-case
Maciej Obuchowski
We parse flink conf to get that information: https://github.com/OpenLineage/OpenLineage/blob/26494b596e9669d2ada164066a73c44e04[âŠ]ink/src/main/java/io/openlineage/flink/client/EventEmitter.java
> But we see that the OpenLineageFlinkJobListener constructor is defined as a private constructor - just wanted to check with the team whether it was just a default scope, or intended to be private.
The way to construct is is a public builder in the same class
I think easier way than wrapper class would be use existing flink configuration, or to set up
> But we see that the OpenLineageFlinkJobListener constructor is defined as a private constructor - just wanted to check with the team whether it was just a default scope, or intended to be private.
The way to construct is is a public builder in the same class
I think easier way than wrapper class would be use existing flink configuration, or to set up
OPENLINEAGE_URL env variable, or have openlineage.yml config file - not sure why this is the way you've chosen?Athitya Kumar
For example, if we wanna move away from http transport to kafka transport - we'd be changing only this wrapper class and ask folks to update their wrapper class dependency version. If we went without the wrapper class, then the exact config changes would need to be synced and done by many different teams, who may not have enough context.
Similarly, if we wanna enable some other default best-practise configs, or inject any company-specific configs etc, the wrapper would be useful in abstracting out the details and be the 1 place that handles all openlineage related integrations for any future changes.
That's why we wanna extend openlineage's listener class & leverage most of the OSS code as-is; and at the same time, have the ability to extend & inject customisations. I think that's where some things like having getters for the class object attributes, or having public constructors would be really helpful đ
I think easier way than wrapper class would be use existing flink configuration, or to set up@Maciej Obuchowski - The reasoning behind going with a wrapper class is that we can abstract out the nitty-gritty like how/where we're publishing openlineage events etc - especially for companies that have a lot of teams that may be adopting openlineage.OPENLINEAGE_URLenv variable, or haveopenlineage.ymlconfig file - not sure why this is the way you've chosen?
For example, if we wanna move away from http transport to kafka transport - we'd be changing only this wrapper class and ask folks to update their wrapper class dependency version. If we went without the wrapper class, then the exact config changes would need to be synced and done by many different teams, who may not have enough context.
Similarly, if we wanna enable some other default best-practise configs, or inject any company-specific configs etc, the wrapper would be useful in abstracting out the details and be the 1 place that handles all openlineage related integrations for any future changes.
That's why we wanna extend openlineage's listener class & leverage most of the OSS code as-is; and at the same time, have the ability to extend & inject customisations. I think that's where some things like having getters for the class object attributes, or having public constructors would be really helpful đ
Maciej Obuchowski
@Athitya Kumar that makes sense. Feel free to provide PR adding getters and stuff.
đ1
Yannick Libert
Hi all, we (I work with @U05VDHJJ9T7 and @Abdallah) have a quick question regarding the spark integration:
if a spark app contains several jobs, they will be named "my_spark_app_name.job1" and "my_spark_app_name.job2"
eg:
spark_job.collect_limit
spark_job.map_partitions_parallel_collection
If I understood correctly, the spark integration maps one Spark job to a single OpenLineage Job, and the application itself should be assigned a Run id at startup and each job that executes will report the application's Run id as its parent job run (taken from: https://openlineage.io/docs/integrations/spark/).
In our case, the app Run Id is never created, and the jobs runs don't contain any parent facets. We tested it with a recent integration version in 1.4.1 and also an older one (0.26.0).
Did we miss something in the OL spark integration config?
if a spark app contains several jobs, they will be named "my_spark_app_name.job1" and "my_spark_app_name.job2"
eg:
spark_job.collect_limit
spark_job.map_partitions_parallel_collection
If I understood correctly, the spark integration maps one Spark job to a single OpenLineage Job, and the application itself should be assigned a Run id at startup and each job that executes will report the application's Run id as its parent job run (taken from: https://openlineage.io/docs/integrations/spark/).
In our case, the app Run Id is never created, and the jobs runs don't contain any parent facets. We tested it with a recent integration version in 1.4.1 and also an older one (0.26.0).
Did we miss something in the OL spark integration config?
PaweĆ LeszczyĆski
hey, a name of the output dataset should be put at the end of the job name. This was introduced to help with jobs that call multiple spark actions
Yannick Libert
Hi PaweĆ,
Thanks for your answer, yes indeed with the newer version of OL, we automatically have the name of the output dataset at the end of the job name, but no App run id, nor any parent run facet.
Thanks for your answer, yes indeed with the newer version of OL, we automatically have the name of the output dataset at the end of the job name, but no App run id, nor any parent run facet.
PaweĆ LeszczyĆski
yes, you're right. I mean you can set in config
spark.openlineage.parentJobName which will be shared through whole app run, but this needs to be set manuallyYannick Libert
I see, thanks a lot for your reply we'll try that
ldacey
if I have a dataset on adls gen2 which synapse connects to as an external delta table, is that the use case of a symlink dataset? the delta table is connected to by PBI and by Synapse, but the underlying data is exactly the same
Maciej Obuchowski
Sounds like it, yes - if the logical dataset names are different but physical one is the same
Rodrigo Maia
Has anyone here tried OpenLineage with Spark on Amazon EMR?
Jason Yip
No but it should work the same I tried on AWS and Google Colab and Azure
đ1
Michael Robinson
@channel
Friendly reminder: this monthâs TSC meeting, open to all, is tomorrow at 10 am PT: https://openlineage.slack.com/archives/C01CK9T7HKR/p1699027207361229
Friendly reminder: this monthâs TSC meeting, open to all, is tomorrow at 10 am PT: https://openlineage.slack.com/archives/C01CK9T7HKR/p1699027207361229
đ1
Jason Yip
@PaweĆ LeszczyĆski regarding to https://github.com/OpenLineage/OpenLineage/issues/2124, OL is parsing out the table location in Hive metastore, it is the location of the table in the catalog and not the physical location of the data. It is both right and wrong because it is a table, just it is an external table.
https://docs.databricks.com/en/sql/language-manual/sql-ref-external-tables.html
https://docs.databricks.com/en/sql/language-manual/sql-ref-external-tables.html
Jason Yip
Here's for more reference: https://dilorom.medium.com/finding-the-path-to-a-table-in-databricks-2c74c6009dbb
Jason Yip
@PaweĆ LeszczyĆski this is why if create a table with adls location it won't show input and output:
https://github.com/OpenLineage/OpenLineage/blob/main/integration/spark/spark35/src[âŠ]k35/agent/lifecycle/plan/CreateReplaceOutputDatasetBuilder.java
Because the catalog object is not there.
https://github.com/OpenLineage/OpenLineage/blob/main/integration/spark/spark35/src[âŠ]k35/agent/lifecycle/plan/CreateReplaceOutputDatasetBuilder.java
Because the catalog object is not there.
Jason Yip
Databricks needs to be re-written in a way that supports Databricks it seems like
Jason Yip
@PaweĆ LeszczyĆski I went back to 1.4.1, output does show adls location. But environment facet is gone in 1.4.1. It shows up in 1.5.0 but namespace is back to dbfs....
Jason Yip
@PaweĆ LeszczyĆski I diff CreateReplaceDatasetBuilder.java and CreateReplaceOutputDatasetBuilder.java and they are the same except for the class name, so I am not sure what is causing the change. I also realize you don't have a test case for ADLS
PaweĆ LeszczyĆski
Thanks @Jason Yip for your engagement in finding the cause and solution to this issue.
Among the technical problems, another problem here is that our databricks integration tests are run on AWS and the issue you describe occurs in Azure. I would consider this a primary issue as it is difficult for me to verify the behaviour you describe and fix it with a failing integration test at the start.
Are you able to reproduce the issue on AWS Databricks environment so that we could include it in our integration tests and make sure the behvaiour will not change later on in future?
Among the technical problems, another problem here is that our databricks integration tests are run on AWS and the issue you describe occurs in Azure. I would consider this a primary issue as it is difficult for me to verify the behaviour you describe and fix it with a failing integration test at the start.
Are you able to reproduce the issue on AWS Databricks environment so that we could include it in our integration tests and make sure the behvaiour will not change later on in future?
Jason Yip
I didn't know Azure and AWS Databricks are different. Let me try it on AWS as well
Naresh reddy
Hi
Can anyone point me to the deck on how Airflow can be integrated using Openlineage?
Can anyone point me to the deck on how Airflow can be integrated using Openlineage?
Naresh reddy
thank you @Maciej Obuchowski
Naresh reddy
Can anyone tell me why OL is better than other competitors if you can provide an analysis that would be great
Harel Shein
Hey @Naresh reddy can you help me understand what you mean by competitors?
OL is a specification that can be used to solve various problems, so if you have a clear problem statement, maybe I can help with pros/cons for that problem
OL is a specification that can be used to solve various problems, so if you have a clear problem statement, maybe I can help with pros/cons for that problem
Naresh reddy
what are the pros and cons of OL. we often talk about positives to market it but what are the pain points using OL,how it's addressing user issues?
Michael Robinson
Hi @Naresh reddy, thanks for your question. Weâve heard that OpenLineage is attractive because of its desirable integrations, including a best-in-class Spark integration, its extensibility, the fact that itâs not destructive, and the fact that itâs open source. Iâm not aware of pain points per se, but there are certainly features and integrations that we wish we could focus on but canât at the moment â like the Dagster integration, which needs a new maintainer. OpenLineage is like any other open standard in that ecosystem coverage is a constant process rather than a journey, and it requires contributions in order to get close to 100%. Thankfully, we are gaining users and contributors all the time, and integrations are being added or improved upon daily. See the Ecosystem page on the website for a list of consumers and producers and links to more resources, and check out the GitHub repo for the codebase, commit history, contributors, governance procedures, and more. Weâre quick to respond to messages here and issues on GitHub â usually within one day.
