Feature Implementation: AWS Glue Job Execution Support #308

knakazawa99 · 2023-09-04T22:22:08Z

This pull request implements the ability to execute AWS Glue Jobs directly from Prefect. This feature was developed based on the existing ECS Task program and is intended to provide a powerful preprocessor for data used in aggregate processing and machine learning models.

Closes #307

Example

from prefect import flow
from prefect_aws import AwsCredentials
from prefect_aws.glue_job import GlueJobBlock


@flow
def example_run_glue_job():
    aws_credentials = AwsCredentials(
        aws_access_key_id="your_access_key_id",
        aws_secret_access_key="your_secret_access_key"
    )
    glue_job_run = GlueJobBlock(
        job_name="your_glue_job_name",
        arguments={"--YOUR_EXTRA_ARGUMENT": "YOUR_EXTRA_ARGUMENT_VALUE"},
    ).trigger()

    return glue_job_run.wait_for_completion()


example_run_glue_job()

Screenshots

Blocks Catalog

Examples Catalog

API Reference

Checklist

References any related issue by including "Closes #" or "Closes ".
- If no issue exists and your change is not a small fix, please create an issue first.
Includes tests or only affects documentation.
Passes pre-commit checks.
- Run pre-commit install && pre-commit run --all locally for formatting and linting.
Includes screenshots of documentation updates.
- Run mkdocs serve view documentation locally.
Summarizes PR's changes in CHANGELOG.md

jobs doc: https://docs.aws.amazon.com/glue/latest/dg/aws-glue-api-jobs-job.html

feliperazeek · 2023-09-14T16:58:07Z

Would love to use this!

feliperazeek · 2023-09-18T10:10:12Z

prefect_aws/glue_job.py

+                aws_secret_access_key="your_secret_access_key"
+            )
+            glue_job = GlueJob(
+                glue_job_name="your_glue_job_name",


Here you are referencing glue_job_name but I believe you are actually using job_name instead.

@feliperazeek
Thanks for your comment.
fixed it.
8f9e9af

desertaxle

Thanks for the contribution @knakazawa99! It looks like the tests are failing with a missing pyparsing dependency. You can add pyparsing to requirements.txt to ensure it gets installed with this library. Let me know when the tests are passing and I can give the change a more thorough review.

Also, we introduced a worker concept a little while ago that can be used in place of infrastructure blocks. If you're willing, it'd be great if you could also create an AWS Glue worker! It shouldn't be too tough since you've already created an infrastructure block. We have a guide on creating a worker here which will be helpful if you decide to create an AWS Glue worker.

ref: - PrefectHQ#308 (review)

knakazawa99 · 2023-10-07T06:26:50Z

@desertaxle
Thank you for your feedback!
I have noticed that the tests are failing due to a NoRegionError from the AWS SDK. This error occurs because the region is not specified when creating the AWS client in the tests.
In my local environment, the tests pass because the AWS SDK automatically uses the region specified in my local AWS configuration. However, in the GitHub Actions environment, there is no such configuration, hence the error.
To fix this, I am considering adding a region to the mock_aws_credentials used in the tests. However, I understand that this might be outside the scope of this PR.
Could you please advise if it would be acceptable to make this change within this PR?
Looking forward to your guidance on this matter.

desertaxle · 2023-10-10T20:11:19Z

To fix this, I am considering adding a region to the mock_aws_credentials used in the tests.

This fix sounds good to me!

tests/workers/test_glue_job_worker.py

desertaxle

I added a few high-level comments! I'll do another pass once the tests are passing!

prefect_aws/workers/glue_job_worker.py

desertaxle · 2023-11-13T15:43:23Z

prefect_aws/workers/glue_job_worker.py

+    arguments: Optional[dict] = Field(
+        default=None,
+        title="AWS Glue Job Arguments",
+        description="The job arguments associated with this run.",
+    )


I like how flexible this field is, but it'd be nice to have some stronger typing while maintaining flexibility. Introducing a variables class that contains some values that will often vary between deployments and creating a default template can help with that. You can checkout the ECS worker's variables class as an example.

Thanks, fix it.
Does this match what you had in mind?
e71b947

desertaxle · 2023-11-13T15:46:03Z

prefect_aws/workers/glue_job_worker.py

+    job_name: str = Field(
+        ...,
+        title="AWS Glue Job Name",
+        description="The name of the job definition to use.",
+    )


The BaseJobConfiguration class has a name attribute that automatically gets populated with the flow name. I think it'd be worthwhile to use that attribute rather than introducing another name attribute.

Suggested change

job_name: str = Field(

...,

title="AWS Glue Job Name",

description="The name of the job definition to use.",

)

job_name is necessarily to use start glue job.
https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/glue/client/start_job_run.html

Ah, does this require a Glue job to exist already? Usually, workers dynamically create jobs to execute flow runs.

I am user very interested in using this feature, I actually I have forked this into my repo already. I think it's expected that the glue job exists already.

@feliperazeek are you using the worker to run Glue jobs or the block in this PR?

The variable name is good and will work, but I think the implementation of the worker needs to change to operate the same as our other workers. I'd expect a Prefect Glue worker to create a Glue job for each flow run that it picks up and then monitor that Glue job for completion.

@knakazawa99 I think we can take this PR in one of two directions:

Update the worker and block to create new Glue jobs for executing flow runs

Remove the worker and update the block to implement the JobBlock interface instead of the Infrastructure interface. This will make it more suited for executing existing Glue jobs (which I'm realizing might have been the original intent of the PR, which I misunderstood).

Let me know which direction you'd like to go with this PR, and I will do whatever I can to help. Thank you for your patience and sticking with this PR!

@feliperazeek are you using the worker to run Glue jobs or the block in this PR?

Running glue jobs

@desertaxle
Is the following direction correct?

delete Worker

modify the part that inherits Infrastructure to match the Interface of JobBlock

Apologies for the delay. Yes, that is the direction that I'm suggesting.

@desertaxle
I have fixed it. However, the test is failing due to the boto3. I have not been able to reproduce this error in my local environment.

prefect_aws/workers/glue_job_worker.py

PrefectHQ#308 (comment)

desertaxle · 2024-02-27T19:29:26Z

Hey @knakazawa99, apologies for the delay. I pushed a change that fixes the error you saw in the test. I'll give this one more review today so we can merge it as soon as possible.

desertaxle · 2024-02-27T20:35:50Z

prefect_aws/glue_job.py

+    async def trigger(self) -> GlueJobRun:
+        """trigger for GlueJobRun"""
+        return GlueJobRun(
+            job_name=self.job_name,
+            arguments=self.arguments,
+            job_watch_poll_interval=self.job_watch_poll_interval,
+        )


I think we should start the job run here and then return the GlueJobRun object. The user can choose to wait for the job, or not, but the job should run whether or not they wait for it.

Thank you for your comment.
I misunderstood the responsibilities of Glue Job Run.
It has been corrected.

f3c7d2c

desertaxle · 2024-02-27T20:36:36Z

prefect_aws/glue_job.py

+    client: Any = Field(default=None, description="")
+    job_id: str = Field(
+        default="",
+    )


These look like they could be private fields.

The Glue Job Run responsibilities have changed and require public access.

desertaxle · 2024-02-27T20:37:53Z

prefect_aws/glue_job.py

+from prefect_aws import AwsCredentials
+
+
+class GlueJobRun(JobRun, BaseModel):


This class doesn't need to be a Pydantic BaseModel since we don't need validation. I think it might be simpler to make it a plain Python class

Suggested change

class GlueJobRun(JobRun, BaseModel):

class GlueJobRun(JobRun):

I find it useful to depend on pydantic for uniformity with other object definitions, validation, and attribute descriptions.

add region name

knakazawa99 · 2024-03-17T09:42:23Z

@desertaxle
Thank you for your comment.
I've made improvements based on your comments.

desertaxle

LGTM! Thanks so much for your patience and perseverance @knakazawa99!

knakazawa99 added 2 commits September 4, 2023 11:41

Add Glue Job

1f82b74

jobs doc: https://docs.aws.amazon.com/glue/latest/dg/aws-glue-api-jobs-job.html

Add Glue Job Doc

4fb7a2c

knakazawa99 requested a review from a team as a code owner September 4, 2023 22:22

feliperazeek reviewed Sep 18, 2023

View reviewed changes

Fix Example of PyDoc

8f9e9af

knakazawa99 requested a review from feliperazeek September 18, 2023 10:59

knakazawa99 mentioned this pull request Sep 26, 2023

Feature Request: Support for AWS Glue Job Execution #307

Closed

1 task

desertaxle reviewed Sep 28, 2023

View reviewed changes

knakazawa99 added 3 commits October 7, 2023 13:45

Add pyparsing

9d6ac4f

ref: - PrefectHQ#308 (review)

Add Glue Job Worker

5af0057

ref: - PrefectHQ#308 (review)

Change mkdocs

5c46568

knakazawa99 requested a review from desertaxle October 7, 2023 06:17

knakazawa99 added 2 commits October 11, 2023 09:27

Add region to mock_aws_credentials

7adaf5b

Add aws_default_region to mock_aws_credentials

d2183db

desertaxle reviewed Oct 13, 2023

View reviewed changes

tests/workers/test_glue_job_worker.py Outdated Show resolved Hide resolved

knakazawa99 and others added 3 commits October 14, 2023 08:25

Fix region name for glue job client test

71552b9

Merge branch 'main' into main

f44c66f

Merge branch 'main' into main

956c7da

desertaxle suggested changes Nov 13, 2023

View reviewed changes

knakazawa99 added 8 commits November 14, 2023 09:30

Add Class Doc

c726304

PrefectHQ#308 (comment)

add pydantic v2 support

bae5e12

PrefectHQ#308 (comment)

Remove optional type hint

71934c2

PrefectHQ#308 (comment)

Change dict type hint

e71b947

PrefectHQ#308 (comment)

Add module comment

68b02b8

PrefectHQ#308 (comment)

Support pydantic v2

3b501e2

Change region name

1dceb73

use aws_credentials in fixture

8390a94

knakazawa99 and others added 13 commits January 24, 2024 17:21

Remove Glue Job inherited from Infrastructure, Implement Glue Job Brock

7ea43d4

Merge remote-tracking branch 'origin/main'

491191e

Remove Glue Job Worker From Docs

9e06097

Merge branch 'main' into main

12299d4

Add aws_credentials to test_get_client

41367dd

Merge remote-tracking branch 'origin/main'

0d04479

Add aws_credentials to test_get_client

cb36769

remove aws_credentials from test_get_client

7b9cbad

fix: ref PrefectHQ@6b3ff0b

32a43c0

Merge branch 'main' into main

18e71d7

Merge branch 'main' into main

bd69c20

Updates BaseModel import

3bf96cb

Fixes syntax

f086f03

Merge branch 'main' into main

7048c2a

desertaxle suggested changes Feb 27, 2024

View reviewed changes

knakazawa99 added 8 commits March 17, 2024 16:28

Refactor: Moved responsibilities of JobRun to JobBlock

f3c7d2c

Merge remote-tracking branch 'origin/main'

705ffe4

Merge branch 'main' into main

993076b

fix degrade

d92238d

fix for ut

5a4fd2d

fix for ut

12781be

fix for ut

d5c37e2

add region name

Fix for generated doc

9f71f03

knakazawa99 requested a review from desertaxle March 17, 2024 09:42

desertaxle approved these changes Apr 24, 2024

View reviewed changes

Merge branch 'main' into main

474c045

zzstoatzz merged commit 3bb6261 into PrefectHQ:main Apr 24, 2024
7 checks passed

zzstoatzz mentioned this pull request Apr 24, 2024

Migrate prefect-aws to core PrefectHQ/prefect#12888

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Feature Implementation: AWS Glue Job Execution Support #308

Feature Implementation: AWS Glue Job Execution Support #308

knakazawa99 commented Sep 4, 2023 •

edited

Loading

feliperazeek commented Sep 14, 2023

feliperazeek Sep 18, 2023

knakazawa99 Sep 18, 2023

desertaxle left a comment

knakazawa99 commented Oct 7, 2023

desertaxle commented Oct 10, 2023 •

edited

Loading

desertaxle left a comment

desertaxle Nov 13, 2023

knakazawa99 Nov 15, 2023

desertaxle Nov 13, 2023

knakazawa99 Nov 15, 2023

desertaxle Nov 28, 2023

feliperazeek Nov 29, 2023

desertaxle Dec 7, 2023

desertaxle Dec 8, 2023

feliperazeek Dec 8, 2023

knakazawa99 Dec 20, 2023

desertaxle Jan 18, 2024

knakazawa99 Jan 25, 2024

desertaxle commented Feb 27, 2024

desertaxle Feb 27, 2024

knakazawa99 Mar 17, 2024

desertaxle Feb 27, 2024

knakazawa99 Mar 17, 2024 •

edited

Loading

desertaxle Feb 27, 2024

knakazawa99 Mar 17, 2024

knakazawa99 commented Mar 17, 2024

desertaxle left a comment

		from prefect_aws import AwsCredentials


		class GlueJobRun(JobRun, BaseModel):

	class GlueJobRun(JobRun, BaseModel):
	class GlueJobRun(JobRun):

Feature Implementation: AWS Glue Job Execution Support #308

Feature Implementation: AWS Glue Job Execution Support #308

Conversation

knakazawa99 commented Sep 4, 2023 • edited Loading

Example

Screenshots

Checklist

feliperazeek commented Sep 14, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

desertaxle left a comment

Choose a reason for hiding this comment

knakazawa99 commented Oct 7, 2023

desertaxle commented Oct 10, 2023 • edited Loading

desertaxle left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

desertaxle commented Feb 27, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

knakazawa99 Mar 17, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

knakazawa99 commented Mar 17, 2024

desertaxle left a comment

Choose a reason for hiding this comment

knakazawa99 commented Sep 4, 2023 •

edited

Loading

desertaxle commented Oct 10, 2023 •

edited

Loading

knakazawa99 Mar 17, 2024 •

edited

Loading