Implement SLASensor #177
Open
Implement SLASensor #177
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
chaelant
requested review from
asingamaneni,
stikkireddy,
newfront and
a team
as code owners
Signed-off-by: Michael Espiritu <[email protected]>
asingamaneni
reviewed
Nov 8, 2024
| databricks_token=api_token_key, | ||
| custom_description="message to provide additional context", | ||
| slack_webhook_url="https://hooks.slack.com/your/webhook/url", | ||
| email_params={ |
There was a problem hiding this comment.
Can we make email_params in the class initialization as a pydantic object or a type object to validate the email_params.
There was a problem hiding this comment.
Hi, @asingamaneni - updated sla_sensor.py to have an EmailParams pydantic object and use that to validate the email params passed to the SLASensor in the initialization. Will throw a ValidationError if missing required fields
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## main #177 +/- ##
==========================================
+ Coverage 90.33% 92.08% +1.75%
==========================================
Files 22 22
Lines 3384 4322 +938
==========================================
+ Hits 3057 3980 +923
- Misses 327 342 +15 ☔ View full report in Codecov by Sentry. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
This implements an SLASensor to monitor the on-time performance of Databricks workflows. It is a special instance of the WorkflowTaskDependencySensor that accepts an expected SLA time in the form of a UTC datetime object or a quartz cron expression in the workflow tags. The user provides the target workflow name and the task to monitor in that workflow that would indicate the workflow has completed. If the workflow does not finish by the specified time, an alert will be sent to a user-provided email and/or slack channel.
Related Issue
This PR closes #166.
Motivation and Context
Current alerting functionality only allows for alerts to be sent if a workflow runs longer than a specified amount of time. This change adds the functionality to be able to be alerted if a workflow does not finish by a certain time, regardless of when the workflow was kicked off.
How Has This Been Tested?
These changes were tested with
make testand by building a wheel file and testing within a workflow. This implementation does not affect other features of brickflow.Screenshots (if appropriate):
Types of changes
Checklist: