Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FEATURE] Create workflow with FOR EACH type #163

Open
1 of 4 tasks
Skdakua opened this issue Sep 18, 2024 · 0 comments
Open
1 of 4 tasks

[FEATURE] Create workflow with FOR EACH type #163

Skdakua opened this issue Sep 18, 2024 · 0 comments
Labels
enhancement New feature or request
Milestone

Comments

@Skdakua
Copy link

Skdakua commented Sep 18, 2024

Is your feature request related to a problem? Please describe.
This is a feature provided by Databricks to use the For each task to run a task in a loop, passing a different set of parameters to each iteration of the task.

Cloud Information

  • AWS
  • Azure
  • GCP
  • Other

Describe the solution you'd like
-Adding the For each task to a job requires defining two tasks: The For each task and a nested task. The nested task is the task to run for each iteration of the For each task and is one of the standard Databricks Jobs task types. You cannot add another For each task as the nested task.
-For example, you could use the For each task to perform a common set of transformations on multiple tables, passing a table name from a list of table names to each iteration of the task.
-Instead of creating 100 tasks for 100 table ingestion you can create one task and iterate over the list. It also helps to put a concurrency limit for your tasks which is not available in other type of jobs.
-It will also allow to run many processes using single task.

Describe alternatives you've considered
Currently we are creating 100 tasks for 100 tables ingestion.

Additional context
Add any other context or screenshots about the feature request here.

@Skdakua Skdakua added the enhancement New feature or request label Sep 18, 2024
@asingamaneni asingamaneni added this to the v1.3.0 milestone Oct 3, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants