You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Is your feature request related to a problem? Please describe.
This is a feature provided by Databricks to use the For each task to run a task in a loop, passing a different set of parameters to each iteration of the task.
Cloud Information
AWS
Azure
GCP
Other
Describe the solution you'd like
-Adding the For each task to a job requires defining two tasks: The For each task and a nested task. The nested task is the task to run for each iteration of the For each task and is one of the standard Databricks Jobs task types. You cannot add another For each task as the nested task.
-For example, you could use the For each task to perform a common set of transformations on multiple tables, passing a table name from a list of table names to each iteration of the task.
-Instead of creating 100 tasks for 100 table ingestion you can create one task and iterate over the list. It also helps to put a concurrency limit for your tasks which is not available in other type of jobs.
-It will also allow to run many processes using single task.
Describe alternatives you've considered
Currently we are creating 100 tasks for 100 tables ingestion.
Additional context
Add any other context or screenshots about the feature request here.
The text was updated successfully, but these errors were encountered:
Is your feature request related to a problem? Please describe.
This is a feature provided by Databricks to use the For each task to run a task in a loop, passing a different set of parameters to each iteration of the task.
Cloud Information
Describe the solution you'd like
-Adding the For each task to a job requires defining two tasks: The For each task and a nested task. The nested task is the task to run for each iteration of the For each task and is one of the standard Databricks Jobs task types. You cannot add another For each task as the nested task.
-For example, you could use the For each task to perform a common set of transformations on multiple tables, passing a table name from a list of table names to each iteration of the task.
-Instead of creating 100 tasks for 100 table ingestion you can create one task and iterate over the list. It also helps to put a concurrency limit for your tasks which is not available in other type of jobs.
-It will also allow to run many processes using single task.
Describe alternatives you've considered
Currently we are creating 100 tasks for 100 tables ingestion.
Additional context
Add any other context or screenshots about the feature request here.
The text was updated successfully, but these errors were encountered: