[BUG] Missing common task parameters when task type is Notebook

[sidharth-shridhar]

Describe the bug
Notebook task type doesn't get common task parameters injected even when they are defined under workflow config.

Package Info

• brickflow version: 0.11.2
• Databricks CLI v0.213.0

To Reproduce
Steps to reproduce the behavior:

1. Create a workflow as per the following configuration:

from datetime import timedelta

from brickflow import (
    ctx,
    Cluster,
    BrickflowTriggerRule,
    TaskSettings,
    EmailNotifications,
    Workflow,
    WorkflowPermissions,
    User,
    NotebookTask,
)

from brickflow.engine.task import PypiTaskLibrary

wf = Workflow(
    "brickflow-demo",
    # replace <all-purpose-cluster-id> with your cluster id
    default_cluster=Cluster.from_existing_cluster("<all-purpose-cluster-id>"),
    tags={
        "product_id": "brickflow_demo",
        "slack_channel": "YOUR_SLACK_CHANNEL",
    },
    # COMMON TASK PARAMS
    common_task_parameters={
        "catalog": "some_val",
        "database": "some_db",
    },
    # replace <emails> with existing users' email on databricks
    permissions=WorkflowPermissions(
        can_manage_run=[User("[email protected]"), User("[email protected]")],
        can_view=[User("[email protected]")],
        can_manage=[User("[email protected]")],
    ),
    libraries=[PypiTaskLibrary(package="snowflake==0.5.1")],
)

@wf.notebook_task
def example_notebook():
    return NotebookTask(
        notebook_path="notebooks/example_notebook.py"
    )

3. Configure the necessary installation of the packages like brickflow (=0.11.2), databricks-cli
4. Initialize the project setup as guided in the quick start guide
5. Try to deploy the workflow to the desired workspace using:
   brickflow projects deploy --project brickflow-demo -e test
6. From Databricks UI jobs config page, look into the task parameters of the task, common task parameters will be absent or missing.

Expected behavior
If common params are defined under the workflow object, the notebooks tasks should also be injected with the common parameters along with base parameters if supplied.

Screenshots

Missing params: catalog and database

Cloud Information

• AWS
• Azure
• GCP
• Other

Desktop (please complete the following information):

• OS: macOS 14.3 Sonoma
• Browser Chrome

Additional context
Add any other context about the problem here.