Libraries requirements not correctly replaced by Workspace path

[ovidiu-eremia]

Describe the issue

We use DABs to deploy workflow jobs that run python_wheel_tasks from the package loaded using the tasks libraries as a whl artifact with a requirements txt file. Both the whl and the requirements txt files are loaded providing the local relative path to them from the workflow yml where they are referenced, to the repository local folder where they are located.

Configuration

# yaml-language-server: $schema=../../bundle_config_schema.json

resources:
  jobs:
    ingest:
      name: ingest

      tasks:
        - task_key: ingest
          job_cluster_key: default
          libraries:
            - requirements: ../../deps/requirements.txt
            - whl: ../../dist/*.whl
          python_wheel_task:
            package_name: my_package
            entry_point: my_package.entry_point
            parameters:
              - "--execution-time"
              - "{{ job.trigger.time.iso_datetime }}"
              - "--environment"
              - "${bundle.target}"

      job_clusters:
        - job_cluster_key: default
          new_cluster:
            spark_version: 15.4.x-scala2.12
            node_type_id: Standard_DS3_v2
            num_workers: 0
            spark_conf:
              spark.master: local[*, 4]
              spark.databricks.cluster.profile: singleNode
            custom_tags:
              ResourceClass: SingleNode
            azure_attributes:
              first_on_demand: 1
              availability: SPOT_WITH_FALLBACK_AZURE
              spot_bid_max_price: -1

Steps to reproduce the behavior

Please list the steps required to reproduce the issue, for example:

1. Run databricks bundle deploy ...
2. Run databricks bundle run ...
3. See error from the cluster driver:

run failed with error message [LibraryInstallationError] 
 Library installation failed for library due to user error. Error messages:
Requirements file must be stored in UC Volumes, dbfs, s3, adls, gs or as a workspace file/local file. Make sure the URI begins with 'dbfs:', 'file:', 's3:', 'abfss:', 'gs:', 'wasbs:', '/Volumes', or '/Workspace'but the URI is '/Shared/.bundle/target/ingest/files/deps/requirements.txt').

Expected Behavior

The dependent libraries requirements file path should be similar to the one of the deployed library whl artifact:

/Workspace/Shared/.bundle/target/ingest/artifacts/.internal/ingest-0.0.1-py3-none-any.whl

like this:

/Workspace/Shared/.bundle/target/ingest/files/deps/requirements.txt

Actual Behavior

The path to the requirements file is in fact:

/Shared/.bundle/target/ingest/files/deps/requirements.txt

OS and CLI version

Databricks CLI: v0.227.1
OS: Debian - Ubuntu 22.04

Is this a regression?

I think this was supposed to be fixed in this PR:
#1543

[andrewnester]

Yes, indeed that's the case at the moment. This is going to be fix with this PR #1724

[frankivo]

Workaround available here: #1517 (comment)