[Bitnami/Airflow] DAG Failure - Read-Only File/ Permission denied System Error in Scheduler Logs (No host supplie )

[siddjellali]

Name and Version

bitnami/airflow:21.5.1

What architecture are you using?

amd64

What steps will reproduce the bug?

1. Run the DAG in the Airflow environment.
2. Check the DAG logs and Kubernetes logs for the task/pod errors.

Are you using any custom parameters or values?

executor: KubernetesExecutor
    rbac:
      create: true
    serviceAccount:
      create: true
    dags:
      enabled: true
      repositories:
        - repository: "https://sdjellali:[email protected]/operators.git"
          branch: "development"
          name: "dane-dags"
          path: "dags"
    scheduler:
      automountServiceAccountToken: true
      resources:
        requests:
          cpu: 500m
          memory: 1Gi
    worker:
      automountServiceAccountToken: true
      terminationGracePeriodSeconds: 360
      resources:
        requests:
          cpu: 500m
          memory: 1Gi
    auth:
      username: airflow
      existingSecret: airflow-secrets
    postgresql:
      existingSecret: airflow-secrets
    redis:
      existingSecret: airflow-secrets

What is the expected behavior?

Dags running

What do you see instead?

DAGs failing

Additional information

The Airflow DAG is failing due to an error related to log access in a read-only file system. The following error message was encountered:

*** Could not read served logs: Invalid URL 'http://:8793/log/dag_id=tutorial/run_id=manual__2024-11-09T11:06:34.771195+00:00/task_id=print_date/attempt=2.log': No host supplied

Additionally, Kubernetes logs for the test-deployer-start-task pod show an issue in loading the configuration:

Unable to load the config, contains a configuration error.
Traceback (most recent call last):
  File "/opt/bitnami/python/lib/python3.12/logging/config.py", line 608, in configure
    handler = self.configure_handler(handlers[name])
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/bitnami/python/lib/python3.12/logging/config.py", line 876, in configure_handler
    result = factory(**kwargs)
             ^^^^^^^^^^^^^^^^^
  File "/opt/bitnami/airflow/venv/lib/python3.12/site-packages/airflow/utils/log/file_processor_handler.py", line 53, in __init__
    Path(self._get_log_directory()).mkdir(parents=True, exist_ok=True)
  File "/opt/bitnami/python/lib/python3.12/pathlib.py", line 1311, in mkdir
    os.mkdir(self, mode)
OSError: [Errno 30] Read-only file system: '/opt/bitnami/airflow/logs/scheduler/2024-11-09'

[siddjellali]

I encounter a permission error preventing Airflow from accessing the log directory, even after setting readOnlyRootFilesystem to false.

kubectl logs test-deployer-start-task-s7hkx0ow -c k8s-executor-init-config
Unable to load the config, contains a configuration error.
Traceback (most recent call last):
  File "/opt/bitnami/python/lib/python3.12/logging/config.py", line 608, in configure
    handler = self.configure_handler(handlers[name])
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/bitnami/python/lib/python3.12/logging/config.py", line 876, in configure_handler
    result = factory(**kwargs)
             ^^^^^^^^^^^^^^^^^
  File "/opt/bitnami/airflow/venv/lib/python3.12/site-packages/airflow/utils/log/file_processor_handler.py", line 53, in __init__
    Path(self._get_log_directory()).mkdir(parents=True, exist_ok=True)
  File "/opt/bitnami/python/lib/python3.12/pathlib.py", line 1311, in mkdir
    os.mkdir(self, mode)
PermissionError: [Errno 13] Permission denied: '/opt/bitnami/airflow/logs/scheduler/2024-11-09'

Need to check if you follow the https://airflow.apache.org/docs/apache-airflow/stable/howto/docker-compose/index.html#setting-the-right-airflow-user instructions.

[siddjellali]

@carrodher if I'm not mistaken, the permissions aren’t quite correct. I would have set the group to 1001 instead of root. What do you think?

https://github.com/bitnami/containers/blob/main/bitnami/airflow/2/debian-12/rootfs/opt/bitnami/scripts/airflow/postunpack.sh

for dir in "$AIRFLOW_TMP_DIR" "$AIRFLOW_LOGS_DIR" "$AIRFLOW_SCHEDULER_LOGS_DIR" "$AIRFLOW_DAGS_DIR" "${AIRFLOW_BASE_DIR}/nss-wrapper"; do
    ensure_dir_exists "$dir"
    configure_permissions_ownership "$dir" -d "775" -f "664" -g "root"
done

[carrodher]

Bitnami containers are designed to operate as non-root by default. Consequently, any files or directories used by the application should be owned by the root group, as the random user (1001 by default) is a member of this root group. To ensure proper permissions, you'll need to adjust the ownership of your local directory accordingly.

For more comprehensive information about non-root containers and their significance for security, you can explore the following resources:

• Why non-root containers are important for security
• Use Non-Root Containers
• Work With Non-Root Containers for Bitnami Applications
• Running non-root containers on Openshift

These references provide valuable insights into the best practices and considerations when working with non-root containers in Bitnami applications.

[siddjellali]

Thanks @carrodher what could be the permission issue of the logs scheduler directory ?

[siddjellali]

real problem was due to misconfiguration on DAG...