Configuration

The configurations for a pipeline can be defines using a config.yaml file. Before every run, the pipeline would read the configurations and accordingly ingest, process and export data from the pipeline. The following is an example of a config.yaml file.

Configuration

# 'project' defines the metadata of the project
project:
    project_name: mock_project
    description: mock_description
    version: mock_version

# 'input_data' defines how the data would be ingested by the pipeline
input_data:
    data_location: mock # location of the data (local path or 'mock')
    data_format: csv # format of the data ('csv')

# 'features' define what features to use for data processing
features:
    - 'mock'

# 'input_data' defines how the data would be exported by the pipeline
output_data:
    output_location: mock