Skip to content

Migrate from V3 to V4

Distribute defaults across multiple files

Warning

The --defaults flag is removed

It is possible now to use multiple default values. The defaults.yaml (or defaults_<env>.yaml) files can be distributed across multiple files. These will be picked up by KPOps and get merged into a single pipeline.yaml file. KPOps starts from reading the default files from where the pipeline path is defined and picks up every defaults file on its way to where the pipeline_base_dir is defined.

For example, imagine the following folder structure:

1
2
3
4
5
6
7
└─ pipelines
   └── distributed-defaults
       ├── defaults.yaml
       ├── defaults_dev.yaml
       └── pipeline-deep
           ├── defaults.yaml
           └── pipeline.yaml

The pipeline_base_dir is configured to pipelines. Now if we generate this pipeline with the following command:

1
2
3
kpops generate \
      --environment dev
      ./pipelines/distributed-defaults/pipeline-deep/pipeline.yaml

The defaults would be picked in the following order (high to low priority):

  • ./pipelines/distributed-defaults/pipeline-deep/defaults.yaml
  • ./pipelines/distributed-defaults/defaults_dev.yaml
  • ./pipelines/distributed-defaults/defaults.yaml

The deepest defaults.yaml file in the folder hierarchy (i.e., the closest one to the pipeline.yaml) overwrites the higher-level defaults' values.