Skip to content

Substitution

KPOps supports the usage of placeholders and environment variables in pipeline definition and defaults.

Component-specific variables

These variables can be used in a component's definition to refer to any of its attributes, including ones that the user has defined in the defaults.

All of them are prefixed with component. and follow the following form: component.{attribute_name}. If the attribute itself contains attributes, they can be referred to like this: component.{attribute_name}.{subattribute_name}.

Example 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
- type: scheduled-producer
  values:
    labels:
      app_type: "${component.type}"
      app_name: "${component.name}"
      app_schedule: "${component.values.schedule}"
    commandLine:
      FAKE_ARG: "fake-arg-value"
    schedule: "30 3/8 * * *"
- type: converter
  values:
    commandLine:
      CONVERT_XML: true
    resources:
      limits:
        memory: 2G
      requests:
        memory: 2G
- type: filter
  name: "filter-app"
  values:
    labels:
      app_type: "${component.type}"
      app_name: "${component.name}"
      app_resources_requests_memory: "${component.values.resources.requests.memory}"
      ${component.type}: "${component.values.labels.app_name}-${component.values.labels.app_type}"
      test_placeholder_in_placeholder: "${component.values.labels.${component.type}}"
    commandLine:
      TYPE: "nothing"
    resources:
      requests:
        memory: 3G
    replicaCount: 4
    autoscaling:
      minReplicas: 4
      maxReplicas: 4

Pipeline-config-specific variables

These variables include all fields in the config and refer to the pipeline configuration that is independent of the components.

All such variables are prefixed with config. and are of the same form as the component-specific variables.

Info

error_topic_name is an alias for config.topic_name_config.default_error_topic_name
output_topic_name is an alias for config.topic_name_config.default_output_topic_name

Environment variables

Environment variables such as $PATH can be used in the pipeline definition and defaults without any transformation following the form ${ENV_VAR_NAME}. This, of course, includes variables like the ones relevant to the KPOps cli that are exported by the user.

See all KPOps environment variables

Pipeline name variables

These are special variables that refer to the name and path of a pipeline.

  • ${pipeline.name}: Concatenated path of the parent directory where pipeline.yaml is defined in. For instance, ./data/pipelines/v1/pipeline.yaml, here the value for the variable would be data-pipelines-v1.

  • ${pipeline_name_<level>}: Similar to the previous variable, each <level> contains a part of the path to the pipeline.yaml file. Consider the previous example, ${pipeline_name_0} would be data, ${pipeline_name_1} would be pipelines, and ${pipeline_name_2} equals to v1.

Advanced use cases

  1. Refer to default component field values: As long as a value is assigned to a component attribute, it is possible to refer to it with a placeholder. To see all component fields, take a look at the pipeline schema.
  2. Chaining variables: It is possible to chain any number of variables, see the example above.
  3. Cross-component substitution: YAML is quite an intricate language and with some of its magic one could write cross-component references.