Skip to content

Configuration

KPOps reads its global configuration that is unrelated to a pipeline's components from config.yaml.

Consider enabling KPOps' editor integration feature to enjoy the benefits of autocompletion and validation when configuring your pipeline.

To learn about any of the available settings, take a look at the example below.

config.yaml 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
# CONFIGURATION
#
# The path to the folder containing the defaults.yaml file and the environment
# defaults files.
defaults_path: .
# The environment you want to generate and deploy the pipeline to. Suffix your
# environment files with this value (e.g. defaults_development.yaml and
# pipeline_development.yaml for environment=development).
# REQUIRED
environment: development
# The Kafka brokers address.
# REQUIRED
brokers: "http://broker1:9092,http://broker2:9092"
# The name of the defaults file and the prefix of the defaults environment file.
defaults_filename_prefix: defaults
# Configures topic names.
topic_name_config: 
  # Configures the value for the variable ${output_topic_name}
  default_output_topic_name: ${pipeline_name}-${component_name}
  # Configures the value for the variable ${error_topic_name}
  default_error_topic_name: ${pipeline_name}-${component_name}-error
# Address of the Schema Registry
schema_registry_url: "http://localhost:8081"
# Address of the Kafka REST Proxy.
kafka_rest_host: "http://localhost:8082"
# Address of Kafka Connect.
kafka_connect_host: "http://localhost:8083"
# The timeout in seconds that specifies when actions like deletion or deploy
# timeout.
timeout: 300
# Flag for `helm upgrade --install`.
# Create the release namespace if not present.
create_namespace: false
# Global flags for Helm.
helm_config:
  # Set the name of the kubeconfig context. (--kube-context)
  context: name
  # Run Helm in Debug mode.
  debug: false
# Configure Helm Diff.
helm_diff_config: 
  # Set of keys that should not be checked.
  ignore: 
    - name
    - imageTag
# Whether to retain clean up jobs in the cluster or uninstall the, after
# completion.
retain_clean_jobs: false

Environment-specific pipeline definitions

Similarly to defaults, it is possible to have an unlimited amount of additional environment-specific pipeline definitions. The naming convention is the same: add a suffix of the form _{environment} to the filename.