Skip to content

Configuration

KPOps reads its global configuration that is unrelated to a pipeline's components from config.yaml.

Consider enabling KPOps' editor integration feature to enjoy the benefits of autocompletion and validation when configuring your pipeline.

To learn about any of the available settings, take a look at the example below.

config.yaml 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
# CONFIGURATION
#
# The path to the folder containing the defaults.yaml file and the environment
# defaults files.
defaults_path: .
# Custom Python module defining project-specific KPOps components
components_module: null
# Base directory to the pipelines (default is current working directory)
pipeline_base_dir: .
# The Kafka brokers address.
# REQUIRED
kafka_brokers: "http://broker1:9092,http://broker2:9092"
# The name of the defaults file and the prefix of the defaults environment file.
defaults_filename_prefix: defaults
# Configure the topic name variables you can use in the pipeline definition.
topic_name_config: 
  # Configures the value for the variable ${output_topic_name}
  default_output_topic_name: ${pipeline.name}-${component.name}
  # Configures the value for the variable ${error_topic_name}
  default_error_topic_name: ${pipeline.name}-${component.name}-error
# Configuration for Schema Registry.
schema_registry:
  # Whether the Schema Registry handler should be initialized.
  enabled: false
  # Address of the Schema Registry.
  url: "http://localhost:8081"
# Configuration for the Kafka REST Proxy.
kafka_rest:
  # Address of the Kafka REST Proxy.
  url: "http://localhost:8082"
# Configuration for Kafka Connect.
kafka_connect:
  # Address of Kafka Connect.
  url: "http://localhost:8083"
# The timeout in seconds that specifies when actions like deletion or deploy
# timeout.
timeout: 300
# Flag for `helm upgrade --install`.
# Create the release namespace if not present.
create_namespace: false
# Global flags for Helm.
helm_config:
  # Name of kubeconfig context (`--kube-context`)
  context: name
  # Run Helm in Debug mode.
  debug: false
  # Kubernetes API version used for Capabilities.APIVersions
  api_version: null
# Configure Helm Diff.
helm_diff_config: 
  # Set of keys that should not be checked.
  ignore:
    - name
    - imageTag
# Whether to retain clean up jobs in the cluster or uninstall the, after
# completion.
retain_clean_jobs: false

Environment-specific pipeline definitions

Similarly to defaults, it is possible to have an unlimited amount of additional environment-specific pipeline definitions. The naming convention is the same: add a suffix of the form _{environment} to the filename.