Skip to content

Migrate from V2 to V3

Jump to the summary

Use hash and trim long Helm release names instead of only trimming

KPOps handles long (more than 53 characters) Helm releases names differently. Helm will not find your (long) old release names anymore. Therefore, it is recommended that you should once destroy your pipeline with KPOps v2 to remove old Helm release names. After a clean destroy, re-deploy your pipeline with the KPOps v3.

For example if you have a component with the Helm release name example-component-name-too-long-fake-fakefakefakefakefake. The new release name will shorten the original name to 53 characters and then replace the last 6 characters of the trimmed name with the first 5 characters of the result of SHA-1(helm_release_name).

1
2
3
example-component-name-too-long-fake-fakefakef-0a7fc ----> 53 chars
---------------------------------------------- -----
  ^Shortened helm_release_name                 ^first 5 characters of SHA1(helm_release_name)

Create HelmApp component

All Helm-specific parts of the built-in KubernetesApp have been extracted to a new child component that is more appropriately named HelmApp. It has to be renamed in your existing pipeline defintions and custom components module.

pipeline.yaml

1
2
3
-- type: kubernetes-app
+- type: helm-app
   name: foo

custom_module.py

1
2
3
4
5
6
7
- from kpops.components import KubernetesApp
+ from kpops.components import HelmApp


- class CustomHelmApp(KubernetesApp):
+ class CustomHelmApp(HelmApp):
      ...

Create StreamsBootstrap component & refactor cleanup jobs as individual HelmApp

Previously the default KafkaApp component configured the streams-bootstrap Helm Charts. Now, this component is no longer tied to Helm (or Kubernetes). Instead, there is a new StreamsBootstrap component that configures the Helm Chart repository for the components that use it, e.g. StreamsApp and ProducerApp. If you are using non-default values for the Helm Chart repository or version, it has to be updated as shown below.

defaults.yaml

1
2
3
4
5
6
7
  kafka-app:
    app:
      streams: ...

+ streams-bootstrap:
    repo_config: ...
    version: ...

Refactor Kafka Connector resetter as individual HelmApp

Internally, the Kafka Connector resetter is now its own standard HelmApp, removing a lot of the shared code. It is configured using the resetter_namespace (formerly namespace) and resetter_values attributes.

defaults.yaml

1
2
3
  kafka-connector:
-   namespace: my-namespace
+   resetter_namespace: my-namespace

Make Kafka REST Proxy & Kafka Connect hosts default and improve Schema Registry config

The breaking changes target the config.yaml file:

  • The schema_registry_url is replaced with schema_registry.url (default http://localhost:8081) and schema_registry.enabled (default false).

  • kafka_rest_host is renamed to kafka_rest.url (default http://localhost:8082).

  • kafka_connect_host is replaced with kafka_connect.url (default http://localhost:8083).

  • brokers is renamed to kafka_brokers.

The environment variable names of these config fields changed respectively. Please refer to the environment variables documentation page to see the newest changes.

config.yaml

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
  environment: development
- brokers: "http://k8kafka-cp-kafka-headless.kpops.svc.cluster.local:9092"
- kafka_rest_host: "http://my-custom-rest.url:8082"
- kafka_connect_host: "http://my-custom-connect.url:8083"
- schema_registry_url: "http://my-custom-sr.url:8081"
+ kafka_brokers: "http://k8kafka-cp-kafka-headless.kpops.svc.cluster.local:9092"
+ kafka_rest:
+   url: "http://my-custom-rest.url:8082"
+ kafka_connect:
+   url: "http://my-custom-connect.url:8083"
+ schema_registry:
+   enabled: true
+   url: "http://my-custom-sr.url:8081"

pipeline.yaml and default.yaml

The variable is now called kafka_brokers.

1
2
3
4
5
6
7
8
9
...
  app:
    streams:
-     brokers: ${brokers}
+     brokers: ${kafka_brokers}
      schemaRegistryUrl: ${schema_registry_url}
    nameOverride: override-with-this-name
    imageTag: "1.0.0"
...

Define custom components module & pipeline base dir globally

Warning

The previous CLI parameters have been removed.

The options for a custom components_module and pipeline_base_dir are now global settings, defined in config.yaml.

config.yaml

1
2
3
4
  kafka_brokers: "http://k8kafka-cp-kafka-headless.kpops.svc.cluster.local:9092"
  environment: development
+ components_module: components
+ pipeline_base_dir: pipelines

Move GitHub action to repsitory root

The location of the GitHub action has changed, and it's now available directly as bakdata/kpops.

You'll need to change it in your GitHub CI workflows.

1
2
3
4
5
6
7
steps:
  - name: kpops deploy
-   uses: bakdata/kpops/actions/kpops-runner@main
+   uses: bakdata/kpops@main
    with:
      command: deploy --execute
      # ...

Allow overriding config files

Specifying the environment is no longer mandatory. If not defined, only the global files will be used.

environment is no longer specified in config.yaml. Instead, it can be either set via the CLI flag --environment or with the environment variable KPOPS_ENVIRONMENT.

The --config flag in the CLI now points to the directory that contains config*.yaml files. The files to be used are resolved based on the provided (or not) environment.

config.yaml

1
2
3
4
5
- environment: development
  kafka_brokers: "http://k8kafka-cp-kafka-headless.kpops.svc.cluster.local:9092"
  schema_registry:
    enabled: true
    url: "http://my-custom-sr.url:8081"

Change substitution variables separator to .

The delimiter in the substitution variables is changed to ..

pipeline.yaml and default.yaml

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
steps:
  - type: scheduled-producer
    app:
      labels:
-       app_type: "${component_type}"
-       app_name: "${component_name}"
-       app_schedule: "${component_app_schedule}"
+       app_type: "${component.type}"
+       app_name: "${component.name}"
+       app_schedule: "${component.app.schedule}"

config.yaml

1
2
3
4
5
topic_name_config:
- default_error_topic_name: "${pipeline_name}-${component_name}-dead-letter-topic"
- default_output_topic_name: "${pipeline_name}-${component_name}-topic"
+ default_error_topic_name: "${pipeline_name}-${component.name}-dead-letter-topic"
+ default_output_topic_name: "${pipeline_name}-${component.name}-topic"

Refactor generate template for Python API usage

The template method of every pipeline component has been renamed to manifest as it is no longer strictly tied to Helm template. Instead, it can be used to render the final resources of a component, such as Kubernetes manifests.

There is also a new kpops manifest command replacing the existing kpops generate --template flag.

If you're using this functionality in your custom components, it needs to be updated.

1
2
3
4
5
6
7
  from kpops.components.base_components.models.resource import Resource

  @override
- def template(self) -> None:
+ def manifest(self) -> Resource:
  """Render final component resources, e.g. Kubernetes manifests."""
      return []  # list of manifests

Namespace substitution vars

The global configuration variables are now namespaced under the config key, such as ${config.kafka_brokers}, ${config.schema_registry.url}. Same with pipeline variables, e.g. ${pipeline_name} → ${pipeline.name}. This would make it more uniform with the existing ${component.<key>} variables.

pipeline.yaml

1
2
3
4
5
6
7
8
9
  name: kafka-app
- prefix: ${pipeline_name}-
+ prefix: ${pipeline.name}-
  app:
    streams:
-     brokers: ${kafka_brokers}
-     schemaRegistryUrl: ${schema_registry.url}
+     brokers: ${config.kafka_brokers}
+     schemaRegistryUrl: ${config.schema_registry.url}

Summary

Warning

Helm will not find your (long) old release names anymore.

defaults.yaml 
1
2
3
4
5
6
7
  kafka-app:
    app:
      streams: ...

+ streams-bootstrap:
    repo_config: ...
    version: ...
pipeline.yaml 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
- - type: kubernetes-app
+ - type: helm-app
  ...
  - type: kafka-app
    app:
-     brokers: ${brokers}
+     brokers: ${config.kafka_brokers}
      labels:
-       app_schedule: "${component_app_schedule}"
+       app_schedule: "${component.app.schedule}"
  ...
  - type: kafka-connector
-   namespace: my-namespace
+   resetter_namespace: my-namespace
  ...
config.yaml 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
- environment: development

+ components_module: components

+ pipeline_base_dir: pipelines

- brokers: "http://k8kafka-cp-kafka-headless.kpops.svc.cluster.local:9092"
+ kafka_brokers: "http://k8kafka-cp-kafka-headless.kpops.svc.cluster.local:9092"

- kafka_rest_host: "http://my-custom-rest.url:8082"
+ kafka_rest:
+   url: "http://my-custom-rest.url:8082"

- kafka_connect_host: "http://my-custom-connect.url:8083"
+ kafka_connect:
+   url: "http://my-custom-connect.url:8083"

- schema_registry_url: "http://my-custom-sr.url:8081"
+ schema_registry:
+   enabled: true
+   url: "http://my-custom-sr.url:8081"

  topic_name_config:
- default_error_topic_name: "${pipeline_name}-${component_name}-dead-letter-topic"
+ default_error_topic_name: "${pipeline.name}-${component.name}-dead-letter-topic"
  ...
custom_module.py 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
- from kpops.components import KubernetesApp
+ from kpops.components import HelmApp
+ from kpops.components.base_components.models.resource import Resource

- class CustomHelmApp(KubernetesApp):
+ class CustomHelmApp(HelmApp):

  @override
- def template(self) -> None:
+ def manifest(self) -> Resource:
  """Render final component resources, e.g. Kubernetes manifests."""
      return []  # list of manifests
  ...
github_ci_workflow.yaml 
1
2
3
4
5
  steps:
    - name: ...
-     uses: bakdata/kpops/actions/kpops-runner@main
+     uses: bakdata/kpops@main
  ...