Skip to content

ProducerApp

Subclass of KafkaApp and StreamsBootstrap.

Usage

Configures a streams-bootstrap Kafka producer app

Configuration

pipeline.yaml 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
# Holds configuration to use as values for the streams bootstrap producer-app Helm
# chart.
# More documentation on ProducerApp:
# https://github.com/bakdata/streams-bootstrap
- type: producer-app
  name: producer-app # required
  # Pipeline prefix that will prefix every component name. If you wish to not
  # have any prefix you can specify an empty string.
  prefix: ${pipeline.name}-
  # from: # While the producer-app does inherit from kafka-app, it does not need a
  # `from` section, hence it does not support it.
  # Topic(s) into which the component will write output
  to:
    topics:
      ${pipeline.name}-output-topic:
        type: output # Implied when role is NOT specified
      ${pipeline.name}-extra-topic:
        role: topic-role # Implies `type` to be extra; Will throw an error if `type` is defined
      ${pipeline.name}-error-topic:
        type: error
        # Currently KPOps supports Avro and JSON schemas.
        key_schema: key-schema # must implement SchemaProvider to use
        value_schema: value-schema
        partitions_count: 1
        replication_factor: 1
        configs: # https://kafka.apache.org/documentation/#topicconfigs
          cleanup.policy: compact
    models: # SchemaProvider is initiated with the values given here
      model: model
  namespace: namespace # required
  # Allowed configs:
  # https://github.com/bakdata/streams-bootstrap/tree/master/charts/producer-app
  app: # required
    streams: # required, producer-app-specific
      brokers: ${config.kafka_brokers} # required
      schemaRegistryUrl: ${config.schema_registry.url}
      outputTopic: output_topic
      extraOutputTopics:
        output_role1: output_topic1
        output_role2: output_topic2
    nameOverride: override-with-this-name # kafka-app-specific
  # Helm repository configuration (optional)
  # If not set the helm repo add will not be called. Useful when using local Helm charts
  repo_config:
    repository_name: bakdata-streams-bootstrap # required
    url: https://bakdata.github.io/streams-bootstrap/ # required
    repo_auth_flags:
      username: user
      password: pass
      ca_file: /home/user/path/to/ca-file
      insecure_skip_tls_verify: false
  version: "2.12.0" # Helm chart version

Operations

deploy

In addition to KubernetesApp's deploy:

  • Create topics if provided (optional)
  • Submit Avro schemas to the registry if provided (optional)

destroy

Uninstall Helm release.

reset

Do nothing, producers are stateless.

clean

  • Delete the output topics of the Kafka producer
  • Delete all associated schemas in the Schema Registry