KafkaApp
Subclass of KubernetesApp.
Usage
Configuration
pipeline.yaml
| # Base component for Kafka-based components.
# Producer or streaming apps should inherit from this class.
- type: kafka-app # required
name: kafka-app # required
# Pipeline prefix that will prefix every component name. If you wish to not
# have any prefix you can specify an empty string.
prefix: ${pipeline_name}-
from: # Must not be null
topics: # read from topic
${pipeline_name}-input-topic:
type: input # Implied when role is NOT specified
${pipeline_name}-extra-topic:
role: topic-role # Implies `type` to be extra
${pipeline_name}-input-pattern-topic:
type: pattern # Implied to be an input pattern if `role` is undefined
${pipeline_name}-extra-pattern-topic:
type: pattern # Implied to be an extra pattern if `role` is defined
role: some-role
components: # read from specific component
account-producer:
type: output # Implied when role is NOT specified
other-producer:
role: some-role # Implies `type` to be extra
component-as-input-pattern:
type: pattern # Implied to be an input pattern if `role` is undefined
component-as-extra-pattern:
type: pattern # Implied to be an extra pattern if `role` is defined
role: some-role
# Topic(s) into which the component will write output
to:
topics:
${pipeline_name}-output-topic:
type: output # Implied when role is NOT specified
${pipeline_name}-extra-topic:
role: topic-role # Implies `type` to be extra; Will throw an error if `type` is defined
${pipeline_name}-error-topic:
type: error
# Currently KPOps supports Avro and JSON schemas.
key_schema: key-schema # must implement SchemaProvider to use
value_schema: value-schema
partitions_count: 1
replication_factor: 1
configs: # https://kafka.apache.org/documentation/#topicconfigs
cleanup.policy: compact
models: # SchemaProvider is initiated with the values given here
model: model
namespace: namespace # required
# `app` can contain application-specific settings, hence the user is free to
# add the key-value pairs they need.
app: # required
streams: # required
brokers: ${brokers} # required
schemaRegistryUrl: ${schema_registry_url}
nameOverride: override-with-this-name # kafka-app-specific
imageTag: "1.0.0" # Example values that are shared between streams-app and producer-app
# Helm repository configuration (optional)
# If not set the helm repo add will not be called. Useful when using local Helm charts
repo_config:
repository_name: bakdata-streams-bootstrap # required
url: https://bakdata.github.io/streams-bootstrap/ # required
repo_auth_flags:
username: user
password: pass
ca_file: /home/user/path/to/ca-file
insecure_skip_tls_verify: false
version: "2.12.0" # Helm chart version
|
Operations
deploy
In addition to KubernetesApp's deploy
:
- Create topics if provided (optional)
- Submit Avro schemas to the registry if provided (optional)
destroy
Uninstall Helm release.
reset
Do nothing.
clean
Do nothing.