Skip to content

Environment variables

Environment variables can be set by using the export command in Linux or the set command in Windows.

dotenv files

KPOps currently supports .env files only for variables related to the config. Full support for .env files is on the roadmap. One of the possible ways to use one and export the contents manually is with the following command: export $(xargs < .env). This would work in bash suppose there are no spaces inside the values.

Config

These variables are a lower priority alternative to the settings in config.yaml. Variables marked as required can instead be set in the global config.

Name Default Value Required Description Setting name
KPOPS_DEFAULTS_PATH . False The path to the folder containing the defaults.yaml file and the environment defaults files. Paths can either be absolute or relative to config.yaml defaults_path
KPOPS_COMPONENTS_MODULE False Custom Python module defining project-specific KPOps components components_module
KPOPS_PIPELINE_BASE_DIR . False Base directory to the pipelines (default is current working directory) pipeline_base_dir
KPOPS_KAFKA_BROKERS True The comma separated Kafka brokers address. kafka_brokers
KPOPS_DEFAULTS_FILENAME_PREFIX defaults False The name of the defaults file and the prefix of the defaults environment file. defaults_filename_prefix
KPOPS_TOPIC_NAME_CONFIG__DEFAULT_OUTPUT_TOPIC_NAME ${pipeline.name}-${component.name} False Configures the value for the variable ${output_topic_name} topic_name_config.default_output_topic_name
KPOPS_TOPIC_NAME_CONFIG__DEFAULT_ERROR_TOPIC_NAME ${pipeline.name}-${component.name}-error False Configures the value for the variable ${error_topic_name} topic_name_config.default_error_topic_name
KPOPS_SCHEMA_REGISTRY__ENABLED False False Whether the Schema Registry handler should be initialized. schema_registry.enabled
KPOPS_SCHEMA_REGISTRY__URL http://localhost:8081/ False Address of the Schema Registry. schema_registry.url
KPOPS_KAFKA_REST__URL http://localhost:8082/ False Address of the Kafka REST Proxy. kafka_rest.url
KPOPS_KAFKA_CONNECT__URL http://localhost:8083/ False Address of Kafka Connect. kafka_connect.url
KPOPS_TIMEOUT 300 False The timeout in seconds that specifies when actions like deletion or deploy timeout. timeout
KPOPS_CREATE_NAMESPACE False False Flag for helm upgrade --install. Create the release namespace if not present. create_namespace
KPOPS_HELM_CONFIG__CONTEXT False Name of kubeconfig context (--kube-context) helm_config.context
KPOPS_HELM_CONFIG__DEBUG False False Run Helm in Debug mode helm_config.debug
KPOPS_HELM_CONFIG__API_VERSION False Kubernetes API version used for Capabilities.APIVersions helm_config.api_version
KPOPS_HELM_DIFF_CONFIG__IGNORE True Set of keys that should not be checked. helm_diff_config.ignore
KPOPS_RETAIN_CLEAN_JOBS False False Whether to retain clean up jobs in the cluster or uninstall the, after completion. retain_clean_jobs
config_env_vars.env
Exhaustive list of all config-related environment variables
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
# Global config environment variables
#
# The default setup is shown. These variables are a lower priority
# alternative to the settings in `config.yaml`. Variables marked as
# required can instead be set in the global config.
#
# defaults_path
# The path to the folder containing the defaults.yaml file and the
# environment defaults files. Paths can either be absolute or relative
# to `config.yaml`
KPOPS_DEFAULTS_PATH=.
# components_module
# Custom Python module defining project-specific KPOps components
KPOPS_COMPONENTS_MODULE # No default value, not required
# pipeline_base_dir
# Base directory to the pipelines (default is current working
# directory)
KPOPS_PIPELINE_BASE_DIR=.
# kafka_brokers
# The comma separated Kafka brokers address.
KPOPS_KAFKA_BROKERS # No default value, required
# defaults_filename_prefix
# The name of the defaults file and the prefix of the defaults
# environment file.
KPOPS_DEFAULTS_FILENAME_PREFIX=defaults
# topic_name_config.default_output_topic_name
# Configures the value for the variable ${output_topic_name}
KPOPS_TOPIC_NAME_CONFIG__DEFAULT_OUTPUT_TOPIC_NAME=${pipeline.name}-${component.name}
# topic_name_config.default_error_topic_name
# Configures the value for the variable ${error_topic_name}
KPOPS_TOPIC_NAME_CONFIG__DEFAULT_ERROR_TOPIC_NAME=${pipeline.name}-${component.name}-error
# schema_registry.enabled
# Whether the Schema Registry handler should be initialized.
KPOPS_SCHEMA_REGISTRY__ENABLED=False
# schema_registry.url
# Address of the Schema Registry.
KPOPS_SCHEMA_REGISTRY__URL=http://localhost:8081/
# kafka_rest.url
# Address of the Kafka REST Proxy.
KPOPS_KAFKA_REST__URL=http://localhost:8082/
# kafka_connect.url
# Address of Kafka Connect.
KPOPS_KAFKA_CONNECT__URL=http://localhost:8083/
# timeout
# The timeout in seconds that specifies when actions like deletion or
# deploy timeout.
KPOPS_TIMEOUT=300
# create_namespace
# Flag for `helm upgrade --install`. Create the release namespace if
# not present.
KPOPS_CREATE_NAMESPACE=False
# helm_config.context
# Name of kubeconfig context (`--kube-context`)
KPOPS_HELM_CONFIG__CONTEXT # No default value, not required
# helm_config.debug
# Run Helm in Debug mode
KPOPS_HELM_CONFIG__DEBUG=False
# helm_config.api_version
# Kubernetes API version used for `Capabilities.APIVersions`
KPOPS_HELM_CONFIG__API_VERSION # No default value, not required
# helm_diff_config.ignore
# Set of keys that should not be checked.
KPOPS_HELM_DIFF_CONFIG__IGNORE # No default value, required
# retain_clean_jobs
# Whether to retain clean up jobs in the cluster or uninstall the,
# after completion.
KPOPS_RETAIN_CLEAN_JOBS=False

CLI

These variables are a lower priority alternative to the commands' flags. If a variable is set, the corresponding flag does not have to be specified in commands. Variables marked as required can instead be set as flags.

Name Default Value Required Description
KPOPS_CONFIG_PATH . False Path to the dir containing config.yaml files
KPOPS_DEFAULT_PATH False Path to defaults folder
KPOPS_DOTENV_PATH False Path to dotenv file. Multiple files can be provided. The files will be loaded in order, with each file overriding the previous one.
KPOPS_ENVIRONMENT False The environment you want to generate and deploy the pipeline to. Suffix your environment files with this value (e.g. defaults_development.yaml for environment=development).
KPOPS_PIPELINE_PATH True Path to YAML with pipeline definition
KPOPS_PIPELINE_STEPS False Comma separated list of steps to apply the command on
cli_env_vars.env
Exhaustive list of all cli-related environment variables
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
# CLI Environment variables
#
# The default setup is shown. These variables are a lower priority
# alternative to the commands' flags. If a variable is set, the
# corresponding flag does not have to be specified in commands.
# Variables marked as required can instead be set as flags.
#
# Path to the dir containing config.yaml files
KPOPS_CONFIG_PATH=.
# Path to defaults folder
KPOPS_DEFAULT_PATH # No default value, not required
# Path to dotenv file. Multiple files can be provided. The files will
# be loaded in order, with each file overriding the previous one.
KPOPS_DOTENV_PATH # No default value, not required
# The environment you want to generate and deploy the pipeline to.
# Suffix your environment files with this value (e.g.
# defaults_development.yaml for environment=development).
KPOPS_ENVIRONMENT # No default value, not required
# Path to YAML with pipeline definition
KPOPS_PIPELINE_PATH # No default value, required
# Comma separated list of steps to apply the command on
KPOPS_PIPELINE_STEPS # No default value, not required