Skip to content

Environment variables

Environment variables can be set by using the export command in Linux or the set command in Windows.

dotenv files

KPOps currently supports .env files only for variables related to the config. Full support for .env files is on the roadmap. One of the possible ways to use one and export the contents manually is with the following command: export $(xargs < .env). This would work in bash suppose there are no spaces inside the values.

Config

These variables take precedence over the settings in config.yaml. Variables marked as required can instead be set in the global config.

Name Default Value Required Description Setting name
KPOPS_COMPONENTS_MODULE False Custom Python module defining project-specific KPOps components components_module
KPOPS_PIPELINE_BASE_DIR . False Base directory to the pipelines (default is current working directory) pipeline_base_dir
KPOPS_KAFKA_BROKERS True The comma separated Kafka brokers address. kafka_brokers
KPOPS_TOPIC_NAME_CONFIG__DEFAULT_OUTPUT_TOPIC_NAME ${pipeline.name}-${component.name} False Configures the value for the variable ${output_topic_name} topic_name_config.default_output_topic_name
KPOPS_TOPIC_NAME_CONFIG__DEFAULT_ERROR_TOPIC_NAME ${pipeline.name}-${component.name}-error False Configures the value for the variable ${error_topic_name} topic_name_config.default_error_topic_name
KPOPS_SCHEMA_REGISTRY__ENABLED False False Whether the Schema Registry handler should be initialized. schema_registry.enabled
KPOPS_SCHEMA_REGISTRY__URL http://localhost:8081/ False Address of the Schema Registry. schema_registry.url
KPOPS_SCHEMA_REGISTRY__TIMEOUT 30 False Operation timeout in seconds. schema_registry.timeout
KPOPS_KAFKA_REST__URL http://localhost:8082/ False Address of the Kafka REST Proxy. kafka_rest.url
KPOPS_KAFKA_REST__TIMEOUT 30 False Operation timeout in seconds. kafka_rest.timeout
KPOPS_KAFKA_CONNECT__URL http://localhost:8083/ False Address of Kafka Connect. kafka_connect.url
KPOPS_KAFKA_CONNECT__TIMEOUT 30 False Operation timeout in seconds. kafka_connect.timeout
KPOPS_CREATE_NAMESPACE False False Flag for helm upgrade --install. Create the release namespace if not present. create_namespace
KPOPS_HELM_CONFIG__CONTEXT False Name of kubeconfig context (--kube-context) helm_config.context
KPOPS_HELM_CONFIG__DEBUG False False Run Helm in Debug mode helm_config.debug
KPOPS_HELM_CONFIG__API_VERSION False Kubernetes API version used for Capabilities.APIVersions helm_config.api_version
KPOPS_HELM_DIFF_CONFIG__IGNORE True Set of keys that should not be checked. helm_diff_config.ignore
KPOPS_RETAIN_CLEAN_JOBS False False Whether to retain clean up jobs in the cluster or uninstall the, after completion. retain_clean_jobs
config_env_vars.env
Exhaustive list of all config-related environment variables
# Global config environment variables
#
# The default setup is shown. These variables take precedence over the
# settings in `config.yaml`. Variables marked as required can instead
# be set in the global config.
#
# components_module
# Custom Python module defining project-specific KPOps components
KPOPS_COMPONENTS_MODULE # No default value, not required
# pipeline_base_dir
# Base directory to the pipelines (default is current working
# directory)
KPOPS_PIPELINE_BASE_DIR=.
# kafka_brokers
# The comma separated Kafka brokers address.
KPOPS_KAFKA_BROKERS # No default value, required
# topic_name_config.default_output_topic_name
# Configures the value for the variable ${output_topic_name}
KPOPS_TOPIC_NAME_CONFIG__DEFAULT_OUTPUT_TOPIC_NAME=${pipeline.name}-${component.name}
# topic_name_config.default_error_topic_name
# Configures the value for the variable ${error_topic_name}
KPOPS_TOPIC_NAME_CONFIG__DEFAULT_ERROR_TOPIC_NAME=${pipeline.name}-${component.name}-error
# schema_registry.enabled
# Whether the Schema Registry handler should be initialized.
KPOPS_SCHEMA_REGISTRY__ENABLED=False
# schema_registry.url
# Address of the Schema Registry.
KPOPS_SCHEMA_REGISTRY__URL=http://localhost:8081/
# schema_registry.timeout
# Operation timeout in seconds.
KPOPS_SCHEMA_REGISTRY__TIMEOUT=30
# kafka_rest.url
# Address of the Kafka REST Proxy.
KPOPS_KAFKA_REST__URL=http://localhost:8082/
# kafka_rest.timeout
# Operation timeout in seconds.
KPOPS_KAFKA_REST__TIMEOUT=30
# kafka_connect.url
# Address of Kafka Connect.
KPOPS_KAFKA_CONNECT__URL=http://localhost:8083/
# kafka_connect.timeout
# Operation timeout in seconds.
KPOPS_KAFKA_CONNECT__TIMEOUT=30
# create_namespace
# Flag for `helm upgrade --install`. Create the release namespace if
# not present.
KPOPS_CREATE_NAMESPACE=False
# helm_config.context
# Name of kubeconfig context (`--kube-context`)
KPOPS_HELM_CONFIG__CONTEXT # No default value, not required
# helm_config.debug
# Run Helm in Debug mode
KPOPS_HELM_CONFIG__DEBUG=False
# helm_config.api_version
# Kubernetes API version used for `Capabilities.APIVersions`
KPOPS_HELM_CONFIG__API_VERSION # No default value, not required
# helm_diff_config.ignore
# Set of keys that should not be checked.
KPOPS_HELM_DIFF_CONFIG__IGNORE # No default value, required
# retain_clean_jobs
# Whether to retain clean up jobs in the cluster or uninstall the,
# after completion.
KPOPS_RETAIN_CLEAN_JOBS=False

CLI

These variables take precedence over the commands' flags. If a variable is set, the corresponding flag does not have to be specified in commands. Variables marked as required can instead be set as flags.

Name Default Value Required Description
KPOPS_CONFIG_PATH . False Path to the dir containing config.yaml files
KPOPS_DOTENV_PATH False Path to dotenv file. Multiple files can be provided. The files will be loaded in order, with each file overriding the previous one.
KPOPS_ENVIRONMENT False The environment you want to generate and deploy the pipeline to. Suffix your environment files with this value (e.g. defaults_development.yaml for environment=development).
KPOPS_PIPELINE_PATHS True Paths to dir containing 'pipeline.yaml' or files named 'pipeline.yaml'.
KPOPS_PIPELINE_STEPS False Comma separated list of steps to apply the command on
cli_env_vars.env
Exhaustive list of all cli-related environment variables
# CLI Environment variables
#
# The default setup is shown. These variables take precedence over the
# commands' flags. If a variable is set, the corresponding flag does
# not have to be specified in commands. Variables marked as required
# can instead be set as flags.
#
# Path to the dir containing config.yaml files
KPOPS_CONFIG_PATH=.
# Path to dotenv file. Multiple files can be provided. The files will
# be loaded in order, with each file overriding the previous one.
KPOPS_DOTENV_PATH # No default value, not required
# The environment you want to generate and deploy the pipeline to.
# Suffix your environment files with this value (e.g.
# defaults_development.yaml for environment=development).
KPOPS_ENVIRONMENT # No default value, not required
# Paths to dir containing 'pipeline.yaml' or files named
# 'pipeline.yaml'.
KPOPS_PIPELINE_PATHS # No default value, required
# Comma separated list of steps to apply the command on
KPOPS_PIPELINE_STEPS # No default value, not required