Environment variables

Environment variables can be set by using the export command in Linux or the set command in Windows.

dotenv files

KPOps currently supports .env files only for variables related to the config. Full support for .env files is on the roadmap. One of the possible ways to use one and export the contents manually is with the following command: export $(xargs < .env). This would work in bash suppose there are no spaces inside the values.

Config

These variables take precedence over the settings in config.yaml. Variables marked as required can instead be set in the global config.

Name	Default Value	Required	Description	Setting name
KPOPS_COMPONENTS_MODULE		False	Custom Python module defining project-specific KPOps components	components_module
KPOPS_PIPELINE_BASE_DIR	.	False	Base directory to the pipelines (default is current working directory)	pipeline_base_dir
KPOPS_KAFKA_BROKERS		True	The comma separated Kafka brokers address.	kafka_brokers
KPOPS_TOPIC_NAME_CONFIG__DEFAULT_OUTPUT_TOPIC_NAME	${pipeline.name}-${component.name}	False	Configures the value for the variable ${output_topic_name}	topic_name_config.default_output_topic_name
KPOPS_TOPIC_NAME_CONFIG__DEFAULT_ERROR_TOPIC_NAME	${pipeline.name}-${component.name}-error	False	Configures the value for the variable ${error_topic_name}	topic_name_config.default_error_topic_name
KPOPS_SCHEMA_REGISTRY__ENABLED	False	False	Whether the Schema Registry handler should be initialized.	schema_registry.enabled
KPOPS_SCHEMA_REGISTRY__URL	http://localhost:8081/	False	Address of the Schema Registry.	schema_registry.url
KPOPS_SCHEMA_REGISTRY__TIMEOUT	30	False	Operation timeout in seconds.	schema_registry.timeout
KPOPS_KAFKA_REST__URL	http://localhost:8082/	False	Address of the Kafka REST Proxy.	kafka_rest.url
KPOPS_KAFKA_REST__TIMEOUT	30	False	Operation timeout in seconds.	kafka_rest.timeout
KPOPS_KAFKA_CONNECT__URL	http://localhost:8083/	False	Address of Kafka Connect.	kafka_connect.url
KPOPS_KAFKA_CONNECT__TIMEOUT	30	False	Operation timeout in seconds.	kafka_connect.timeout
KPOPS_CREATE_NAMESPACE	False	False	Flag for helm upgrade --install. Create the release namespace if not present.	create_namespace
KPOPS_HELM_CONFIG__CONTEXT		False	Name of kubeconfig context (--kube-context)	helm_config.context
KPOPS_HELM_CONFIG__DEBUG	False	False	Run Helm in Debug mode	helm_config.debug
KPOPS_HELM_CONFIG__API_VERSION		False	Kubernetes API version used for Capabilities.APIVersions	helm_config.api_version
KPOPS_HELM_DIFF_CONFIG__IGNORE		True	Set of keys that should not be checked.	helm_diff_config.ignore
KPOPS_RETAIN_CLEAN_JOBS	False	False	Whether to retain clean up jobs in the cluster or uninstall the, after completion.	retain_clean_jobs

config_env_vars.env

# Global config environment variables
#
# The default setup is shown. These variables take precedence over the
# settings in `config.yaml`. Variables marked as required can instead
# be set in the global config.
#
# components_module
# Custom Python module defining project-specific KPOps components
KPOPS_COMPONENTS_MODULE # No default value, not required
# pipeline_base_dir
# Base directory to the pipelines (default is current working
# directory)
KPOPS_PIPELINE_BASE_DIR=.
# kafka_brokers
# The comma separated Kafka brokers address.
KPOPS_KAFKA_BROKERS # No default value, required
# topic_name_config.default_output_topic_name
# Configures the value for the variable ${output_topic_name}
KPOPS_TOPIC_NAME_CONFIG__DEFAULT_OUTPUT_TOPIC_NAME=${pipeline.name}-${component.name}
# topic_name_config.default_error_topic_name
# Configures the value for the variable ${error_topic_name}
KPOPS_TOPIC_NAME_CONFIG__DEFAULT_ERROR_TOPIC_NAME=${pipeline.name}-${component.name}-error
# schema_registry.enabled
# Whether the Schema Registry handler should be initialized.
KPOPS_SCHEMA_REGISTRY__ENABLED=False
# schema_registry.url
# Address of the Schema Registry.
KPOPS_SCHEMA_REGISTRY__URL=http://localhost:8081/
# schema_registry.timeout
# Operation timeout in seconds.
KPOPS_SCHEMA_REGISTRY__TIMEOUT=30
# kafka_rest.url
# Address of the Kafka REST Proxy.
KPOPS_KAFKA_REST__URL=http://localhost:8082/
# kafka_rest.timeout
# Operation timeout in seconds.
KPOPS_KAFKA_REST__TIMEOUT=30
# kafka_connect.url
# Address of Kafka Connect.
KPOPS_KAFKA_CONNECT__URL=http://localhost:8083/
# kafka_connect.timeout
# Operation timeout in seconds.
KPOPS_KAFKA_CONNECT__TIMEOUT=30
# create_namespace
# Flag for `helm upgrade --install`. Create the release namespace if
# not present.
KPOPS_CREATE_NAMESPACE=False
# helm_config.context
# Name of kubeconfig context (`--kube-context`)
KPOPS_HELM_CONFIG__CONTEXT # No default value, not required
# helm_config.debug
# Run Helm in Debug mode
KPOPS_HELM_CONFIG__DEBUG=False
# helm_config.api_version
# Kubernetes API version used for `Capabilities.APIVersions`
KPOPS_HELM_CONFIG__API_VERSION # No default value, not required
# helm_diff_config.ignore
# Set of keys that should not be checked.
KPOPS_HELM_DIFF_CONFIG__IGNORE # No default value, required
# retain_clean_jobs
# Whether to retain clean up jobs in the cluster or uninstall the,
# after completion.
KPOPS_RETAIN_CLEAN_JOBS=False

CLI

These variables take precedence over the commands' flags. If a variable is set, the corresponding flag does not have to be specified in commands. Variables marked as required can instead be set as flags.

Name	Default Value	Required	Description
KPOPS_CONFIG_PATH	.	False	Path to the dir containing config.yaml files
KPOPS_DOTENV_PATH		False	Path to dotenv file. Multiple files can be provided. The files will be loaded in order, with each file overriding the previous one.
KPOPS_ENVIRONMENT		False	The environment you want to generate and deploy the pipeline to. Suffix your environment files with this value (e.g. defaults_development.yaml for environment=development).
KPOPS_PIPELINE_PATHS		True	Paths to dir containing 'pipeline.yaml' or files named 'pipeline.yaml'.
KPOPS_PIPELINE_STEPS		False	Comma separated list of steps to apply the command on

cli_env_vars.env

# CLI Environment variables
#
# The default setup is shown. These variables take precedence over the
# commands' flags. If a variable is set, the corresponding flag does
# not have to be specified in commands. Variables marked as required
# can instead be set as flags.
#
# Path to the dir containing config.yaml files
KPOPS_CONFIG_PATH=.
# Path to dotenv file. Multiple files can be provided. The files will
# be loaded in order, with each file overriding the previous one.
KPOPS_DOTENV_PATH # No default value, not required
# The environment you want to generate and deploy the pipeline to.
# Suffix your environment files with this value (e.g.
# defaults_development.yaml for environment=development).
KPOPS_ENVIRONMENT # No default value, not required
# Paths to dir containing 'pipeline.yaml' or files named
# 'pipeline.yaml'.
KPOPS_PIPELINE_PATHS # No default value, required
# Comma separated list of steps to apply the command on
KPOPS_PIPELINE_STEPS # No default value, not required