Setup Quick¶
In this part, you will set up a Quick cluster. This includes:
- optionally creating a local Kubernetes cluster
- running Apache Kafka and Confluent's Schema Registry
- deploying Quick
Prerequisites¶
- k3d (Version 5.3.0+) and Docker (Version >= v20.10.5) or an existing Kubernetes cluster (>= 1.21.0)
- kubectl (Compatible with server version 1.21.0)
- Helm (Version 3.8.0+)
Setup Kubernetes with k3d¶
If you don't have access to an existing Kubernetes cluster, this section will guide you through creating a local cluster. We recommend the lightweight Kubernetes distribution k3s for this. k3d is a wrapper around k3s in Docker that lets you get started fast. You can install it with k3d's installation script:
wget -q -O - https://raw.githubusercontent.com/k3d-io/k3d/v5.3.0/install.sh | bash
Attention
k3s includes Traefik as a load balancer. If you want to use a different Kubernetes distribution, you might have to install Traefik separately. For more information, please refer to the Traefik deployment section.
With k3d installed, you can create a new cluster called quick
:
k3d cluster create quick
Note
This automatically configures kubectl
to connect to the local cluster by modifying your ~/.kube/config
.
In case you manually set the KUBECONFIG
variable or don't want that k3d modifies your config,
k3d offers many other options.
After the command has run, you can check the cluster status with kubectl get pods -n kube-system
.
When all returned elements have a STATUS
of Running
or Completed
,
you can create a connection to the load balancer.
For that, open a new terminal and run:
export QUICK_PORT=8000
kubectl port-forward -n kube-system deployment/traefik $QUICK_PORT:8000
set QUICK_PORT 8000
kubectl port-forward -n kube-system deployment/traefik $QUICK_PORT:8000
This terminal session has to remain open to keep the connection alive. Further, you can choose a different port if it's already used.
In the following, the guide will use the variables QUICK_HOST
and QUICK_URL
to refer to the load balancer.
Set them like this:
export QUICK_HOST="localhost:$QUICK_PORT"
export QUICK_URL="http://$QUICK_HOST"
set QUICK_HOST "localhost:$QUICK_PORT"
set QUICK_URL "http://$QUICK_HOST"
Traefik (optional)¶
k3s uses Traefik as its load balancer. If you are using k3s as your Kubernetes distribution, you can go directly to the Kafka deployment section. However, if you use another Kubernetes distribution, you can use this guide to deploy Traefik to your Kubernetes cluster.
This section provides a step-by-step guide on deploying Traefik to your Kubernetes cluster. Traefik is the Ingress controller Quick needs for load balancing incoming requests. We recommend the official Helm chart to deploy Traefik.
-
Add Traefik Helm repository
helm repo add traefik https://helm.traefik.io/traefik && helm repo update
-
Deploy Traefik with Helm
helm upgrade --install traefik traefik/traefik \ --namespace infrastructure
Note
This guide uses the default values of Traefik's Helm charts for the deployment. For instance, Traefik won't use TLS with this configuration. For more information on how you can enable TLS, please refer to the Traefik documentation.
Kafka¶
To deploy Kafka, this guide uses Confluent's Helm Chart.
-
Add Confluent's Helm Chart repository
helm repo add confluentinc https://confluentinc.github.io/cp-helm-charts/ && helm repo update
-
Install Kafka, Zookeeper, and the Schema Registry. A single Helm chart installs all three components. Below you can find an example for the
--values ./kafka.yaml
file configuring the deployment.helm upgrade \ --install \ --version 0.6.1 \ --values ./kafka.yaml \ --namespace infrastructure \ --create-namespace \ --wait \ k8kafka confluentinc/cp-helm-charts
Kafka Helm Chart Values (kafka.yaml
)
An example value configuration for Confluent's Helm chart. This configuration deploys a single Broker, a Schema Registry and Zookeeper with minimal resources.
cp-zookeeper:
enabled: true
imageTag: 7.1.3
servers: 1
heapOptions: "-Xms124M -Xmx124M"
overrideGroupId: k8kafka
fullnameOverride: "k8kafka-cp-zookeeper"
resources:
requests:
cpu: 50m
memory: 0.2G
limits:
cpu: 250m
memory: 0.2G
prometheus:
jmx:
enabled: false
cp-kafka:
brokers: 1
imageTag: 7.1.3
enabled: true
podManagementPolicy: Parallel
configurationOverrides:
"auto.create.topics.enable": false
"offsets.topic.replication.factor": 1
"transaction.state.log.replication.factor": 1
"transaction.state.log.min.isr": 1
resources:
requests:
cpu: 50m
memory: 0.5G
limits:
cpu: 250m
memory: 0.5G
prometheus:
jmx:
enabled: false
persistence:
enabled: false
cp-schema-registry:
enabled: true
imageTag: 7.1.3
fullnameOverride: "k8kafka-cp-schema-registry"
overrideGroupId: "k8kafka"
kafka:
bootstrapServers: "PLAINTEXT://k8kafka-cp-kafka-headless:9092"
resources:
requests:
cpu: 50m
memory: 0.25G
limits:
cpu: 250m
memory: 0.25G
prometheus:
jmx:
enabled: false
cp-ksql-server:
enabled: false
cp-kafka-connect:
enabled: false
cp-control-center:
enabled: false
cp-kafka-rest:
enabled: false
Depending on your system, it can take a couple of minutes before all components are up and running.
You can view the status of the created pods by running kubectl get pods -n infrastructure
.
You should now have Zookeeper, Kafka, and the Schema Registry running in a namespace called infrastructure
.
In the Kubernetes cluster,
you can connect to Kafka with k8kafka-cp-kafka.infrastructure:9092
and the Schema Registry with http://k8kafka-cp-schema-registry.infrastructure:8081
.
You are now set to deploy Quick itself.
Quick¶
Quick comes with its Helm chart for installing it in Kubernetes clusters.
- Add the Quick Helm chart repository and update the index:
helm repo add quick https://bakdata.github.io/quick && helm repo update
-
Create a random secret key called
QUICK_API_KEY
:export QUICK_API_KEY="random-key"
set QUICK_API_KEY "random-key"
-
Install Quick with Helm. Below is an example
quick.yaml
as a configuration for Quick in local clusters.helm upgrade --install quick quick/quick \ --namespace quick \ --version 0.9.0-dev \ --create-namespace \ --set apiKey="$QUICK_API_KEY" \ -f "./quick.yaml"
Quick Helm Chart Values (quick.yaml
)
An example configuration for local Kubernetes clusters. It lets Quick work with a single Kafka broker and HTTP instead of HTTPS. For more information about Quick's Helm Chart configuration, please see the reference.
image:
pullPolicy: "Always"
tag: "0.9.0-dev"
ingress:
ssl: False
entrypoint: "web"
manager:
name: "quick-manager"
replicaCount: 1
ingest:
name: "quick-ingest"
replicaCount: 1
quickConfig:
QUICK_DEFAULT_REPLICAS: "1"
QUICK_KAFKA_BOOTSTRAP_SERVER: k8kafka-cp-kafka.infrastructure:9092
QUICK_KAFKA_INTERNAL_PARTITIONS: "3"
QUICK_KAFKA_INTERNAL_REPLICATION_FACTOR: "1"
QUICK_KAFKA_SCHEMA_REGISTRY_URL: http://k8kafka-cp-schema-registry.infrastructure:8081
QUICK_TOPIC_REGISTRY_PARTITIONS: "3"
QUICK_TOPIC_REGISTRY_REPLICATION_FACTOR: "1"
QUICK_TOPIC_REGISTRY_SERVICE_NAME: internal-topic-registry
QUICK_TOPIC_REGISTRY_TOPIC_NAME: __topic-registry
QUICK_SCHEMA_FORMAT: "Avro"
QUICK_SCHEMA_AVRO_NAMESPACE: "quick"
You can check the status of Quick by running kubectl get pods -n quick
.
There should be three running pods: quick-manager
, quick-ingest
, and internal-topics-registry
.