Skip to content

Setup Quick

In this part, you will set up a Quick cluster. This includes:

  • optionally creating a local Kubernetes cluster
  • running Apache Kafka and Confluent's Schema Registry
  • deploying Quick

Prerequisites

Setup Kubernetes with k3d

If you don't have access to an existing Kubernetes cluster, this section will guide you through creating a local cluster. We recommend the lightweight Kubernetes distribution k3s for this. k3d is a wrapper around k3s in Docker that lets you get started fast. You can install it with k3d's installation script:

wget -q -O - https://raw.githubusercontent.com/k3d-io/k3d/v5.3.0/install.sh | bash
For other ways of installing k3d, you can have a look at their installation guide.

Attention

k3s includes Traefik as a load balancer. If you want to use a different Kubernetes distribution, you might have to install Traefik separately. For more information, please refer to the Traefik deployment section.

With k3d installed, you can create a new cluster called quick:

k3d cluster create quick

Note

This automatically configures kubectl to connect to the local cluster by modifying your ~/.kube/config. In case you manually set the KUBECONFIG variable or don't want that k3d modifies your config, k3d offers many other options.

After the command has run, you can check the cluster status with kubectl get pods -n kube-system. When all returned elements have a STATUS of Running or Completed, you can create a connection to the load balancer. For that, open a new terminal and run:

export QUICK_PORT=8000
kubectl port-forward -n kube-system deployment/traefik $QUICK_PORT:8000
set QUICK_PORT 8000
kubectl port-forward -n kube-system deployment/traefik $QUICK_PORT:8000

This terminal session has to remain open to keep the connection alive. Further, you can choose a different port if it's already used.

In the following, the guide will use the variables QUICK_HOST and QUICK_URL to refer to the load balancer. Set them like this:

export QUICK_HOST="localhost:$QUICK_PORT"
export QUICK_URL="http://$QUICK_HOST"
set QUICK_HOST "localhost:$QUICK_PORT"
set QUICK_URL "http://$QUICK_HOST"

Traefik (optional)

k3s uses Traefik as its load balancer. If you are using k3s as your Kubernetes distribution, you can go directly to the Kafka deployment section. However, if you use another Kubernetes distribution, you can use this guide to deploy Traefik to your Kubernetes cluster.

This section provides a step-by-step guide on deploying Traefik to your Kubernetes cluster. Traefik is the Ingress controller Quick needs for load balancing incoming requests. We recommend the official Helm chart to deploy Traefik.

  1. Add Traefik Helm repository

    helm repo add traefik https://helm.traefik.io/traefik && helm repo update
    

  2. Deploy Traefik with Helm

    helm upgrade --install traefik traefik/traefik \
        --namespace infrastructure
    

Note

This guide uses the default values of Traefik's Helm charts for the deployment. For instance, Traefik won't use TLS with this configuration. For more information on how you can enable TLS, please refer to the Traefik documentation.

Kafka

To deploy Kafka, this guide uses Confluent's Helm Chart.

  1. Add Confluent's Helm Chart repository

    helm repo add confluentinc https://confluentinc.github.io/cp-helm-charts/ &&  
    helm repo update
    
  2. Install Kafka, Zookeeper, and the Schema Registry. A single Helm chart installs all three components. Below you can find an example for the --values ./kafka.yaml file configuring the deployment.

    helm upgrade \
        --install \
        --version 0.6.1 \
        --values ./kafka.yaml \
        --namespace infrastructure \
        --create-namespace \
        --wait \
        k8kafka confluentinc/cp-helm-charts
    

Kafka Helm Chart Values (kafka.yaml)

An example value configuration for Confluent's Helm chart. This configuration deploys a single Broker, a Schema Registry and Zookeeper with minimal resources.

kafka.yaml
cp-zookeeper:
  enabled: true
  imageTag: 7.1.3
  servers: 1
  heapOptions: "-Xms124M -Xmx124M"
  overrideGroupId: k8kafka
  fullnameOverride: "k8kafka-cp-zookeeper"
  resources:
    requests:
      cpu: 50m
      memory: 0.2G
    limits:
      cpu: 250m
      memory: 0.2G
  prometheus:
    jmx:
      enabled: false

cp-kafka:
  brokers: 1
  imageTag: 7.1.3
  enabled: true
  podManagementPolicy: Parallel
  configurationOverrides:
    "auto.create.topics.enable": false
    "offsets.topic.replication.factor": 1
    "transaction.state.log.replication.factor": 1
    "transaction.state.log.min.isr": 1
  resources:
    requests:
      cpu: 50m
      memory: 0.5G
    limits:
      cpu: 250m
      memory: 0.5G
  prometheus:
    jmx:
      enabled: false
  persistence:
    enabled: false

cp-schema-registry:
  enabled: true
  imageTag: 7.1.3
  fullnameOverride: "k8kafka-cp-schema-registry"
  overrideGroupId: "k8kafka"
  kafka:
    bootstrapServers: "PLAINTEXT://k8kafka-cp-kafka-headless:9092"
  resources:
    requests:
      cpu: 50m
      memory: 0.25G
    limits:
      cpu: 250m
      memory: 0.25G
  prometheus:
    jmx:
      enabled: false

cp-ksql-server:
  enabled: false
cp-kafka-connect:
  enabled: false
cp-control-center:
  enabled: false
cp-kafka-rest:
  enabled: false

Depending on your system, it can take a couple of minutes before all components are up and running. You can view the status of the created pods by running kubectl get pods -n infrastructure. You should now have Zookeeper, Kafka, and the Schema Registry running in a namespace called infrastructure.

In the Kubernetes cluster, you can connect to Kafka with k8kafka-cp-kafka.infrastructure:9092 and the Schema Registry with http://k8kafka-cp-schema-registry.infrastructure:8081. You are now set to deploy Quick itself.

Quick

Quick comes with its Helm chart for installing it in Kubernetes clusters.

  1. Add the Quick Helm chart repository and update the index:
    helm repo add quick https://bakdata.github.io/quick && helm repo update
    
  2. Create a random secret key called QUICK_API_KEY:

    export QUICK_API_KEY="random-key"
    
    set QUICK_API_KEY "random-key"
    
  3. Install Quick with Helm. Below is an example quick.yaml as a configuration for Quick in local clusters.

    helm upgrade --install quick quick/quick \
      --namespace quick \
      --version 0.9.0-dev \
      --create-namespace \
      --set apiKey="$QUICK_API_KEY" \
      -f "./quick.yaml"
    

Quick Helm Chart Values (quick.yaml)

An example configuration for local Kubernetes clusters. It lets Quick work with a single Kafka broker and HTTP instead of HTTPS. For more information about Quick's Helm Chart configuration, please see the reference.

quick.yaml
image:
  pullPolicy: "Always"
  tag: "0.9.0-dev"

ingress:
  ssl: False
  entrypoint: "web"

manager:
  name: "quick-manager"
  replicaCount: 1

ingest:
  name: "quick-ingest"
  replicaCount: 1

quickConfig:
  QUICK_DEFAULT_REPLICAS: "1"
  QUICK_KAFKA_BOOTSTRAP_SERVER: k8kafka-cp-kafka.infrastructure:9092
  QUICK_KAFKA_INTERNAL_PARTITIONS: "3"
  QUICK_KAFKA_INTERNAL_REPLICATION_FACTOR: "1"
  QUICK_KAFKA_SCHEMA_REGISTRY_URL: http://k8kafka-cp-schema-registry.infrastructure:8081
  QUICK_TOPIC_REGISTRY_PARTITIONS: "3"
  QUICK_TOPIC_REGISTRY_REPLICATION_FACTOR: "1"
  QUICK_TOPIC_REGISTRY_SERVICE_NAME: internal-topic-registry
  QUICK_TOPIC_REGISTRY_TOPIC_NAME: __topic-registry
  QUICK_SCHEMA_FORMAT: "Avro"
  QUICK_SCHEMA_AVRO_NAMESPACE: "quick"

You can check the status of Quick by running kubectl get pods -n quick. There should be three running pods: quick-manager, quick-ingest, and internal-topics-registry.