Skip to content

ATM fraud detection pipeline

ATM fraud is a demo pipeline for ATM fraud detection. The original by Confluent is written in KSQL and outlined in this blogpost. The one used in this example is re-built from scratch using bakdata's streams-bootstrap library.

What this will demonstrate

  • Deploying a PostgreSQL database using Helm
  • Deploying a pipeline using KPOps
  • Destroying a pipeline using KPOps

Prerequisites

Completed all steps in the setup.

Setup and deployment

PostgreSQL

Deploy PostgreSQL using the Bitnami Helm chart: Add the helm repository:

1
2
helm repo add bitnami https://charts.bitnami.com/bitnami && \
helm repo update

Install the PostgreSQL with helm:

1
2
3
helm upgrade --install -f ./postgresql.yaml \
--namespace kpops \
postgresql bitnami/postgresql
PostgreSQL Example Helm chart values (postgresql.yaml) 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
auth:
  database: app_db
  enablePostgresUser: true
  password: AppPassword
  postgresPassword: StrongPassword
  username: app1
primary:
  persistence:
    enabled: false
    existingClaim: postgresql-data-claim
volumePermissions:
  enabled: true

ATM fraud detection example pipeline setup

Port forwarding

Before we deploy the pipeline, we need to forward the ports of kafka-rest-proxy and kafka-connect. Run the following commands in two different terminals.

1
kubectl port-forward --namespace kpops service/k8kafka-cp-rest 8082:8082

1
kubectl port-forward --namespace kpops service/k8kafka-cp-kafka-connect 8083:8083

Deploying the ATM fraud detection pipeline

  1. Clone the kpops-examples repository and cd into the directory.

  2. Install KPOps pip install -r requirements.txt.

  3. Export environment variables in your terminal:

    1
2
    export DOCKER_REGISTRY=bakdata && \
export NAMESPACE=kpops

  4. Deploy the pipeline

    1
    kpops deploy atm-fraud/pipeline.yaml --execute

Note

You can use the --dry-run flag instead of the --execute flag and check the logs if your pipeline will be deployed correctly.

Check if the deployment is successful

You can use the Streams Explorer to see the deployed pipeline. To do so, port-forward the service in a separate terminal session using the command below:

1
kubectl port-forward -n kpops service/streams-explorer 8080:8080

After that open http://localhost:8080 in your browser. You should be able to see pipeline shown in the image below:

atm-fraud-pipeline

An overview of ATM fraud pipeline shown in Streams Explorer

Attention

Kafka Connect needs some time to set up the connector. Moreover, Streams Explorer needs a while to scrape the information from Kafka connect. Therefore, it might take a bit until you see the whole graph.

Teardown resources

PostrgreSQL

PostgreSQL can be uninstalled by running the following command:

1
helm --namespace kpops uninstall postgresql

ATM fraud pipeline

  1. Export environment variables in your terminal.

    1
2
    export DOCKER_REGISTRY=bakdata && \
export NAMESPACE=kpops

  2. Remove the pipeline

    1
    kpops clean atm-fraud/pipeline.yaml --verbose  --execute

Note

You can use the --dry-run flag instead of the --execute flag and check the logs if your pipeline will be destroyed correctly.

Attention

If you face any issues destroying this example see Teardown for manual deletion.

Common errors

  • deploy fails:
  • Read the error message.
  • Try to correct the mistakes if there were any. Likely the configuration is incorrect, or the port-forwarding is not working as intended.
  • Run clean.
  • Run deploy --dry-run to avoid havig to clean again. If an error is dropped, start over from step 1.
  • If the dry-run is succesful, run deploy.
  • clean fails:
  • Read the error message.
  • Try to correct the indicated mistakes if there were any. Likely the configuration is incorrect, or the port-forwarding is not working as intended.
  • Run clean.
  • If clean fails, follow the steps in teardown.