The docker image is available on Docker Hub

Kafka broker with SSL enabled using Docker

Table of Contents

• Installation (three ways)
  • a. Docker Compose
  • b. Existing docker image
  • c. Building a local image
    • Keystore file
• Environment variables
• Purpose
• Description
  • Kafka with SSL
    • SSL
    • Certificate creation and signing

Installation (three ways)

a. Docker Compose

1. Copy the below block into a file docker-compose.yml

version: "3"
services:
  kafka:
    image: "pardhu1212/kafka-ssl:0.1.0"
    ports:
      # SSL port
      - "9093:9093"
      # Plaintext port
      - "9094:9094"
      # zookeeper
      - "2181:2181"
    init: true
    environment:
      PASSWORD: "PAss$$123worD"
      # This can be domain name or IP address
      DOMAIN: "www.mywebsite.com"
    container_name: kafkassl

1. Run the command

   docker compose up -d
   

b. Existing docker image

1. Pull the image:

   docker pull pardhu1212/kafka-ssl:0.1.0
   

2. Run the container using the command:

   docker run --init -d -p 9093:9093 -p 9094:9094 --name=kafkassl -e PASSWORD=password DOMAIN=www.mywebsite.com pardhu1212/kafka-ssl
   

c. Building a local image

1. Ruild the docker image using

   docker build -t kafka-ssl-local
   

2. Run the container using

   docker run --init -d -p 9093:9093 -p 9094:9094 --name=kafkassl kafka-ssl-local
   

Keystore file

Access the generated key store file by using command docker cp kafkassl:/kafka_2.12-2.5.0/ssl/server.keystore.jks keystore.jks

Environment variables

Variable	Default value	Importance	Description
PASSWORD	password	HIGH	The password that will be used to create keystore file. Must be 8 or more characters.
DOMAIN	www.mywebsite.com	HIGH	Domain name to be used while creating the certificate.
KAFKA_HOME	/kafka_2.12-2.5.0/	LOW	Directory where Kafka is installed inside the container.
KEY_STORE	/kafka_2.12-2.5.0/ssl/server.keystore.jks	LOW	Keystore jks file path to be used inside docker container.

Purpose

• The primary purpose of the project is to create a kafka container with SSL enabled.
• The secondary goal of the project is to learn about kafka with SSL, docker commands and an important supervisor process called runit.

Description

Kafka with SSL

In the file prepStartup.sh we can notice different openssl and keytool commands. To understand what we are doing here, we need to have a basic understanding of how SSL works.

SSL

When a server is SSL enabled, it provides a certificate and the client validates it. When we browse for https://www.google.com, the Google server first responds with a certificate along with some details, Your browser has a list of certificates(in fact Certifcate Authorities) that it will trust. Since the Google’s certificate is signed by a trustworthy Certifcate Authority(CA) like Verizon, your browser allows further connection.

Kafka SSL also works in a similar way. If you create a kafka broker (an equivalent of Google server), you want to make it SSL enabled, you have to provide a certificate. This certificate should be signed by a certificate authority. In the production use case, you have to create the certificate and mail it to an actual and trusted CA so that they will sign it. Then you can use this whenever a client tries to connect to you.

But how can we achieve this in a development scenario? Then you can create your own CA and sign your own certificate. The shell script does exactly that. It will create a certifcate, sign it with a self created CA and store them in a keystore file. We use Keytool(provided by Java) and Openssl to create them.

Certificate creation and signing

• The content of the certificate file is encrypted using an algorithm. Most people use RSA and the same has been used here.
• The validity of the certifcate needs to be specified. Here the validity is chosen as 365 days from the day of creation.
• The most important details when creating the certificate is the Common Name(CN). Most clients who receive the SSL certifcate will verify the name of the domain with the CN of the certifcate. If it does not match, although the certificate is issued a trusted CA, the connection might be rejected.