Security Configuration Guide

Complete guide for securing Kafka connections with SSL/TLS encryption and SASL authentication.

Overview
Security Protocols
SSL/TLS Encryption
SASL Authentication
Cloud Provider Examples
Troubleshooting
Best Practices

Overview

StreamForge supports all standard Kafka security features:

Feature	Support	Use Case
SSL/TLS	✅ Full	Encrypted connections
Mutual TLS	✅ Full	Certificate-based authentication
SASL/PLAIN	✅ Full	Username/password (simple)
SASL/SCRAM-SHA-256	✅ Full	Username/password (secure)
SASL/SCRAM-SHA-512	✅ Full	Username/password (more secure)
SASL/GSSAPI	✅ Full	Kerberos authentication
SASL/OAUTHBEARER	✅ Full	OAuth 2.0 token authentication

Security Protocols

Kafka supports four security protocols:

1. PLAINTEXT (Default)

No encryption, no authentication. Not recommended for production.

# No security configuration needed
appid: mirrormaker
bootstrap: kafka:9092

2. SSL

Encryption only, optional certificate-based authentication (mutual TLS).

security:
  protocol: SSL
  ssl:
    ca_location: /path/to/ca-cert.pem

3. SASL_PLAINTEXT

Authentication without encryption. Not recommended for production.

security:
  protocol: SASL_PLAINTEXT
  sasl:
    mechanism: PLAIN
    username: user
    password: pass

4. SASL_SSL (Recommended)

Both encryption (SSL) and authentication (SASL).

security:
  protocol: SASL_SSL
  ssl:
    ca_location: /path/to/ca-cert.pem
  sasl:
    mechanism: SCRAM-SHA-256
    username: user
    password: pass

SSL/TLS Encryption

Simple SSL (One-Way TLS)

Client verifies broker’s certificate:

security:
  protocol: SSL
  ssl:
    # CA certificate to verify broker
    ca_location: /path/to/ca-cert.pem

    # Verify broker hostname (recommended)
    endpoint_identification_algorithm: https

Use Case: Basic encryption for data in transit.

Mutual TLS (mTLS)

Both client and broker verify each other:

security:
  protocol: SSL
  ssl:
    # CA certificate to verify broker
    ca_location: /path/to/ca-cert.pem

    # Client certificate for authentication
    certificate_location: /path/to/client-cert.pem
    key_location: /path/to/client-key.pem
    key_password: optional-key-password

    # Verify broker hostname
    endpoint_identification_algorithm: https

Use Case: Certificate-based authentication, high security environments.

Generating SSL Certificates

# Generate CA certificate
openssl req -new -x509 -keyout ca-key.pem -out ca-cert.pem -days 365

# Generate client key and certificate
openssl req -new -keyout client-key.pem -out client-cert-req.pem -days 365
openssl x509 -req -in client-cert-req.pem -CA ca-cert.pem -CAkey ca-key.pem \
  -CAcreateserial -out client-cert.pem -days 365

SASL Authentication

SASL/PLAIN

Simple username/password authentication:

security:
  protocol: SASL_SSL  # Always use SSL with PLAIN
  ssl:
    ca_location: /path/to/ca-cert.pem
  sasl:
    mechanism: PLAIN
    username: your-username
    password: your-password

Pros:

✅ Simple to configure
✅ Works with most Kafka brokers

Cons:

⚠️ Password transmitted in plain text (must use SSL!)
⚠️ Less secure than SCRAM

Use Case: Development, testing, Confluent Cloud.

SASL/SCRAM-SHA-256

Secure Challenge-Response Authentication Mechanism:

security:
  protocol: SASL_SSL
  ssl:
    ca_location: /path/to/ca-cert.pem
  sasl:
    mechanism: SCRAM-SHA-256
    username: your-username
    password: your-password

Pros:

✅ Password never transmitted over network
✅ Mutual authentication
✅ Replay attack protection

Use Case: Modern Kafka clusters, AWS MSK, production environments.

SASL/SCRAM-SHA-512

More secure variant of SCRAM:

security:
  protocol: SASL_SSL
  ssl:
    ca_location: /path/to/ca-cert.pem
  sasl:
    mechanism: SCRAM-SHA-512  # Changed from SHA-256
    username: your-username
    password: your-password

Use Case: High-security environments requiring stronger hashing.

SASL/GSSAPI (Kerberos)

Enterprise authentication with Kerberos:

security:
  protocol: SASL_SSL
  ssl:
    ca_location: /path/to/ca-cert.pem
  sasl:
    mechanism: GSSAPI
    kerberos_service_name: kafka
    kerberos_principal: client@EXAMPLE.COM
    kerberos_keytab: /path/to/client.keytab

Prerequisites:

Install Kerberos libraries:

# Ubuntu/Debian
apt-get install libkrb5-dev

# RHEL/CentOS
yum install krb5-devel

Configure /etc/krb5.conf:

[libdefaults]
  default_realm = EXAMPLE.COM

[realms]
  EXAMPLE.COM = {
    kdc = kdc.example.com
    admin_server = admin.example.com
  }

Test Kerberos:

kinit -kt /path/to/client.keytab client@EXAMPLE.COM
klist  # Verify ticket

Use Case: Enterprise Hadoop/Kafka clusters, legacy systems.

Cloud Provider Examples

Confluent Cloud

appid: mirrormaker-confluent
bootstrap: pkc-xxxxx.us-east-1.aws.confluent.cloud:9092
input: source-topic
output: destination-topic

security:
  protocol: SASL_SSL
  sasl:
    mechanism: PLAIN
    username: <API_KEY>
    password: <API_SECRET>

How to get credentials:

Go to Confluent Cloud Console
Select your cluster
API Keys → Create Key
Copy API Key (username) and Secret (password)

AWS MSK (Managed Streaming for Kafka)

Option 1: SASL/SCRAM

appid: mirrormaker-msk
bootstrap: b-1.msk-cluster.xxxxx.kafka.us-east-1.amazonaws.com:9096
input: source-topic
output: destination-topic

security:
  protocol: SASL_SSL
  sasl:
    mechanism: SCRAM-SHA-512
    username: <SECRET_USERNAME>
    password: <SECRET_PASSWORD>

How to set up:

Create secret in AWS Secrets Manager
Associate secret with MSK cluster
Use secret values as username/password

Option 2: IAM Authentication

appid: mirrormaker-msk-iam
bootstrap: b-1.msk-cluster.xxxxx.kafka.us-east-1.amazonaws.com:9098
input: source-topic
output: destination-topic

# For IAM auth, use custom properties
consumer_properties:
  security.protocol: SASL_SSL
  sasl.mechanism: AWS_MSK_IAM
  sasl.jaas.config: software.amazon.msk.auth.iam.IAMLoginModule required;
  sasl.client.callback.handler.class: software.amazon.msk.auth.iam.IAMClientCallbackHandler

producer_properties:
  security.protocol: SASL_SSL
  sasl.mechanism: AWS_MSK_IAM
  sasl.jaas.config: software.amazon.msk.auth.iam.IAMLoginModule required;
  sasl.client.callback.handler.class: software.amazon.msk.auth.iam.IAMClientCallbackHandler

Azure Event Hubs (Kafka Protocol)

appid: mirrormaker-eventhubs
bootstrap: <NAMESPACE>.servicebus.windows.net:9093
input: source-topic
output: destination-topic

security:
  protocol: SASL_SSL
  sasl:
    mechanism: PLAIN
    username: $ConnectionString
    password: Endpoint=sb://<NAMESPACE>.servicebus.windows.net/;SharedAccessKeyName=<KEY_NAME>;SharedAccessKey=<KEY>

Troubleshooting

SSL Certificate Issues

Problem: SSL handshake failed

Solutions:

Verify CA certificate path:

openssl verify -CAfile ca-cert.pem broker-cert.pem

Check certificate expiration:

openssl x509 -in ca-cert.pem -noout -dates

Disable hostname verification (testing only):

ssl:
  endpoint_identification_algorithm: ""

SASL Authentication Issues

Problem: Authentication failed

Solutions:

Verify credentials are correct
Check SASL mechanism matches broker configuration

For SCRAM, ensure user exists on broker:

kafka-configs.sh --bootstrap-server kafka:9092 \
  --describe --entity-type users

Kerberos Issues

Problem: GSSAPI authentication failed

Solutions:

Verify Kerberos ticket:
```
klist -e
```
Check keytab:
```
klist -kt client.keytab
```

Test kinit:

kinit -kt client.keytab client@EXAMPLE.COM

Check service name matches broker configuration

Connection Timeout

Problem: Connection times out

Solutions:

Verify broker hostname/port
Check firewall rules allow port (9093, 9094, etc.)
Verify security group rules (cloud providers)
Test with openssl:
```
openssl s_client -connect kafka:9093
```

Best Practices

1. Always Use Encryption

✅ Do:

security:
  protocol: SASL_SSL  # SSL encryption enabled

❌ Don’t:

security:
  protocol: SASL_PLAINTEXT  # No encryption!

2. Secure Credential Storage

✅ Do: Use environment variables or secret management

export KAFKA_USERNAME="myuser"
export KAFKA_PASSWORD="mypass"

❌ Don’t: Store passwords in configuration files

sasl:
  password: "plaintext-password-in-git"  # Bad!

3. Use Strong Authentication

Security Ranking:

🥇 Mutual TLS (mTLS) - Best
🥈 SASL/SCRAM-SHA-512 - Very Good
🥉 SASL/SCRAM-SHA-256 - Good
⚠️ SASL/PLAIN - Acceptable with SSL
❌ PLAINTEXT - Never use in production

4. Certificate Management

✅ Rotate certificates regularly (90 days recommended)
✅ Use separate certificates for each service
✅ Monitor certificate expiration
✅ Keep private keys secure (chmod 600)

5. Network Security

✅ Use VPN or private networks for Kafka traffic
✅ Restrict broker access with firewall rules
✅ Use separate security groups for Kafka
✅ Enable VPC peering for cross-account access (AWS)

6. Monitoring

Monitor these metrics:

Authentication failures
SSL handshake errors
Certificate expiration warnings
Connection timeouts

7. Testing

Test security configuration before production:

# Test SSL connection
openssl s_client -connect kafka:9093 -CAfile ca-cert.pem

# Test with kafkacat
kafkacat -b kafka:9093 -L \
  -X security.protocol=SASL_SSL \
  -X sasl.mechanism=SCRAM-SHA-256 \
  -X sasl.username=user \
  -X sasl.password=pass

# Test with MirrorMaker
CONFIG_FILE=examples/config.security-sasl-scram.yaml cargo run

Configuration Examples

All security examples are in the examples/ folder:

config.security-ssl.yaml - SSL/TLS encryption
config.security-sasl-plain.yaml - SASL/PLAIN authentication
config.security-sasl-scram.yaml - SASL/SCRAM authentication
config.security-kerberos.yaml - Kerberos authentication

References

Official Documentation

Cloud Provider Guides

Quick Reference

Security Configuration Template

security:
  # Protocol: PLAINTEXT | SSL | SASL_PLAINTEXT | SASL_SSL
  protocol: SASL_SSL

  # SSL Configuration (for SSL or SASL_SSL)
  ssl:
    ca_location: /path/to/ca-cert.pem
    certificate_location: /path/to/client-cert.pem  # Optional (mTLS)
    key_location: /path/to/client-key.pem          # Optional (mTLS)
    key_password: key-password                      # Optional
    endpoint_identification_algorithm: https        # Optional

  # SASL Configuration (for SASL_PLAINTEXT or SASL_SSL)
  sasl:
    # Mechanism: PLAIN | SCRAM-SHA-256 | SCRAM-SHA-512 | GSSAPI | OAUTHBEARER
    mechanism: SCRAM-SHA-256

    # For PLAIN/SCRAM
    username: your-username
    password: your-password

    # For GSSAPI (Kerberos)
    kerberos_service_name: kafka
    kerberos_principal: client@REALM
    kerberos_keytab: /path/to/keytab

    # For OAUTHBEARER
    oauthbearer_token: your-token

Need Help? Open an issue on GitHub.

Security Configuration Guide

Table of Contents

Overview

Security Protocols

1. PLAINTEXT (Default)

2. SSL

3. SASL_PLAINTEXT

4. SASL_SSL (Recommended)

SSL/TLS Encryption

Simple SSL (One-Way TLS)

Mutual TLS (mTLS)

Generating SSL Certificates

SASL Authentication

SASL/PLAIN

SASL/SCRAM-SHA-256

SASL/SCRAM-SHA-512

SASL/GSSAPI (Kerberos)

Cloud Provider Examples

Confluent Cloud

AWS MSK (Managed Streaming for Kafka)

Option 1: SASL/SCRAM

Option 2: IAM Authentication

Azure Event Hubs (Kafka Protocol)

Troubleshooting

SSL Certificate Issues

SASL Authentication Issues

Kerberos Issues

Connection Timeout

Best Practices

1. Always Use Encryption

2. Secure Credential Storage

3. Use Strong Authentication

4. Certificate Management

5. Network Security

6. Monitoring

7. Testing

Configuration Examples

References

Official Documentation

Cloud Provider Guides

Quick Reference

Security Configuration Template