Pulsar Primer¶

Prerequisites: Kafka consumer groups, Kafka internals & Pulsar comparison

← Messaging Overview

Why This Exists¶

You've decided Pulsar might be right for your use case. This page answers: How do you actually run it? What does the architecture look like on day one, how do you configure multi-tenancy, what operational tasks are different from Kafka, and what are the gotchas?

This is not Pulsar marketing. This is what you need to know to operate it in production.

Architecture at a Glance¶

┌─────────────────────────────────────────────────────────┐
│ Pulsar Cluster                                          │
├─────────────────────────────────────────────────────────┤
│                                                         │
│  Brokers (stateless, horizontally scalable)             │
│  ├─ pulsar-broker-0:6650 (broker port)                 │
│  ├─ pulsar-broker-1:6650                               │
│  └─ pulsar-broker-2:6650                               │
│     Each handles: topic assignment, producer/consumer  │
│     Can add/remove without rebalancing                 │
│                                                         │
│  BookKeepers (distributed log storage)                  │
│  ├─ bookkeeper-0 → /data/ledgers (where data lives)    │
│  ├─ bookkeeper-1                                       │
│  └─ bookkeeper-2                                       │
│     Stores actual messages in immutable ledgers        │
│     Can scale independently of brokers                │
│                                                         │
│  Metadata Store (ZooKeeper or etcd)                    │
│  ├─ Cluster topology                                  │
│  ├─ Topic assignments                                 │
│  ├─ Consumer offsets                                  │
│  └─ Tenant/namespace definitions                      │
│                                                         │
│  Optional: Tiered Storage                             │
│  └─ S3/GCS (for ledgers older than threshold)         │
└─────────────────────────────────────────────────────────┘

Key difference from Kafka: Brokers are stateless. All data lives in BookKeepers. This means: - Add a broker = instant (no data rebalancing) - Remove a broker = no cascading effect - A broker restart = clients reconnect, no rebalance

Setup: Minimal Cluster¶

Local Development (Docker)¶

# Docker Compose minimal Pulsar (ZK + Broker + BookKeeper)
version: '3'
services:
  zookeeper:
    image: apachepulsar/pulsar:latest
    command: /pulsar/bin/pulsar zookeeper
    ports:
      - "2181:2181"
    environment:
      - PULSAR_MEM="-Xmx512M"

  bookkeeper:
    image: apachepulsar/pulsar:latest
    command: /pulsar/bin/pulsar bookkeeper
    depends_on:
      - zookeeper
    environment:
      - zkServers=zookeeper:2181
      - PULSAR_MEM="-Xmx512M"

  broker:
    image: apachepulsar/pulsar:latest
    command: /pulsar/bin/pulsar broker
    depends_on:
      - zookeeper
      - bookkeeper
    ports:
      - "6650:6650"      # Binary protocol
      - "8080:8080"      # REST API
    environment:
      - zookeeperServers=zookeeper:2181
      - brokerServiceUrl=pulsar://broker:6650
      - PULSAR_MEM="-Xmx1G"

Production Cluster (3 nodes minimum)¶

# Hardware: 3 machines
# Each: 16 cores, 64GB RAM, 1TB NVMe

# Machine 1: ZK + Broker + BookKeeper
# Machine 2: ZK + Broker + BookKeeper
# Machine 3: ZK + Broker + BookKeeper

# Config: /etc/pulsar/broker.conf
brokerServiceUrl=pulsar://pulsar-1.internal:6650
webServiceUrl=http://pulsar-1.internal:8080
zookeeperServers=pulsar-1.internal,pulsar-2.internal,pulsar-3.internal
managedLedgerDefaultEnsembleSize=3
managedLedgerDefaultWriteQuorum=2
managedLedgerDefaultAckQuorum=2

Ensemble/Quorum explanation (Qa ≤ Qw ≤ E): - EnsembleSize=3 (E): How many bookies are in the ensemble for the ledger - WriteQuorum=2 (Qw): How many bookies receive the write - AckQuorum=2 (Qa): How many of those must ack before the write is committed

Do not define both quorums as "wait for 2 to ack" — Qw is fan-out, Qa is the ack threshold. This is the durability analogue of Kafka RF=3 with a write that must land on Qw and be acked by Qa.

Producing Messages¶

Go Client¶

import "github.com/apache/pulsar-client-go/pulsar"

// Setup
client, err := pulsar.NewClient(pulsar.ClientOptions{
    URL: "pulsar://broker:6650",
})
defer client.Close()

// Create producer
producer, err := client.CreateProducer(pulsar.ProducerOptions{
    Topic: "persistent://public/default/orders",
    // persistent:// = durable (stored in BookKeeper)
    // tenant/namespace/topic structure
})
defer producer.Close()

// Send message
msgID, err := producer.Send(context.Background(), &pulsar.ProducerMessage{
    Payload: []byte(`{"order_id": 123, "amount": 50.00}`),
    Key:     "order_123",  // Route to same partition
})

Key Difference from Kafka¶

Kafka topic: "orders"

Pulsar topic: "persistent://tenant/namespace/topic"
              └─ persistent:// = durable to disk (vs ephemeral = memory)
              └─ public = tenant (default tenant)
              └─ default = namespace (logical group)
              └─ orders = topic name

Multi-tenant example:
  company-a: "persistent://company-a/production/orders"
  company-b: "persistent://company-b/production/orders"

  Each tenant is isolated (separate quotas, policies, replication)

Consuming Messages¶

Go Client with Subscription¶

// Create consumer with subscription
consumer, err := client.Subscribe(pulsar.ConsumerOptions{
    Topic:            "persistent://public/default/orders",
    SubscriptionName: "order-processor",
    Type:             pulsar.Shared,  // Multiple consumers share messages
    // Alternative: pulsar.Exclusive (one consumer, others wait)
    // Alternative: pulsar.Failover (one active, others standby)
    // Alternative: pulsar.KeyShared (key-routing, no rebalance)
})
defer consumer.Close()

// Consume messages
for {
    msg, err := consumer.Receive(context.Background())
    if err != nil {
        log.Fatal(err)
    }

    // Process message
    err = processOrder(msg.Payload())

    if err != nil {
        // Negative ack = redelivery
        consumer.Nack(msg)
    } else {
        // Positive ack = committed
        consumer.Ack(msg)
    }
}

Subscription Types¶

Key advantage: Exclusive/Failover don't rebalance on consumer crash. Active consumer dies → standby takes over immediately (<1s). Kafka: eager rebalances stop group consumption while assignments move (often seconds); cooperative/incremental rebalancing reduces that disruption.

Multi-Tenancy Configuration¶

This is where Pulsar shines. Each customer/team gets isolated resources and policies.

Setup: Create Tenant¶

# Create tenant "acme-corp"
pulsar-admin tenants create acme-corp \
  --allowed-clusters standalone \
  --admin-roles acme-admin

# Create namespace (logical group within tenant)
pulsar-admin namespaces create acme-corp/production
pulsar-admin namespaces create acme-corp/staging

# Set policies per namespace
pulsar-admin namespaces set-retention \
  acme-corp/production \
  --time 7d \
  --size 1000G

pulsar-admin namespaces set-replication-clusters \
  acme-corp/production \
  --clusters us-east,us-west,eu

# Set quotas (max messages/sec, storage)
pulsar-admin namespaces set-namespace-message-rate-limits \
  acme-corp/production \
  --publish-threshold 100000 \
  --consume-threshold 100000

Topics Within Tenant¶

acme-corp/production/orders       → Durable, replicated globally, 7-day retention
acme-corp/production/analytics    → Same policies
acme-corp/staging/test-events     → Different retention (1 day), local only

Each team can manage their own namespace (create topics, set policies)
No cross-team interference

User Access Control¶

# Create user "acme-team"
pulsar-admin tokens create --secret-key secret.key \
  --subject acme-team

# Grant permissions
pulsar-admin namespaces grant-permission acme-corp/production \
  --role acme-team \
  --actions produce,consume,admin \
  --topics "acme-corp/production/*"

# Now acme-team can:
# - Produce to any topic in acme-corp/production
# - Consume from any topic
# - Manage that namespace
# But CANNOT access acme-corp/staging

Compare to Kafka: Kafka has no built-in multi-tenancy. All teams share one cluster, one set of ACLs. Pulsar: tenant-level isolation is baked in.

Tiered Storage (The Cost-Saver)¶

Pulsar can automatically offload old ledgers to S3/GCS, keeping only recent data "hot."

# Configure tiered storage
pulsar-admin ns-isolation-policy set-policy \
  --auto-failover-policy-type min_available \
  --auto-failover-policy-params min_limit=1,usage_threshold=80 \
  acme-corp/production

# Set tiered storage for namespace
pulsar-admin namespaces set-offload-policies \
  acme-corp/production \
  --offload-driver s3 \
  --offload-bucket my-bucket \
  --offload-region us-west-2 \
  --offload-threshold-in-bytes 10737418240  # 10GB; offload when ledger > 10GB

What happens: - Recent messages (hot data) live in BookKeepers (fast, expensive) - Messages older than 10GB offload to S3 (slow, cheap ~$23/TB-month vs hot disk at $10/TB-month in this example — pick one unit; do not mix $/GB-month with $/TB-month) - If a consumer rewinds to old data, Pulsar fetches from S3 on-demand

Cost saving example (same unit: $/TB-month):

Scenario: 100MB/s production rate, 30-day retention
Logical volume: 100MB/s × 2,592,000s ≈ 259TB

Kafka: 3 replicas × 259TB ≈ 777TB on disk
       Cost: ~$7,770/month at $10/TB-month

Pulsar:
  Hot (BookKeeper, 3 days): 100MB/s × 259,200s ≈ 26TB × 3 copies ≈ 78TB
       ≈ $780/month at $10/TB-month
  Cold (S3, 27 days): ≈ 233TB × $23/TB-month ≈ $5,360/month
  (The win vs Kafka is not paying RF=3 on the full 30-day set.
   75TB-at-$750 only works as $/TB, never as $/GB.)

Geo-Replication (Global Cluster)¶

Pulsar makes global replication trivial.

# Create clusters (datacenters)
pulsar-admin clusters create us-east \
  --broker-url http://us-east-broker:8080 \
  --broker-url-tls https://us-east-broker:6651

pulsar-admin clusters create us-west \
  --broker-url http://us-west-broker:8080

pulsar-admin clusters create eu \
  --broker-url http://eu-broker:8080

# Enable replication on namespace
pulsar-admin namespaces set-replication-clusters \
  acme-corp/production \
  --clusters us-east,us-west,eu

# Now every message in acme-corp/production is replicated across all 3 regions
# Consumers in any region read latest
# Latency: ~50-200ms between regions (typical inter-region RTT)

How it works: - Producer writes to leader (e.g., us-east) - Leader replicates to us-west and eu automatically - Consumer in us-west reads local replica (low latency) - If us-east goes down, consumers can fail over to us-west/eu. Async geo-replication is not a zero-loss guarantee — in-flight or not-yet-replicated messages can still be lost; treat it as RPO > 0 unless you designed a synchronous ack across clusters.

Compare to Kafka: You need MirrorMaker or Confluent Replicator (separate tools, complex). Pulsar: one namespace setting, automatic.

Operational Differences from Kafka¶

Operational wins for Pulsar: - No rebalancing cascades - Broker add/remove is 5-minute task, not 5-hour project - Tiered storage avoids disk-size scaling - Geo-replication is operational, not architectural

Pulsar vs Kafka: When to Migrate¶

Migrate to Pulsar if...¶

✓ You have multiple teams/tenants (multi-tenancy isolation is critical)
✓ You need geo-replication (currently use MirrorMaker or accept lag)
✓ You're hitting Kafka's rebalancing pain (hourly cascading rebalances)
✓ You want cheaper long-term storage (tiered storage savings are huge)
✓ You're running Kafka on expensive hardware (BookKeepers are cheaper)
✓ You're deploying Kubernetes (Pulsar Helm charts are excellent)

Stay on Kafka if...¶

✓ You have one team, one use case (Kafka simpler)
✓ You have Kafka expertise in-house (migration tax is real)
✓ Your latency requirement is <5ms p99 (Pulsar adds ~5-10ms due to tiering)
✓ You're deeply integrated with Kafka ecosystem (Connectors, Stream Processing)
✓ Your message throughput is modest (<50MB/s) (Kafka fine at this scale)

Production Checklist¶

Before Launch¶

• [ ] Cluster: 3 nodes minimum (ZK + Broker + BookKeeper on each)
• [ ] Replication: EnsembleSize=3, WriteQuorum=2, AckQuorum=2
• [ ] Retention: Set per-namespace (retention-seconds, retention-size)
• [ ] Tiered storage: Configured for long-term data (if needed)
• [ ] Replication: Replicas across multiple data centers (if HA needed)
• [ ] Monitoring: Prometheus metrics collection
• [ ] Authentication: SASL + Tokens for multi-tenant setup
• [ ] Backups: Backup BookKeeper ledgers and ZK data

Ongoing¶

• [ ] Monitor broker/BookKeeper disk usage (alert at 80%)
• [ ] Monitor consumer lag per subscription (alert if lag > SLO)
• [ ] Monitor replication lag (alert if > 100ms)
• [ ] Health-check topic: produce ping message, consume within SLO
• [ ] Upgrade strategy: Rolling upgrade (1 broker/BK at a time)
• [ ] Disaster recovery: Test failover to secondary cluster quarterly

Performance Tuning¶

# Broker settings (/etc/pulsar/broker.conf)
# Increase for high throughput
managedLedgerCacheSizeMB=2048  # More cache = fewer disk hits
managedLedgerNumWorkerThreads=32  # Parallelism
managedLedgerMaxOpenFencedLedgers=10000

# BookKeeper tuning
# Increase for higher throughput
numAddWorkerThreads=8
numReadWorkerThreads=8
numJournalCallbackThreads=8
journalFormatVersionToWrite=5  # Latest format

Interview Questions¶

Key Takeaways¶

Next: Kafka internals & Pulsar comparison