How I'd Design a Kafka-Based Payment Event Pipeline from Scratch

Short summary

A practical architecture walkthrough for building Kafka-based payment event pipelines, drawn from production experience in Brazilian fintech. Key decisions include partitioning by payment ID for per-payment ordering, using acks=all with min.insync.replicas=2 for durability, separating topics by event lifecycle stage, enforcing idempotent consumers with outbox pattern, and maintaining dead letter topics. Schema evolution via Avro/Protobuf with schema registry and careful monitoring of consumer lag and DLT depth are also covered.

• •Partition by payment ID for per-payment event ordering; use acks=all and min.insync.replicas=2 for durability
• •Separate topics by event lifecycle stage (initiated, authorized, settled, failed) for independent scaling and retention
• •Enforce idempotent consumers with outbox pattern, dead letter topics, and schema registry for backward/forward compatibility

Generated with AI, which can make mistakes.