Can you build observability ingestion on S3 alone — no Kafka, no disks, no coordination layer?

Short summary

A self-hosted observability system can replace expensive Kafka+Flink pipelines with S3-backed infrastructure using Apache Iceberg for storage. By using S3 conditional requests for coordination instead of a separate layer, this architecture eliminates operational burden while reducing costs from $800 to $100 per month. The single-binary design trades increased API latency for unlimited scale and fault tolerance.

• •Replaces Kafka+Flink with S3+Iceberg architecture, cutting costs from $800 to $100/month
• •Uses S3 conditional requests (If-Match) for distributed coordination without a separate coordination layer
• •Trades API latency for unlimited scale, fault tolerance, and zero operational maintenance

Generated with AI, which can make mistakes.