How to Eliminate the 5 Hidden Cost Traps in Your Kafka-to-Iceberg Pipeline
Every enterprise running Apache Kafka alongside a data lakehouse is haemorrhaging money in places they never thought to look. The OSO engineers have spent years helping organisations untangle their streaming data architectures, and the same pattern emerges time and again: teams are paying three, four, even five times more than they should to move data from Kafka into Apache Iceberg tables.
The culprit isn’t Kafka itself. It’s the connective tissue between your streaming layer and your lakehouse – the connectors, the replication, the duplicated data, and the over-provisioned compute that enterprises accept as the cost of doing business. But it doesn’t have to be this way.
In this article, we’ll break down the five cost traps that are silently draining your cloud budget, explain why the traditional Kafka-to-Iceberg architecture is fundamentally flawed from a cost perspective, and show how the OSO engineers built K2I an open-source, purpose-built streaming ingestion engine—to eliminate every single one of them.
The Architecture That’s Costing You a Fortune
Before we get into the specific cost traps, it helps to understand why the traditional Kafka-to-lakehouse architecture is so expensive in the first place.
In a typical enterprise deployment, Kafka acts as your real-time streaming platform and Iceberg serves as your open table format for analytics. Connecting the two usually involves Kafka Connect with a sink connector, Apache Flink, Spark Structured Streaming, or some custom ETL pipeline. The data flows from Kafka, through a connector layer, and lands in your lakehouse as Parquet files registered in Iceberg.
This looks clean on a whiteboard. In production, it looks like this:
TRADITIONAL KAFKA-TO-ICEBERG ARCHITECTURE
==========================================
┌──────────────────────────────────────────────────────────────────┐
│ AVAILABILITY ZONE A │
│ │
│ ┌──────────┐ ┌──────────┐ ┌──────────┐ │
│ │ App (A) │────$───>│ Broker 1 │───$────>│ Broker 2 │ │
│ └──────────┘ inter │ (Leader) │ repli- │(Follower)│ │
│ -zone └──────────┘ cation └──────────┘ │
│ │
├──────────────────────────────────────────────────────────────────┤
│ AVAILABILITY ZONE B │
│ │
│ ┌──────────┐ ┌──────────┐ ┌──────────┐ │
│ │ App (B) │────$───>│ Broker 3 │───$────>│ Broker 4 │ │
│ └──────────┘ inter │(Follower)│ repli- │ (Leader) │ │
│ -zone └──────────┘ cation └──────────┘ │
│ │
├──────────────────────────────────────────────────────────────────┤
│ CONNECTOR LAYER │
│ │
│ ┌──────────────────────────────────┐ │
│ │ Kafka Connect / Flink / Spark │──── $ ────> DUPLICATE │
│ │ (dedicated compute + storage) │ STORAGE │
│ └──────────────────────────────────┘ │
│ │ reads from brokers (more $) │
│ │ writes to object storage │
│ ▼ │
│ ┌──────────────────────────────────┐ │
│ │ Iceberg Tables (S3 / GCS) │ ◄── same data, again │
│ └──────────────────────────────────┘ │
│ │
│ $ = cost hot spot │
└──────────────────────────────────────────────────────────────────┘Every dollar sign in that diagram represents a cost hot spot. Let’s quantify each one.
Cost Trap #1: Inter-Zone Data Transfer
This is, without question, the single biggest hidden cost in enterprise Kafka deployments. When your applications produce messages to Kafka, they don’t always connect to a broker in the same availability zone. A producer in AZ-A might need to write to a partition whose leader sits in AZ-B. Every one of those cross-zone hops incurs an egress charge from your cloud provider.
But it gets worse. Kafka’s replication protocol requires that every message written to a leader partition is replicated to follower replicas, which typically sit in different availability zones for high availability. With a replication factor of three (the standard for production), every single message crosses zone boundaries at least twice during replication alone.
When you’re dealing with petabytes of data—and the OSO engineers routinely work with clients processing billions of messages per day—these inter-zone transfer costs can easily become the single largest line item on your cloud bill. At current cloud pricing, a stream processing 1 GiB/s can burn through millions in annual egress charges before you’ve even queried the data.
Then your consumers read the data. If a consumer sits in a different zone from the partition leader, that’s another cross-zone charge. And your Kafka Connect sink connector? That’s yet another consumer, pulling data across zones to write it somewhere else entirely.
The OSO engineers have seen organisations where inter-zone Kafka traffic accounts for over 40% of their total cloud spend. Most of them had no idea until we ran the numbers.
Cost Trap #2: Replicated Storage Costs
Kafka’s storage model relies on disk-based persistence with multiple replicas for durability. With a replication factor of three, you’re storing three full copies of every message on separate broker disks. If those brokers use high-performance SSDs—which they typically must to handle production throughput demands—your storage costs multiply quickly.
Consider a topic retaining seven days of data at high throughput. You’re not paying for seven days of storage. You’re paying for seven days multiplied by your replication factor, multiplied by the cost of SSD-tier storage across multiple availability zones. The maths is brutal.
This is a direct consequence of Kafka’s tightly coupled storage and compute architecture. Because every broker must maintain its own local copy of the data it serves, you cannot scale storage independently of compute. Need more retention? You need bigger disks on every replica. Need more throughput? You need more brokers, each with their own full set of replicated data.
Cost Trap #3: Over-Provisioned Compute
Kafka’s architecture requires you to provision for peak load. If your workload is spiky—and the OSO engineers find that most enterprise workloads are—you’re maintaining excess compute capacity that sits idle during normal operations.
This is especially pronounced in industries with predictable traffic patterns. Consider an IoT platform processing sensor telemetry from millions of devices. When those devices come online simultaneously—during a firmware deployment, for instance—the spike can be orders of magnitude above baseline. You must provision your Kafka cluster to handle that peak, which means paying for compute resources you only need a fraction of the time.
Because Kafka’s storage and compute are tightly coupled, you can’t simply scale down the brokers during quiet periods. Removing a broker means rebalancing its partitions and data to other brokers, which itself is an expensive and risky operation that takes time proportional to the data volume. Newly added brokers have no data to serve and cannot receive traffic until partitions have been assigned and data has been replicated to them.
The result is substantial waste. You’re paying for compute resources around the clock to handle spikes that occur for minutes or hours.
Cost Trap #4: Connector Overhead
Moving data from Kafka to your Iceberg lakehouse requires an intermediary—typically Kafka Connect with a sink connector, Apache Flink, or Spark Structured Streaming. Each of these options consumes additional compute resources, adds latency, and introduces operational complexity.
The connector infrastructure needs its own cluster of workers, its own monitoring, its own alerting, and its own maintenance schedule. Connector upgrades, schema compatibility issues, offset management, and failure recovery all require dedicated engineering time. The OSO engineers have worked with teams that spend more time managing their connector infrastructure than the Kafka cluster itself.
But the real cost isn’t just the compute. It’s the architectural complexity. Every connector is another potential failure point, another system to monitor, and another piece of the puzzle that can silently lose or duplicate data if misconfigured. When a connector fails at 3am, someone has to understand not just Kafka and not just Iceberg, but the specific connector’s offset tracking, retry behaviour, and commit semantics to diagnose the issue.
CONNECTOR OVERHEAD BREAKDOWN
=============================
┌──────────────────────────────────────────────┐
│ CONNECTOR COSTS │
├──────────────────────────────────────────────┤
│ │
│ Compute: Dedicated worker nodes │
│ Network: Reads from Kafka (cross-zone) │
│ Storage: Internal state + staging │
│ Latency: Seconds to minutes of delay │
│ Ops: Monitoring, upgrades, debugging │
│ Risk: Silent data loss or duplication │
│ │
│ All of this just to move bytes from │
│ one format to another. │
│ │
└──────────────────────────────────────────────┘Cost Trap #5: Duplicate Data
This is the trap that ties all the others together. In a traditional architecture, you maintain separate copies of the same data: one in Kafka for streaming consumption and one in your Iceberg lakehouse for analytics. This data duplication doesn’t just double your storage costs—it creates consistency challenges that require even more tooling and engineering time to manage.
Your Kafka cluster retains data for real-time consumers. Your connector reads that data and writes it again to object storage in Parquet format. Now you have two copies: the Kafka log segments and the Iceberg table files. They contain the same information, stored in different formats, managed by different systems, with different retention policies and different access patterns.
Keeping these copies synchronised is harder than it sounds. Schema changes in Kafka must propagate correctly to Iceberg. Partitioning strategies may differ. Time windows for data availability create gaps where the streaming and analytical views disagree. The OSO engineers have seen organisations where discrepancies between Kafka and their lakehouse went undetected for weeks, silently corrupting downstream analytics.
The K2I Approach: Eliminating Cost Traps by Design
When the OSO engineers set out to build K2I (Kafka to Iceberg), the goal was architectural: eliminate the cost traps at their root rather than optimising around them. K2I is a purpose-built, open-source streaming ingestion engine that reads directly from Kafka and writes natively to Apache Iceberg tables—no connectors, no intermediate storage, no duplicate data.
Here’s what the architecture looks like with K2I:
K2I ARCHITECTURE: DIRECT KAFKA-TO-ICEBERG
===========================================
┌──────────────────────────────────────────────────────────────────┐
│ │
│ KAFKA CLUSTER ICEBERG LAKEHOUSE │
│ │
│ ┌──────────────┐ ┌──────────────────┐ │
│ │ topic-1 │ │ iceberg table-1 │ │
│ ├──────────────┤ ┌───────┐ ├──────────────────┤ │
│ │ topic-2 │────>│ K2I │───>│ iceberg table-2 │ │
│ ├──────────────┤ └───────┘ ├──────────────────┤ │
│ │ topic-3 │ single Rust │ iceberg table-3 │ │
│ └──────────────┘ binary, no └──────────────────┘ │
│ JVM, no │
│ events cluster Parquet on S3/GCS/Azure │
│ queryable via any engine │
│ │
└──────────────────────────────────────────────────────────────────┘
WHAT'S ELIMINATED:
✗ No connector cluster (Cost Trap #4 gone)
✗ No duplicate data (Cost Trap #5 gone)
✗ No over-provisioned workers (Cost Trap #3 reduced)
✗ Minimal network overhead (Cost Trap #1 reduced)
✗ One copy on object storage (Cost Trap #2 reduced)How K2I Eliminates Connector Overhead
K2I replaces your entire connector layer with a single Rust binary. There’s no Kafka Connect cluster to manage, no Flink job to tune, no Spark Streaming application to monitor. K2I consumes directly from Kafka, buffers messages in memory using Apache Arrow’s columnar format, and flushes optimised Parquet files directly to your object storage when size, time, or count thresholds are met.
The write path is straightforward:
K2I INTERNAL PIPELINE
======================
1. CONSUME 2. BUFFER 3. FLUSH 4. COMMIT
┌───────────┐ ┌───────────────┐ ┌──────────────┐ ┌──────────────┐
│ │ │ │ │ │ │ │
│ Kafka │───>│ Hot Buffer │───>│ Parquet │───>│ Iceberg │
│ Consumer │ │ (Arrow) │ │ Encoder │ │ Catalog │
│ │ │ │ │ │ │ │
│ - batches │ │ - in-memory │ │ - compress │ │ - atomic │
│ - backpr. │ │ - O(1) index │ │ - upload S3 │ │ - CAS commit │
│ - retry │ │ - TTL evict │ │ - checksum │ │ - snapshot │
└───────────┘ └───────────────┘ └──────────────┘ └──────────────┘
│ │ │ │
▼ ▼ ▼ ▼
┌─────────────────────────────────────────────────────────────────────┐
│ TRANSACTION LOG │
│ Append-only | CRC32 checksums | Crash recovery | Exactly-once│
└─────────────────────────────────────────────────────────────────────┘Because K2I is a single process—not a distributed framework—there’s no coordination overhead, no network serialisation between processing stages, and no complex cluster management. The OSO engineers designed it to run as one instance per topic or partition set, giving you deterministic behaviour and predictable resource consumption.
How K2I Eliminates Duplicate Data
With K2I, data flows from Kafka directly into Iceberg tables. There is no intermediate staging area, no separate streaming store, and no second copy maintained for analytics. Once K2I flushes a batch of records as a Parquet file and commits the corresponding Iceberg snapshot, that data is immediately available to any analytical engine—Trino, Spark, Databricks, Snowflake, or anything else that reads Iceberg.
This is the stream-table duality concept made practical. The same data that was just consumed from a Kafka topic is now queryable as a structured Iceberg table. One copy. One format. One source of truth for both your streaming consumers and your analytical workloads.
How K2I Solves the Small File Problem
One of the most insidious challenges with streaming data into Iceberg is the small file problem. Naive streaming approaches create thousands of tiny Parquet files per hour, which degrades query performance, explodes metadata overhead, and spirals storage costs.
K2I solves this with intelligent buffering. The in-memory hot buffer accumulates records using Apache Arrow’s columnar format, flushing only when configurable thresholds are reached. The default configuration targets 512 MB Parquet files—large enough for excellent query performance, small enough to maintain reasonable data freshness.
But K2I goes further. It includes automated maintenance that runs in the background: compaction merges small files into larger ones, snapshot expiration controls metadata growth, and orphan cleanup removes debris from failed operations. No cron jobs. No manual intervention. The OSO engineers built this because we were tired of seeing teams dedicate entire sprint cycles to Iceberg table maintenance.
How K2I Guarantees Exactly-Once Delivery
The single biggest risk in any Kafka-to-Iceberg pipeline is data loss or duplication during the handoff. K2I eliminates this risk with a write-ahead transaction log that coordinates every step of the pipeline.
The sequence is precise. First, messages are consumed from Kafka and buffered in memory. When a flush is triggered, K2I writes a FlushStart entry to its transaction log. Then it encodes the Arrow buffer to Parquet, uploads the file to object storage, and commits the new data file to the Iceberg catalog using compare-and-swap (CAS) semantics. Only after the catalog commit succeeds does K2I acknowledge the Kafka offsets.
If K2I crashes at any point in this sequence, the transaction log tells it exactly where to resume on restart. Files that were uploaded but not committed get cleaned up. Offsets that weren’t acknowledged get re-consumed. The result is exactly-once semantics without the complexity of distributed transactions.
EXACTLY-ONCE GUARANTEE
=======================
Crash at any point → K2I recovers automatically
Step 1: Consume from Kafka → crash? Re-consume (offsets not committed)
Step 2: Buffer in Arrow → crash? Re-consume (offsets not committed)
Step 3: Encode to Parquet → crash? Re-encode from buffer
Step 4: Upload to object storage → crash? Orphan cleanup removes partial file
Step 5: Commit to Iceberg catalog → crash? CAS retry or skip (idempotent)
Step 6: Commit Kafka offsets → crash? Re-process is idempotent
Result: Zero data loss. Zero duplication.Practical Takeaways: Getting Started with K2I
If you’re running a traditional Kafka-to-Iceberg pipeline and recognise these cost traps in your own architecture, here’s how to start eliminating them.
Installation
K2I ships as a single binary. Install it on macOS or Linux with one command:
curl -fsSL https://k2i.io/install | shOr pull the Docker image:
docker pull ghcr.io/osodevops/k2i:latestConfiguration Example
Here’s a minimal configuration that connects K2I to a Kafka cluster and writes to an Iceberg table on S3 using a REST catalog:
# k2i-config.toml
[kafka]
bootstrap_servers = "kafka-broker-1:9092,kafka-broker-2:9092"
topic = "events"
group_id = "k2i-events-ingestion"
batch_size = 1000
[buffer]
max_size_mb = 512
flush_interval_seconds = 30
flush_batch_size = 10000
[iceberg]
catalog_type = "rest"
catalog_uri = "http://iceberg-catalog:8181"
warehouse = "s3://my-data-lake/warehouse"
database = "analytics"
table = "events"
compression = "zstd"
target_file_size_mb = 512
[iceberg.storage]
type = "s3"
bucket = "my-data-lake"
region = "eu-west-1"
[maintenance]
compaction_enabled = true
expiration_enabled = true
orphan_cleanup_enabled = trueRun the ingestion:
k2i ingest --config k2i-config.tomlThat’s it. One binary, one config file, one command. K2I handles consumption, buffering, encoding, uploading, catalog commits, offset management, compaction, snapshot expiration, and orphan cleanup—all automatically.
What to Measure
Once K2I is running, track these metrics to quantify your savings:
K2I exports Prometheus-compatible metrics out of the box. Monitor k2i_messages_total for throughput, k2i_flush_duration_seconds for latency, and k2i_buffer_size_bytes for memory utilisation. Compare your cloud bill before and after decommissioning your connector infrastructure.
The OSO engineers typically see organisations eliminate their connector compute costs entirely, reduce their storage footprint by removing duplicate data copies, and dramatically cut inter-zone transfer charges by simplifying their network topology.
Conclusion
The five cost traps we’ve described—inter-zone data transfer, replicated storage, over-provisioned compute, connector overhead, and duplicate data—are not inevitable consequences of running Kafka alongside a lakehouse. They’re architectural choices that can be eliminated with the right approach.
K2I represents a fundamentally different way of thinking about the Kafka-to-Iceberg pipeline. Instead of treating streaming and analytics as separate concerns connected by brittle middleware, K2I unifies them into a single, direct path: consume from Kafka, buffer intelligently, write natively to Iceberg. No connectors. No duplicate data. No operational overhead.
The tool is open source under the Apache 2.0 licence, built in Rust for performance and memory safety, and designed to be deployed in minutes rather than days. Whether you’re running a handful of topics or processing billions of messages daily, the architecture scales with your needs while keeping your cloud bill under control.
The OSO engineers built K2I because we believe the streaming-to-lakehouse pipeline should be boring infrastructure—reliable, efficient, and invisible. If your current pipeline is anything but boring, it might be time to rethink the architecture.
This post first appeared on Read More