From OTel to Rotel: 4x Throughput Increase in PB-Scale Tracing

Efficiency matters at scale: even small reductions in resource consumption can lead to significant cost savings and efficiency gains.

OTel + ClickHouse benchmark: we built a data pipeline to evaluate the performance of streaming trace data into ClickHouse.

Rotel achieves a 4x throughput improvement: we demonstrate how we boosted from 137K trace spans per core per second on the OTel Collector to 462K on Rotel, detailing several key performance optimizations.

Tools and resources: we provide the list of tools used in our benchmark at the end.

Introduction

Operating an observability platform at petabyte scale requires continuous focus on resource efficiency. Even small improvements in per-core performance or memory usage can dramatically reduce infrastructure costs.

This article is based on a talk we gave at a ClickHouse meetup in Denver. In it, we shared our work on Rotel, a high-performance OpenTelemetry data plane designed for large-scale systems.

Thanks to its high compression and cost efficiency, ClickHouse is increasingly used for large-scale OpenTelemetry workloads. In these systems, the OTel Collector is often the most expensive part of the pipeline.

Recently, ClickHouse published an article on the importance of efficiency at scale (https://clickhouse.com/blog/scaling-observability-beyond-100pb-wide-events-replacing-otel), sharing their experience running OTel on their internal LogHouse platform.

One detail caught our attention:

"OTel Collector: using over 800 CPU cores to transport 2 million logs per second."

That translates to roughly 2,500 logs per core per second. For typical log lines, an 8-core server would handle only about 10 MB/s—far below modern hardware capabilities. Meanwhile, ClickHouse can process over 12,500 OTel events per core per second, more than 5x the ingestion side. The bottleneck is clearly on the collection side, not storage. Although we cannot replicate ClickHouse's internal LogHouse platform, we thought it meaningful to benchmark modern OTel data pipelines. So we wanted to explore a core question: how do different OTel data planes perform when sending traces to ClickHouse?

Benchmark Framework

Before presenting the results, we first describe the methodology used to evaluate both the OpenTelemetry Collector and Rotel. You can also skip directly to the results section via this link.

Trace Pipeline

Our benchmark focuses on writing trace spans to ClickHouse, as trace data grows extremely fast in large systems. We modeled a highly reliable streaming pipeline, choosing Kafka as the log streaming layer. The test simulates a scenario where many edge collectors send data to a Kafka stream, and a few core collectors batch-write it to ClickHouse. In this pipeline, Rotel and the OTel Collector use the same Kafka Protobuf encoding, so they are interchangeable.

Evaluation Method

We wanted to determine the maximum stable throughput a single collector can sustain under fixed hardware, focusing on collection efficiency rather than scalability. The key was to see how far a single node could be pushed before degradation, staying within our evaluation budget. We chose the gateway collector as the core component under test, because it writes directly to ClickHouse, which is most efficient with large batches. Deploying a few powerful gateway collectors is ideal for batching, so we concentrated on optimizing and measuring that component.

Saturation Detection

We used two key signals to judge when a collector reached its limit:

Memory surge: when backpressure from downstream builds up, the collector buffers data in memory, causing rapid growth. Kafka consumer lag: if the collector cannot keep up with the Kafka stream, the lag grows—the gap between the last read message time and current time widens.

Test Configuration

For each test, we pushed the pipeline to near-saturation for 15 minutes and recorded two key metrics:

Spans per second processed, as recorded by the load generator. Data throughput written to ClickHouse (MB/s), monitored via AWS CloudWatch. Bandwidth helps evenly compare capacity across environments because trace span sizes can vary significantly in practice. In our setup, edge collectors pack and optimize data before sending to Kafka, reducing the total number of messages while increasing individual message size.

We also recorded average CPU usage of Rotel and ClickHouse during tests.

OpenTelemetry ClickHouse Schema

The table schema used in our tests is compatible with both the OpenTelemetry Collector and Rotel. This is one of the officially recommended schemas for ClickHouse and the ClickStack observability platform, supporting OTel metrics, logs, and traces.

OTel's data model relies heavily on key/value attributes to describe critical infrastructure and application characteristics. Originally, these were stored as Map types in ClickHouse. Recently, the OTel Collector introduced support for the JSON column type, allowing the schema to use JSON columns. Although this increases CPU pressure on ClickHouse, it greatly enhances query expressiveness. Our tests enabled this new feature. You can view the complete ClickHouse trace schema we used here (https://gist.github.com/mheffner/dc332a61f3b9ba1d03fd7c7d5c1b7fbb).

We also used the ClickHouse Null table engine, a key optimization. The Null engine accepts writes but does not store data, thus eliminating disk I/O from the equation so we can focus on write throughput and schema correctness. After peak throughput tests, we later evaluate how ClickHouse handles real disk writes.

Load Generator

We initially tried the telemetrygen CLI tool but it struggled to reach the required data volumes. Instead, we used an in-house load generator originally built for testing OpenTelemetry and other telemetry pipelines. The project is available on GitHub in the otel-loadgen repository. It includes enhanced features like end-to-end data delivery validation, which we'll cover in future posts.

We constructed each trace with approximately 100 spans, containing rich attributes and metadata. Like all synthetic tests, this data does not fully mirror real production traffic.

Test Hardware

All benchmarks were run on AWS EC2 instances. Each pipeline component was deployed on a separate instance within the same availability zone to ensure consistency and accuracy.

To maximize disk throughput, we mounted Kafka and ClickHouse data volumes directly on the instance's local NVMe SSDs. All tests used Amazon Linux 2023 with components orchestrated via Docker Compose.

The goal was to evaluate the maximum throughput a single gateway collector host could handle. We chose the m8i.2xlarge instance with 8 vCPUs and 32 GB of RAM. As test scale increased, other pipeline nodes were scaled up, but the gateway collector remained fixed for fair comparison.

Testing the OpenTelemetry Collector

We began with the OpenTelemetry Collector, which served as both edge and gateway collector in the test.

Test Configuration

You can view the Docker Compose config here (https://github.com/streamfold/rotel-clickhouse-benchmark/blob/main/docker-compose-otelcoll.yml).

Results:

With a single Collector instance, we hit a performance wall at around 700K spans per second (~40 MB/s). Memory usage then climbed steadily even though CPU utilization remained below 50%.

The OTel Kafka receiver uses a single goroutine to process messages, likely capping throughput. We tried tuning various Kafka parameters, including message size limits, but saw no significant improvement. So we scaled horizontally by adding a second Collector instance on the same host (via Docker Compose scale: 2).

With two instances each consuming half the Kafka partitions, maximum stable throughput reached 1.1M spans per second (69 MB/s). Beyond this threshold, the send queue filled up, memory spiked, and once the queue was full the receiver continued reading but silently dropped data. So while Kafka lag did not increase, we were actually losing trace data!

During the test, the gateway collector's CPU peaked over 83%, becoming the primary bottleneck, while ClickHouse CPU stayed around 23%.

Testing Rotel

Test Configuration

You can view the Rotel Docker Compose config here (https://github.com/streamfold/rotel-clickhouse-benchmark/blob/main/docker-compose-rotel.yml).

Results:

Rotel also uses a single receive loop to pull data from Kafka, similar to the OTel Collector. This suggested a serial bottleneck. As expected, running two Rotel instances on the host and fully utilizing the CPU delivered a significant throughput boost.

With two Rotel instances in parallel, we achieved a maximum stable throughput of 1.45M spans per second (76 MB/s), about 1.3x over the OTel Collector. Pushing load further, we observed Kafka consumer lag slowly increasing, indicating that consumption was nearing its limit.

At this point, the gateway collector CPU hit 91.3%, becoming the new bottleneck; ClickHouse CPU rose to 60.4%.

Optimizing JSON Encoding in RowBinary Format

In Rotel, we built upon the official Rust ClickHouse library clickhouse-rs. It uses the RowBinary format—a row-oriented binary serialization protocol over HTTP. In contrast, the OTel Collector's Go driver and ClickHouse internals use a column-oriented native protocol.

When handling JSON columns, clickhouse-rs defaults to converting JSON to a string before sending. While ClickHouse does not store JSON columns as raw strings, this stringification is required for transport. However, at high concurrency this is costly: the client must serialize JSON, the server must re-parse it, and both must scan for quotes, backslashes, etc. to escape them. This is especially expensive with large string values.

With help from the ClickHouse Slack community, we discovered that JSON columns can be encoded natively in RowBinary. In this encoding, JSON is serialized as a sequence of key-value pairs: each key is a string, followed by a type tag and the value's raw binary representation. This bypasses the full JSON serialize/parse cycle, enabling more efficient structured data transfer.

For example, consider this simple JSON object:

{ "a": 42, "b": "dog", "c": 98}

The RowBinary encoding looks like this:

First, the number of key-value pairs is varint-encoded (e.g., 03). Then each pair is encoded: the key length as a varint (01), the key string, then the value. If the value type is statically known (e.g., a is an integer 42), a fixed encoding is used (2a 00 00 00 00 00 00 00). Otherwise, a dynamic type tag is used: for c, 0a denotes Int64, followed by 98 (62 00 00 00 00 00 00 00). Finally, b as string (15), length 03, and "dog".

Compared to traditional JSON transport, this significantly reduces serialization/parsing overhead on both client and server. While clickhouse-rs does not yet natively support this encoding, we plan to contribute it.

Re-test

After upgrading Rotel with the optimized encoding, we re-ran the benchmark. The overall throughput remained at 1.45M spans/sec, but ClickHouse server CPU dropped by about 10%, and gateway collector CPU also slightly decreased. This improvement was consistent across runs, confirming reduced server-side parsing overhead.

Admittedly, our synthetic load did not contain many long strings, and the number of attributes per span may differ from real production. So while client-side gains were minimal here, we believe in real-world scenarios with many attributes and large strings, this faster JSON serialization path will yield more noticeable benefits.

Doubling Throughput with an Unexpected Trick (Diagnosing and Fixing Allocator Lock Contention)

In this round, to fully utilize the gateway host's 8 vCPUs and push throughput to 1.45M spans/sec into ClickHouse, we had to run two Rotel instances. The Kafka receiver logic ran inside a single Tokio async task. The simplified flow:

This had two main issues:

1. The whole flow was serial, lacking parallelism.
2. Unmarshaling is CPU-intensive and can block Tokio's executor threads.

Tokio is Rust's async runtime using cooperative scheduling. Tasks must yield control at .await points. The community recommends limiting execution between .await points to 10-100 microseconds to avoid hogging the executor.

In Rotel's exporter, we already used a dedicated thread pool for CPU-heavy tasks like serialization and compression. For the Kafka receiver, rust-rdkafka performs decompression on background threads before recv(). But we initially kept unmarshaling inside the Tokio task. Upon analysis, we confirmed unmarshaling was heavily CPU-bound, so we decided to offload it asynchronously to the same exporter thread pool.

After the refactor, the main receive loop looked like:

loop { select! { message = recv() => { unmarshaling_futures.push(spawn_blocking(unmarshal(message))) }, unmarshaled_res = unmarshaling_futures.next() => { send_to_pipeline(unmarshaled_res) } }}

We re-ran the test with a single Rotel instance at the previous 1.45M spans/sec load. The system remained stable. Surprisingly, CPU usage dropped by about 40% compared to before!

Previously, two instances were needed because the single instance couldn't fully utilize the host due to the serial bottleneck in the receiver. Offloading unmarshaling removed that constraint. We expected a single instance could now match the dual-instance throughput.

Seeing such a large CPU drop, we increased the load. Eventually, we pushed throughput to 3.6M spans/sec—a 2x increase from the original 1.45M!

At 3.6M spans/sec, CPU again hit ~93%, saturating the system.

Profiling Rotel's Kafka Receiver with Flamegraph

We ran both old and new versions of the receiver and captured flamegraphs. At first glance, there was no obvious difference. Can you spot the key? In both versions, we see most time spent preparing and exporting trace data to ClickHouse, plus significant unmarshaling in prost::message::Message::decode. Such workloads create many short-lived objects, so a lot of time goes to memory allocation and deallocation.

Old version:

New version:

Assessing Receiver Changes with Linux Perf

Running perf stat revealed huge differences under the hood.

perf stat -c cycles,instructions,cache-misses,cache-references,context-switches,cpu-migrations

Old version:

295612663445 cycles 264853636815 instructions # 0.90 insn per cycle 615670230 cache-misses # 32.963 % of all cache refs 1867733351 cache-references 1224819 context-switches 1230 cpu-migrations 50.296446757 seconds time elapsed

New version:

150590256805 cycles 287007890213 instructions # 1.91 insn per cycle 598469068 cache-misses # 51.429 % of all cache refs 1163669589 cache-references 37675 context-switches 43 cpu-migrations 43.716966122 seconds time elapsed

The new version averaged 1.91 instructions per cycle and only 862 context switches per second—not stellar ILP, but a clear improvement. The old version achieved only 0.90 IPC with a staggering 24,350 context switches per second—the new version reduced that by ~32.5x! The old version was barely parallel, with threads constantly sleeping and waking, causing huge scheduling overhead.

CPU migrations also improved: the new version saw only 1 migration per second, indicating good cache affinity, versus 24.5/sec in the old version. The scheduler struggled to keep threads pinned.

The new version's better parallelism let us push throughput even higher—all from simply splitting tasks into separate threads.

We initially suspected the old bottleneck was due to blocking Tokio executor threads, leading to heavy polling, spinning, or work-stealing. However, deeper perf report analysis pointed to a more specific issue.

Glibc Multi-threaded Memory Allocator Lock Contention

From the data, we can now explain why CPU dropped and throughput increased after the refactor. The old version was wasting CPU cycles on fruitless overhead—specifically, spinning on locks while freeing memory.

To understand this, we need a brief look at glibc's default allocator. Memory is divided into multiple arenas, each protected by a mutex for thread-safe allocation and deallocation. To reduce contention, different threads attempt to use separate arenas, and the number of arenas grows with the thread pool.

The catch: if memory is allocated on thread A but freed on thread B, B must lock the arena associated with A, potentially causing other threads to wait and creating lock contention.

In the old version, we allocated memory for trace data during unmarshaling on Tokio's I/O executor threads—a limited pool of just 8 threads, one per core. Later, that memory was freed during serialization in the ClickHouse exporter, which ran on a large blocking thread pool (tens to hundreds of threads). This cross-thread allocation/free pattern caused frequent arena locking and contention.

In the new version, we moved unmarshaling's memory allocation to the same blocking thread pool as deallocation, eliminating cross-thread frees. As the pipeline scaled, the thread pool expanded, and so did the number of arenas, drastically reducing lock contention.

Verifying Arena Contention: A Jemalloc Comparison

A reviewer asked: "Would switching to jemalloc cause the same issues?" Jemalloc is designed for multi-threaded environments with reduced lock contention. We had tried jemalloc earlier without notable gains, but with higher load on the exporter and receiver changes, memory pressure increased, prompting a re-evaluation.

Switching the old version to jemalloc dropped CPU from 93% to 40%, matching the effect of moving unmarshaling to the shared thread pool—further confirming that lock contention was the core bottleneck.

Although jemalloc eased CPU pressure, under full load Kafka consumer lag increased. Additionally, jemalloc is no longer actively maintained, so we decided against making it the default. However, in the future we may introduce feature flags to allow users to choose custom allocators like jemalloc or mimalloc.

Extra Boost: Enabling Fast LZ4 Compression

Rotel uses ClickHouse-recommended LZ4 compression to reduce network data. We use the lz4_flex crate—the same dependency as clickhouse-rs, but as a direct dependency. When adding compression support, we overlooked the feature flag in Cargo.toml.

Lz4-flex offers both safe and unsafe implementations; the unsafe version is higher performance (see Rust docs on unsafe here: https://doc.rust-lang.org/book/ch20-01-unsafe-rust.html). By default, you must explicitly enable the unsafe feature.

clickhouse-rs enables unsafe mode by default, but we had not. After turning it on, throughput increased from 3.6M to 3.7M spans/sec, with compressed network input reaching 209 MB/s.

Gateway collector CPU also dropped slightly.

Full End-to-End Performance Evaluation

After multiple rounds of optimization and initial testing with the Null table engine, we boosted single-instance throughput from 1.1M spans/sec to 3.7M spans/sec—that's 462.5K spans per core per second, a >4x improvement over the initial OTel Collector benchmark.

We then focused on the final link: writing data to ClickHouse with actual disk persistence. Scaling ClickHouse writes typically requires optimizing table schemas for both write throughput and query efficiency. Here we stuck with the default OTel schema, so we primarily relied on high-write-capacity instances.

To handle the massive write load, we deployed ClickHouse on a beefier AWS instance with 4 local NVMe SSDs in RAID0 to avoid disk bandwidth bottlenecks.

During the test, we disabled Rotel's async inserts and increased batch size to --batch-max-size=102400 for efficiency. With --clickhouse-exporter-async-inserts=false, we sustained 3.7M spans/sec at the gateway collector.

ClickHouse CPU hovered around 50%, with compressed write traffic at 210 MB/s.

Visual inspection

Visually, we successfully queried over 3 billion trace records in ClickHouse, confirming end-to-end viability.

Conclusion

Efficiency Matters at Extreme Scale

ClickHouse's LogHouse platform handling PB-scale observability shows that efficiency is not optional but a necessity. They boosted pipeline throughput 20x while using only 10% of the previous resources. Continuing on the old path would have made operational costs unsustainable. Giants like Netflix and OpenAI have reached the same conclusion: at this scale, efficiency directly impacts business viability.

Against this backdrop, our goal was to improve OpenTelemetry data ingestion efficiency, resulting in Rotel.

Nearly 4x Throughput Increase

We've turned Rotel into a high-throughput OpenTelemetry streaming collector. On the same hardware, Rotel processes almost 4x more than the OTel Collector, leading to significant resource savings at scale. Rotel natively supports OpenTelemetry traces, metrics, and logs. This article focused on traces; we plan to extend benchmarks to logs and metrics.

We're also interested in what features users value most when processing massive data. If you have experience or want to share your scaling practices, join our Discord (https://rotel.dev/discord) or contribute on GitHub (https://github.com/streamfold/rotel).

Future Directions

Here are areas we plan to explore further:

In-depth Kafka Reliable Transport

We briefly touched on Rotel's message reliability. Rotel supports end-to-end message acknowledgement, ensuring at-least-once semantics when reading from Kafka, avoiding potential data loss from relying on Kafka's default auto-commit. We made several pipeline modifications and rigorously tested to prevent both loss and duplication. We plan a dedicated article on its design and validation.

Evaluate ClickHouse Native Protocol

Rotel currently uses clickhouse-rs over HTTP-based RowBinary. In contrast, the OTel Collector uses a Go ClickHouse driver employing the native protocol, which is what ClickHouse uses for inter-node communication. Benchmarks show the native protocol is ~20% faster than RowBinary. ClickHouse also now supports Apache Arrow Flight for efficient columnar transfer. We plan to evaluate replacing RowBinary with these protocols for further gains.

Further Analyze Blocking Tasks in Tokio

Blocking operations like unmarshaling severely impact Tokio's runtime. Having seen this firsthand, we want to explore similar bottlenecks in other paths. For example, Rotel's OTLP receiver currently performs heavy Protobuf deserialization directly in async tasks via the tonic crate. Initial perf observations suggest huge optimization potential by splitting this out.

Optimize Memory Allocation Paths

Although Rust has no GC, high-frequency allocation/deallocation can still be a bottleneck. Rotel creates many short-lived objects. Using a freelist to reuse common buffers could significantly reduce overhead. This is tricky to implement and could increase memory footprint if done incorrectly; it may require deep changes in the tonic crate.

Appendix

Considered but Not Included

When designing the benchmark, we hoped to include more OTel-compatible data plane tools. However, we found they were incompatible with our pipeline. We chose distributed tracing because it's a key OTel adoption driver and grows fast at scale. Logs and metrics are traditional monitoring areas; many tools still lack robust trace support. Though not covered here, we plan future benchmarks for logs and metrics.

Vector

Vector is a lightweight tool for building high-performance telemetry pipelines with broad source/sink support. It is now developed primarily by DataDog for their observability pipelines product.

However, Vector's OpenTelemetry support is early-stage and cannot interface with many target systems. Its internal data model initially lacked trace structures, so OTel trace support is limited. Because Kafka and ClickHouse output plugins don't yet support trace data, we couldn't include it.

For example, we tried:

Sending trace data from OpenTelemetry source to Kafka sink (https://github.com/vectordotdev/vector/discussions/21018); Storing trace spans in ClickHouse (https://github.com/vectordotdev/vector/issues/17307#issuecomment-1641075239).

Fluent Bit

Fluent Bit is a performance-focused, C-based Fluentd alternative. It offers OpenTelemetry input/output for logs, metrics, and traces, and supports Kafka. In theory, it could form a reliable streaming pipeline.

However, in testing, we found that simultaneously using OTel input and outputting trace and metric data via Kafka or HTTP is not yet fully supported in the current version, ruling it out for this evaluation.

Simplifying OTel and ClickHouse Migrations

ClickHouse docs recommend manually creating tables before deployment, rather than relying on exporter auto-creation. But migration scripts are usually bundled inside the OTel exporter and lack a standalone deployment option, making the process non-intuitive.

To address this, we factored the schema management logic out of Rotel into a standalone CLI tool called clickhouse-ddl. It can easily deploy a schema compatible with both Rotel and the OTel Collector.

We packaged it as a Docker image; a single command creates the trace table. Example for a trace span table:

docker run streamfold/rotel-clickhouse-ddl create
--endpoint https://abcd1234.us-east-1.aws.clickhouse.cloud:8443
--traces --enable-json

You can also use the Null table engine for benchmarking, as we did:

docker run streamfold/rotel-clickhouse-ddl create
--endpoint https://abcd1234.us-east-1.aws.clickhouse.cloud:8443
--traces --enable-json
--engine Null

References

Benchmark framework: https://github.com/streamfold/rotel-clickhouse-benchmark Rotel project: https://rotel.dev OTel load generator: https://github.com/streamfold/otel-loadgen

/END/

Call for Papers

We welcome community submissions on ClickHouse technical research, project practices, and innovations. Style should be informative and richly illustrated. Quality articles will be published on this channel, and outstanding ones may be recommended to the ClickHouse website. Please send your manuscript in WORD format to [email protected].