Josh Lee is an open source advocate at Altinity, which runs and supports open-source ClickHouse. In this 37-minute Altinity webinar from April 2026 he lays out where he thinks observability storage is heading, then runs a small Docker demo that combines OpenTelemetry, Apache Parquet, Apache Iceberg and ClickHouse.
The core claim is his recurring line: “Observability begs to be open.” OpenTelemetry already opened up instrumentation and pipelines. The storage layer is the piece still locked up, and a real-time open lakehouse on object storage is how he proposes to open it.
Three kinds of openness
Josh starts with a criticism of something he loves. OpenTelemetry is often sold as vendor neutrality that lets you switch vendors at the flip of a switch. That isn’t how a migration really goes, but vendor neutrality is still valuable. His test for openness is one question: can you take it with you?
Open instrumentation. The OpenTelemetry specification, APIs and SDKs let you instrument code in a vendor-neutral way, so your code artifacts are portable. Vendors no longer maintain a tracing library for every framework, and library authors can build telemetry in from the start without picking a favourite vendor.
Open pipelines. Three or four years ago most vendors had you ship OTel data to their own agent, and they owned the pipeline all the way to their SaaS backend. In the last two years the OpenTelemetry Collector has matured and been run at very large scale (he names eBay). That gives you interoperability: you can fork the stream and layer in tools. One use is a bake-off: send the same telemetry to two vendors for a short time, then keep one.
Open storage. However the pipeline is built, the telemetry still ends up in some database with its own format. Open storage would mean you can get it back out. He calls it “GDPR for your telemetry”.
Moving day isn’t the main point, though. Most organisations only care about the last 24 hours of telemetry, sometimes the last two weeks, with exceptions such as seven years of logs kept for legal reasons. The bigger gain is a multi-access pattern: store the data once and let several tools read it, without paying for a replica per tool.
| Kind of openness | What you get | What it lets you do |
|---|---|---|
| Instrumentation | Portable code and artifacts | Change vendors without re-instrumenting every service |
| Pipelines | Interoperable building blocks | Fork, filter and route telemetry; run a bake-off |
| Storage | One copy, many readers, full ownership | Add analysis tools, keep executive reports stable while the SRE tool changes, train your own ML and AI models on your own data |
The visualisation layer matters less than it did
The obvious objection is that vendors ship polished visualisation and analysis. Josh’s answer is that OpenTelemetry + a database + an MCP server that can talk to that database gets you about 80% of what a traditional observability platform gives you. For small organisations or narrow use cases that may be enough. Agents also don’t mind running one-off commands or SSHing to a host to pull logs.
His “hot take” follows from that: the visualisation layer is quickly becoming the least interesting part of observability. Agents can query the data directly, and anyone with access to the data can now build a visualisation layer tailored to their own needs.
One data store for every signal
The next objection is operational: a separate database per signal means several systems to secure and scale, each with its own query language. His answer, from an earlier talk, is a unified telemetry data store:
- Many sources feed one pipeline: the OpenTelemetry Collector, plus Kafka when the volume is very high.
- The pipeline feeds one storage layer for logs, metrics, traces and now profiles.
You could get there by wrapping APIs around several stores. He prefers one store, and argues ClickHouse now handles all of those signals well enough to be it.
Scale: the real-time open lakehouse
Telemetry volume is the remaining problem. Some Altinity customers ingest over two petabytes a day of observability data, and he’s been told that’s small elsewhere. Block storage such as EBS gets very expensive at terabyte and petabyte scale. The answer is built up one word at a time:
- A data lake is object storage like AWS S3 behind an API. You can throw anything at it with no concern for structure or metadata. That’s handy, but making sense of it afterwards is slow, and it can turn into a data swamp.
- A lakehouse adds schema and structure, so the files look like a database table you can query with SQL.
- Open means the formats and metadata are open specifications that many tools can read and write.
- Real-time is the hard part, and it’s where Altinity’s Project Antalya comes in (see Getting to real time below).
The stack
The proof of concept sits entirely on S3-compatible object storage, which he puts at about 12 times cheaper than NVMe or EBS.
| Piece | Role | What it is |
|---|---|---|
| OpenTelemetry | Source and pipeline | Vendor-neutral instrumentation, the OTLP protocol and the Collector |
| Apache Parquet | Data format | An open columnar file format; he calls it JSON for columnar data. Values for one column sit together, which suits aggregations over millions or billions of rows |
| Apache Iceberg | Catalog and table format | An open metadata specification over Parquet files, so they can be queried like a table. Adds ACID transactions through snapshots |
| Apache Arrow | Wire format | Columnar data in transit: ADBC as a faster replacement for ODBC/JDBC, and the OpenTelemetry Arrow project |
| Project Antalya | Query engine | Altinity’s ClickHouse builds with data-lake features, including hybrid tables |
| S3 | Storage | Any S3-compatible object store |
Two details explain the rest of the design.
Iceberg commits are optimistic. A writer first writes Parquet files to object storage. Once the writes have finished without errors, it commits to the Iceberg catalog, which creates a new snapshot: a point-in-time view that now includes the new files. Readers only see data after the commit. This is safe, and also a little slow.
Arrow keeps data columnar on the wire. Data doesn’t have to be marshalled into rows (JSON or Protobuf) and back. The protocol is stateful: client and server share some state, which costs a little but allows dictionary encoding, so much less data is sent.
The demo architecture
The demo is in Altinity/demo-otel-parquet-antalya. It handles logs only, and Josh is clear that it’s a proof of concept, not a production design.
How the pieces fit:
- Sources send OTLP to the collector on
localhost:4318. The demo usestelemetrygencontainers that emit fake logs for three services. It doesn’t have to be OTLP: anything the collector has a receiver for will do. - otel-collector provides the OTLP endpoint and, more importantly, batches. Writing a Parquet file per log record would create a flood of tiny files that hurts both the object store and the query engine. The repo’s collector config uses the
batchprocessor withsend_batch_size: 10000andtimeout: 5s, which he says is still smaller than you’d want in production. - otlp2parquet, by Clay Smith at ServiceNow, acts like an OTel exporter: it takes OTLP and writes Parquet to S3.
- RustFS stands in for S3 in Docker. Any S3-compatible store works, including AWS S3 itself.
- log-sync is a small bash script. Every 60 seconds it checks the bucket for new Parquet files and registers them with the Iceberg REST catalog, which is the commit that makes them visible.
- ice-rest-catalog is Altinity’s Iceberg REST catalog. It holds the table metadata.
- ClickHouse (an Altinity Antalya build) mounts the catalog as a database named
ice. It reads metadata from the catalog and fetches the column data from S3. - Grafana queries ClickHouse with SQL.
The table schema closely matches what the Collector’s ClickHouse exporter produces (snake_case columns such as timestamp, service_name, severity_text, body, trace_id, log_attributes).
What he showed running:
- New rows show up at most about every 60 seconds, because that’s how often the sync job commits. His host dashboard shows disk-read spikes on the same 60-second rhythm.
- The stack ran on a Framework 12 laptop with an i3 CPU, and he briefly pushed it to 10,000 logs per second there.
- Memory sat at about 9 GB with the generators turned up, and he says it drops to about 4 GB with them tuned down.
- Queries are plain ClickHouse SQL. The dashboard panel buckets logs by time and lists the latest 100.
Try it yourself
You need Docker with Docker Compose and at least 4 GB of RAM. The steps below follow the repo’s README.
1. Start the stack
git clone https://github.com/Altinity/demo-otel-parquet-antalya
cd demo-otel-parquet-antalya
docker compose up -d
How to check it worked: docker compose ps shows rustfs, ice-rest-catalog, clickhouse, otlp2parquet, otel-collector, log-sync and grafana as running, and the RustFS console opens at http://localhost:9001 (the README gives the demo login rustfsuser / rustfspassword).
2. Send a test log
Run this two or three times:
curl -X POST http://localhost:4318/v1/logs \
-H "Content-Type: application/json" \
-d '{
"resourceLogs": [{
"resource": {
"attributes": [{"key": "service.name", "value": {"stringValue": "my-app"}}]
},
"scopeLogs": [{
"scope": {"name": "my-scope"},
"logRecords": [{
"timeUnixNano": "'$(date +%s)'000000000",
"severityText": "INFO",
"body": {"stringValue": "Hello from my-app!"}
}]
}]
}]
}'
How to check it worked: the request returns HTTP 200. After the batch flushes, Parquet files appear under bucket1/logs/ in the RustFS console.
3. Query the Iceberg table from ClickHouse
Wait a minute or so for the batch to flush and for log-sync to register the files, then:
docker compose exec clickhouse clickhouse-client --query \
"SELECT timestamp, service_name, severity_text, body FROM ice.\`otel.logs\` ORDER BY timestamp DESC LIMIT 10"
How to check it worked: you get rows back, including my-app | INFO | Hello from my-app! and logs from the generator services. If the result is empty, wait another 60 seconds and check docker compose logs log-sync.
A second query to try:
SELECT service_name, severity_text, count()
FROM ice.`otel.logs`
GROUP BY service_name, severity_text;
4. Open the dashboard
Go to http://localhost:3000. The repo ships a Grafana dashboard with basic log panels.
How to check it worked: the log panel lists recent records and refreshes with new data about once a minute.
5. Point your own telemetry at it
For an SDK:
OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318
OTEL_EXPORTER_OTLP_PROTOCOL=http/json
For another Collector:
exporters:
otlphttp:
endpoint: http://localhost:4318
tls:
insecure: true
service:
pipelines:
logs:
exporters: [otlphttp]
How to check it worked: SELECT DISTINCT service_name FROM ice.\otel.logs`` now includes your service.
6. Clean up
docker compose down -v
rm -rf ./data
How to check it worked: docker compose ps lists nothing, and ./data is gone.
Choosing an Iceberg catalog
Iceberg is a specification, so there are several catalog implementations. His picks:
| Catalog | His note |
|---|---|
| Apache Polaris | The canonical choice and the most mature implementation |
| Altinity Ice REST Catalog | Altinity’s own; the one in the demo, and a one-click option in Altinity.Cloud |
| Lakekeeper by Vakamo | Open source, with strong access controls, which the Iceberg spec itself doesn’t cover |
Scaling it to production
He says you could run this pattern at production scale today, and shows three ways to build the write path.
| Pattern | Write path | Who commits to Iceberg | Status |
|---|---|---|---|
| otlp2parquet + Spark | Collector (batching) → otlp2parquet → S3 | Apache Spark, periodically, in place of the bash script | Available now |
| otlp2pipeline | Collector (batching) → otlp2pipeline on AWS Lambda → S3 | otlp2pipeline itself, so no extra job | Available now |
| OpenTelemetry Arrow | Collector writes Parquet and commits directly | The Collector | On the OTel Arrow roadmap |
Notes on each:
- Neither of Clay Smith’s tools batches, so keep an OpenTelemetry Collector in front. Tune its
batchprocessor to get Parquet files of the size you want. - OpenTelemetry Arrow already works today for Collector-to-Collector traffic. It sends the most compact form of the data, which matters when you pay for cross-region transfer. Writing straight to Iceberg is on its roadmap. That would shrink the write path to the Collector you probably already run, with batches committed as soon as they fill.
The point he stresses: in every variant, the outside of the platform stays the same. OpenTelemetry goes in, and SQL through ClickHouse comes out. Services that emit telemetry and tools that read it don’t change while you swap what’s in between.
Getting to real time: hybrid tables
A 60-second commit cycle isn’t real time. For that, data has to land in ClickHouse and be queryable within about a second. Project Antalya’s hybrid tables are built for this.
A hybrid table uses ClickHouse MergeTree as the first storage layer, then exports data out to Iceberg and Parquet on object storage as it ages, in the style of a TTL move. Queries see one table.
He describes hybrid tables as alpha: open source, with some customers already using them in production. Altinity’s 2026 Project Antalya roadmap, tracked as a GitHub issue, lists the work to make them feature-complete.
Benefits and limits
Benefits
- Practically limitless scale. S3 keeps growing, and the Iceberg catalog means query engines read only the data they need.
- Open access. Any tool can reach the data, through ClickHouse or by attaching another engine such as Apache Spark to the same catalog. Several engines can read and write one catalog.
- All open source, and it runs anywhere from a small laptop to 192-core clusters.
- Good fit for MCP. The data is well structured and follows OpenTelemetry conventions that agents already know.
- SQL-based observability.
Limits
- It’s SQL. If your tools depend on PromQL, that’s a problem today. He adds that PromQL support inside ClickHouse is in beta and improving.
- Not quite real time yet. Hybrid tables are the fix.
- Compute overhead in the pipeline, mostly from the conversion steps in this proof of concept. He expects Arrow to remove much of it.
- Many moving parts to run and maintain.
Companies building on the open pieces
He closes with vendors that help without putting anything proprietary in the stack:
- Odigos: an eBPF-based no-code instrumentation agent that emits OpenTelemetry-compatible data.
- Coroot: eBPF no-code instrumentation plus an analysis backend on ClickHouse, with VictoriaMetrics for time series (which also sidesteps the SQL-versus-PromQL issue).
- Bindplane: pipeline management for fleets of Collectors, including OpAMP. Its acquisition by Dynatrace had just been announced.
- OllyGarden: tooling for telemetry quality and how well teams follow semantic conventions.
- Altinity.Cloud: managed ClickHouse, with the Iceberg catalog available as a switch.
Q&A notes
eBPF. It’s built into Linux. You write small programs in a restricted subset of C that are loaded from user space and run safely in kernel space, where they can observe what other processes ask the kernel to do. A probe can watch another process’s network connections, and an agent can start a trace span from what it sees.
ClickHouse for profiles. He thinks ClickHouse is the best suited of the tools he knows, but says the schema still needs work to use ClickHouse well. He’s hoping to explore that with customers as the OpenTelemetry profiles signal matures.
How it ties together
- Open instrumentation and open pipelines already exist thanks to OpenTelemetry. Open storage is the missing third, and its value is one copy of the data that many tools can read.
- Parquet holds the columns, Iceberg turns a pile of Parquet files into a table with snapshots, and S3 makes storage cheap and effectively unbounded.
- The Collector batches, a converter writes Parquet, and something commits to the catalog: a bash script in the demo, Spark or otlp2pipeline in production, and the Collector itself once OTel Arrow can write to Iceberg.
- ClickHouse reads metadata from the catalog and data from S3, and exposes it all as SQL to Grafana or an MCP server.
- Hybrid tables put MergeTree in front for the first seconds of data and move it to Iceberg afterwards, which is what makes the lakehouse real-time.
- The outside contract stays fixed: OTLP in, SQL out. Everything in between can be swapped as the tools mature.
Sources: the webinar on YouTube, the slides (PDF), the demo repo and otlp2parquet.