A quieter week after KubeCon. Microsoft’s final report on the 12 November outage is the standout read. AWS shipped CloudWatch features ahead of re:Invent, and vendors kept treating AI agents as both a workload to observe and a new kind of observability user.
Deep dives
- Is your OpenTelemetry pipeline becoming a performance bottleneck? — CNCF blog. Based on a Cloud Native Days Austria talk, the post load-tests a Collector with telemetrygen and looks at five settings that drive its CPU and memory cost: batching, resource attributes, receiver polling, trace sampling and where
memory_limitersits. A useful checklist before you scale a Collector fleet. - Observability as the control plane for multi-region failover — InfoQ. The author argues that telemetry should trigger and then confirm automated regional failover instead of just paging someone. Covers composite Prometheus alerts, Route 53 weighted health checks, DNS TTL traps and a verification loop, with an S3 incident as the worked example.
- How to explain error budget recovery in a rolling SLO window — OneUptime. A clear explanation of why a rolling error budget “heals” as old failures leave the window, even if nothing new ships. It also gives a PromQL ratio built on
increase()over request counters, and warns against averaging daily percentages. Handy for explaining SLO dashboards to stakeholders. - How to find at-risk Pub/Sub messages across Google Cloud projects — Dynatrace. A scenario-driven tutorial in which a throttled Cloud Run service pushes messages into a dead-letter subscription. The idea carries over to any tool: alert on how much of the retention period the oldest unacked message has used, not on backlog depth, and look for dead-letter topics that have no subscription attached.
Releases
- CloudWatch performance monitors for VPC network paths — AWS. Agentless, near real-time latency, network impact score and traffic metrics for the AWS network paths your VPC workloads use, across accounts and Regions through AWS Organizations. It complements the agent-based flow monitors.
- CloudWatch Omni now ingests Azure logs, metrics and traces — AWS. Connect an Entra ID tenant and CloudWatch can pull Azure resource logs, Azure Monitor metrics, VNet flow logs and Application Insights traces. It is a notable step for AWS’s multi-cloud observability push.
- Datadog brings production experimentation to AI agent observability — InfoQ. Online Experiments splits live traffic between agent versions and compares evaluator scores alongside latency, tokens, cost and errors. The example result is worth reading: a newer model cut estimated cost by about 44% at similar quality, but made more invalid tool calls.
Incidents
- Microsoft 365 outage of 12–13 November: the final report — 365Fanatic. New since last week: Microsoft’s final post-incident report is out, and this plain-language summary links to it. Four things combined: a latent lock-contention defect in a gateway update (which hit only newer, higher-core servers), about 14% of capacity out for routine deployment, a US East routing fault that failed Teams traffic over onto Exchange’s gateway, and US peak load. The fix was to revert the connection-handling change.
Industry news
- Honeycomb surpasses growth milestones as customers manage swarms of AI agents — Honeycomb. Mostly a growth announcement, but the usage numbers stand out: queries made through Honeycomb’s MCP server grew more than 200x this year, to over 4.5 million a month. That is strong evidence that agents are becoming major consumers of observability data.
- Dynatrace brings AI observability into Gemini Enterprise agent workflows — Dynatrace. Dynatrace insights can now be queried in natural language inside Gemini Enterprise. Arize, which Dynatrace recently acquired, handles evaluation of AI traces on Google Cloud.
- Updates to our blog policy — OpenTelemetry blog. The OTel blog will now accept posts only from community members in good standing and will strongly favour project and organisational announcements. The project cites lower search traffic, the rise of AI tools and the time maintainers spend reviewing posts. Expect fewer vendor how-tos there.
— END OF ISSUE —More issues →