I was wrong, and I want to walk through exactly why, warts and all, because I think the honest version of this story is more useful than a polished feature list copied off a landing page.

Why Observability, and Why SigNoz

I'd been running a small Node.js + Python microservices project — an API gateway, an auth service, and a background worker — with nothing but console.log and vibes. It worked until it didn't. A request would go slow somewhere in the chain and I'd have no way to see where. That's the exact problem observability platforms exist to solve: correlating logs, metrics, and distributed traces so you can go from "something's wrong" to "here's the exact span that's slow" in one flow, instead of tailing five different log files and guessing.

SigNoz caught my attention because it's built OpenTelemetry-native from the ground up rather than bolting OTel support onto a legacy agent. That matters more than it sounds — it means you instrument once with open standards, and you're not locked into a vendor SDK you'll regret later. It's also positioned as an open-source alternative to Datadog and New Relic, with all three signal types — logs, metrics, and traces — living in one application instead of three separate tools stitched together with luck.

Setting It Up: Faster Than I Expected

I went the self-hosted route with Docker Compose, mostly because I wanted to see what running the whole stack myself actually felt like before trusting anything to the cloud version. The install itself was refreshingly boring — clone the repo, run the Docker Compose file, and within a few minutes I had the SigNoz UI up on localhost with the query service, the frontend, and the ClickHouse-backed data store all talking to each other.

That last part is worth pausing on. SigNoz uses ClickHouse, a columnar database built for exactly this kind of high-cardinality, high-volume telemetry workload, instead of relying on something like Elasticsearch for logs. In practice that translated to noticeably fast queries even once I'd thrown a decent amount of trace and log volume at it locally — no waiting around for a dashboard to "think."

Wiring It Up: The OpenTelemetry Part

This is where most observability tools either win or lose me, and it's where SigNoz earned some real trust. I pointed my services at the OpenTelemetry Collector, added the OTel SDK to my Node service, auto-instrumented the Python worker, and configured the collector to export everything toward SigNoz. Configuration is largely driven by environment variables with a predictable, readable convention, which made debugging my own mistakes painless instead of painful.

Within minutes of restarting my services, I watched my service map populate itself — auth service talking to the gateway, the gateway talking to the worker, request volume and error rate showing up per service without me writing a single custom metric. That "oh, it's actually working" moment is rare with infra tooling, and SigNoz gave it to me on the first try.

Exploring the Features

Once data was flowing, I spent a genuinely fun evening just clicking around:

• Service Map — a live topology view of how my services actually talk to each other, which was more accurate than my own mental model of my own system (slightly embarrassing, but useful).

• Distributed Tracing — flamegraphs and waterfall views that let me follow a single request across all three services and see exactly which span ate the latency, down to span events and filters.

• Logs Explorer — fast, filterable log search that didn't choke even as I dumped in noisy debug logs from every service.

• Exceptions tracking — automatically surfaced unhandled errors from my Node service without any extra setup.

I didn't stop at the surface-level tour, though. Since the challenge specifically rewards going deep on the platform, I deliberately pushed into the parts of SigNoz that most "quick look" reviews skip: the Query Builder, dashboards, alerts, and the SigNoz MCP server.

Query Builder: Where I Actually Learned My Own System's Behavior

The Query Builder is the part of SigNoz where traces, logs, and metrics stop being separate tabs and start being one queryable dataset. Instead of memorizing a proprietary query language, I could filter, group, and aggregate across all three signal types using a visual builder — or drop straight into PromQL or raw ClickHouse SQL when I needed something the UI didn't expose yet.

I used it to answer questions I'd never bothered asking before, like "what's my P99 latency per endpoint, grouped by status code, over the last 24 hours, filtered to just the auth service" — and got an answer in seconds instead of writing a one-off script. Being able to fall back to raw ClickHouse SQL when the visual builder hit its limits, without switching tools, is exactly the kind of power-user escape hatch that made me trust the platform more, not less.

Dashboards: Turning Ad-Hoc Queries into a Real Operating Picture

Once I had queries I cared about, building dashboards out of them was the natural next step. I imported one of SigNoz's curated templates (there are ready-made ones for things like Postgres, Redis, JVM, Kubernetes, and general APM) as a starting point, then customized panels with metric, trace, and log data side by side. Having latency, error rate, throughput, and Apdex on one screen — built from the same Query Builder I'd already learned — meant I wasn't reinventing a mental model every time I switched from "debugging one request" to "checking the health of the whole system."

Alerts: Closing the Loop from "I Noticed Something" to "I Get Paged Before It Matters"

Dashboards are great for humans actively looking; alerts are what catch things while you're not looking. I set up alert rules across metrics, logs, and traces — threshold breaches on error rate, latency spikes on my slowest endpoint, and absent-data conditions in case a service quietly stopped reporting entirely. Being able to define alert conditions on any signal type, not just metrics, meant I could catch failure modes (like a service going silent) that a purely metrics-based alerting tool would have missed entirely.

The SigNoz MCP Server: Bringing Observability Into My Coding Agent

This is the part that genuinely surprised me. SigNoz ships an official MCP (Model Context Protocol) server, which means my coding agent — I connected it through Claude Code — can query my real telemetry directly, in plain English, without me hand-writing a single query.

I pointed the MCP server at my self-hosted instance, and suddenly I could ask things like "why did the auth service's P95 latency spike around 3pm" and have the agent pull traces, logs, and metrics from that exact window, correlate them, and hand back a ranked list of likely causes — the same investigation I'd normally do by hand across three UI tabs. It could also create dashboards from a natural-language description, draft alert rules for conditions I described conversationally, and manage saved Explorer views, all through MCP tool calls instead of me clicking through menus.

What struck me most is that this isn't a bolt-on chatbot — it's built on the fact that SigNoz already stores logs, metrics, and traces under one consistent schema. That's exactly what makes an agent effective here: it doesn't have to normalize three different data models before it can reason about an incident. For a challenge literally called "Agents of SigNoz," using the MCP server felt like the most honest way to explore what "agent-native observability" actually means in practice, rather than just reading the phrase off a landing page.

The Feature I Liked the Most

Hands down, it's the trace-to-log-to-metric correlation. The first time I clicked on a slow span in a trace and it took me straight to the exact log line from that request, on that service, at that timestamp — no manual timestamp matching, no grepping — I actually said "nice" out loud to an empty room. That's the entire promise of "unified observability," and it's the one thing that made me stop context-switching between tools. Once you've debugged a latency spike this way, going back to separate logging and tracing tools feels like working with one hand tied behind your back.

What Could Be Improved

In the spirit of an honest review rather than a marketing page:

• The learning curve on advanced query building (raw ClickHouse SQL, complex alert conditions) is real. It's powerful, but it rewards existing observability experience — a newer developer will need to spend time in the docs before writing anything beyond basic filters.

• Self-hosting has real operational weight. Running your own ClickHouse cluster, ingestion pipeline, and dashboards is empowering for data ownership, but it's not zero-ops. I'd like to see the self-hosted onboarding docs get even more opinionated with sane defaults for smaller teams, so first-timers don't have to make infra decisions they're not ready to make yet.

• The newer AI-assisted anomaly detection and LLM observability layer (token-level tracing, per-model cost attribution) is genuinely promising, but it still feels earlier-stage compared to the rock-solid core tracing and logging experience. I'd love to see it mature at the same pace as the rest of the platform.

None of these are dealbreakers — they're the kind of feedback you give a tool you actually want to keep using.

Final Thoughts

Going in, I expected another dashboard to fight with. What I got instead was a genuinely smooth path from zero to a working, OpenTelemetry-native observability stack: real traces, metrics, and logs flowing through the collector; a Query Builder I used to actually understand my own system's behavior; dashboards that turned ad-hoc queries into a real operating picture; alerts that closed the loop so I didn't have to keep watching a screen; and an MCP server that let my coding agent reason over all of it in plain English. That's the full stack the challenge asks you to lean on, and going through each layer myself — not just skimming the landing page — is what actually changed how I think about debugging distributed systems.

For a project that's fully open source, with 24,000+ GitHub stars and an active community behind it, that combination of depth and openness is hard to beat.

If "Agents of SigNoz" is about developers who've actually lived inside the tool, not just read the docs — consider this my field report. I set it up, I broke it a little, I fixed it, I queried it, I dashboarded it, I alerted on it, and I even let an agent investigate it for me. That's the real test, and SigNoz passed it.