I like Prometheus. It's simple, reliable, and nothing beats it inside a single cluster. The trouble starts when you run 20+ clusters and someone in a meeting asks "what was p99 latency across all environments last quarter?" That's when you find out where a single Prometheus runs out of road.

Where a single Prometheus hits the wall

Four walls I've hit repeatedly:

• Retention. Prometheus is a local on-disk TSDB. 15 days is comfortable, a quarter isn't, a year is fantasy. Local storage just doesn't scale.
• No global view. Each cluster has its own Prometheus, its own Grafana datasource. Querying across all of them means manually switching or gluing panels together. Unusable.
• HA without deduplication. For HA you run two identical Prometheis. They're two independent TSDBs with slightly offset scrape times — at query time you see doubled lines, and they don't deduplicate themselves.
• Cardinality and memory. Prometheus keeps the whole head in RAM. One bad high-cardinality label (user_id, a pod in a crash loop, a path with UUIDs) and you've got an OOM kill.

Two integration paths with Thanos

Thanos solves these, but the first real decision is how you get metrics into it.

Path A — Sidecar. A Thanos Sidecar runs next to each Prometheus. It does two things: it uploads completed TSDB blocks to object storage (S3, Azure Blob) and exposes the Prometheus's live data over a gRPC Store API. Upside: minimal change to Prometheus, no push model. Downside: blocks are only uploaded once they're closed (typically every 2 hours), so for fresh data it still relies on the local TSDB. This is my default for clusters I own and where I control Prometheus.

Path B — remote-write to Thanos Receive. Prometheus ships samples in real time via remote_write to the Receive component, which distributes them with consistent hashing, stores them in local TSDBs and gradually pushes them to object storage. Upside: centralization — the edge cluster needs no local storage and no object-storage credentials. This makes sense for thin/edge clusters or multi-tenant. The downside that burned me: back-pressure. When Receive can't keep up or goes down, the remote_write queue in Prometheus grows, ingestion latency rises, and in the worst case you lose samples. Receive becomes a critical path you have to scale and monitor like a production database.

In practice I mix both: Sidecar where I own the cluster, Receive for the edge.

Thanos components and what each does

• Sidecar — block upload + Store API over a live Prometheus.
• Receive — accepts remote-write, consistent hashing, local TSDBs, uploads to object storage.
• Store Gateway — serves historical blocks from object storage over the Store API. Keeps a local index header and an in-memory index cache (--index-cache-size).
• Compactor — compaction, retention and downsampling. Runs off the query path.
• Query (Querier) — implements the Prometheus HTTP API, fans out to all Store API endpoints (Sidecar, Store Gateway, Ruler), deduplicates HA pairs, evaluates PromQL.
• Ruler — evaluates recording and alerting rules against data from Query and writes results back as new blocks.

Long-term storage, downsampling and compaction

Everything long-term lives in object storage. The Compactor does two things that decide both cost and speed. Compaction merges small blocks into larger ones and shrinks indexes. Downsampling produces lower resolutions: raw data (= scrape interval) is aggregated to a 5-minute resolution for blocks older than 40 hours, and the 5m data older than 10 days gets a 1-hour resolution. The Querier then auto-selects the resolution based on the query range — a year-long graph doesn't read billions of raw samples, just 1h aggregates. Watch the trap: if retention at a given resolution is shorter than the minimum age for the next downsampling pass, data gets deleted before it can be aggregated.

Deduplicating HA pairs

This part is elegant. You give each Prometheus in an HA pair identical external labels except for one replica label (e.g. replica=a / replica=b). The Querier gets --query.replica-label=replica and merges both series into one at query time. No doubled lines. Important: external labels must be consistent, otherwise deduplication won't work.

Cardinality is the real cost lever

Object storage is cheap, but you pay for series count everywhere — in Prometheus memory, in Store Gateway indexes, at query time. The biggest impact isn't picking components, it's relabeling. I aggressively drop high-cardinality labels (metric_relabel_configs with action: labeldrop, or drop for whole metrics) at scrape time. A single metric with an id per request can generate more series than the rest of the cluster combined. Cardinality has to be handled at the source, not cured with downstream horsepower.

Production gotchas

• The Compactor MUST be a singleton per stream of blocks in a single bucket. Two instances = overlap issues you resolve by hand. If you need to scale, shard by labels, not by replicas.
• Store Gateway and memory. The index cache and the number of open blocks can OOM it. Watch it, give it a sane --index-cache-size, and consider an external Memcached/Redis cache.
• Receive back-pressure. Monitor the prometheus_remote_storage_* queue metrics. When the queue grows, Receive can't keep up — scale it before you start dropping data.

The global view in Grafana is ultimately trivial: one datasource pointing at the Querier, and you see all 20+ clusters at once, deduplicated, with a year of history. But getting there isn't about adding components — it's about discipline with cardinality and respect for the three gotchas above.