Kubernetes Observability Without a Central Log Archive: Metrics, Privacy and Trade-Offs

Nov 04, 2026 988 views

A Kubernetes platform can stay operationally visible without sending application logs to a central archive. This account describes a production setup with more than 15 microservices on K3s, where Prometheus metrics support routine monitoring and logs are enabled briefly when an incident calls for them. The design choice has a clear consequence: teams give up a searchable historical record in exchange for limiting what user information their observability systems collect and retain.

The cluster runs on virtual private servers in multiple locations, connected over a private WireGuard network. Ansible automates node setup, allowing a bare VPS to join the cluster within minutes. The infrastructure is conventional; the unusual part is an architectural rule established before the first service went live: observability must not become a channel for collecting user information.

That rule shifts the privacy question upstream. A common pattern gathers data first, then relies on retention limits, deletion procedures and access controls to manage it. Those controls can work, but they depend on ongoing operational discipline. The author’s experience with systems whose policies and processes degrade over time led to a different choice: keep sensitive data out of centralized logging from the start.

Prometheus metrics define the routine view

Instead of shipping application logs to a shared store, the nodes expose metrics such as request latency, workload, queue depth and error rates. Prometheus gathers those measurements, and an allowlist in each application scrape job controls which label names can enter metric storage.

The configuration drops labels outside the approved set:

metric_relabel_configs:
  - action: labelkeep
    regex: __name__|job|instance|service|route|method|code|le|quantile

The list retains the metric name, job, instance, service, route, HTTP method, response code, histogram bucket and quantile labels. Keeping __name__ matters because Prometheus represents the metric name as a label during this relabeling step. Remove it, and the samples lose the name needed to identify them.

This filtering happens after Prometheus scrapes an exporter. It limits what gets stored, but it doesn’t stop sensitive values from reaching Prometheus during scraping. Instrumentation therefore still needs to avoid putting personal data into metrics in the first place.

There’s another limit: an allowlist of label names does not validate the values attached to them. Code review supplies a guardrail. For the route label, developers must use a route template such as /users/:id, rather than a request’s raw path, which could contain an individual user’s identifier.

Alerts draw on the same metrics. They focus on user-visible symptoms, including elevated route latency and queues where work has stopped progressing. In this setup, alerting does not depend on searching log entries.

Incident response starts with narrowing the signal

The predictable concern is what happens when a service fails and there’s no central archive to search. The approach relies on metrics to narrow the investigation: high-resolution time series can point responders toward a particular service and behavior within minutes. That makes the quality of instrumentation consequential. If a team hasn’t measured a relevant signal, there may be no historical log trail to fill the gap.

When metrics don’t answer the question—a situation the author says is uncommon—the response is to enable verbose output temporarily. An operator changes the deployment’s LOG_LEVEL, reproduces the problem and follows live output with kubectl logs -f. Once the fault is understood, the deployment setting is reverted.

That procedure is a short-lived diagnostic path, not a claim that logs never exist. Container standard output remains available locally through Kubernetes and the container runtime. Its lifetime depends on node-level log rotation and cleanup, and reverting a deployment does not guarantee that existing local files disappear immediately. The nodes also use volatile storage for journald.

The distinction is practical: there is no central service collecting and retaining application logs, but there can still be temporary output on individual nodes. Teams adopting this model need to understand both the Kubernetes log behavior and the host’s cleanup settings, and they need to ensure that temporary verbose logging is turned off when the investigation ends.

Independent inspection checks the actual traffic

An outside engineer reviewed the platform by tracing its operation rather than accepting architecture diagrams as proof. The review followed a node from a bare VPS into the cluster, examined traffic across WireGuard, and checked what traffic left the nodes. The engineer had warned in advance that limited incident visibility could create blind spots.

The observed telemetry matched the Prometheus configuration: metric series carried the allowed labels, and the review found no separate centralized log-forwarding pipeline. That check matters because a design rule only has value if the running system follows it. An overlooked shipper or an exporter exposing unexpected data could undermine the intended boundary.

The model trades historical detail for a smaller data footprint

Metrics work best when teams decide in advance what they’ll need to understand system behavior. Without an archive, an incident may raise a question the existing measurements can’t answer. The design therefore asks engineers to think carefully about service-level signals, alert conditions and label values before production issues arise.

Temporary diagnostics also require operational discipline. Verbose output left enabled can accumulate the very information the architecture is intended to avoid retaining. A single declarative setting makes the mode easy to switch on, but the incident process must include switching it back off and accounting for any local files that remain.

The author draws a boundary around the approach: organizations that need an immutable audit trail should build an audit pipeline. Metrics and temporary logs aren’t a substitute for that requirement. The model suits teams whose priority is routine service health and capacity visibility while reducing centralized collection of user data.

Cloud-native tooling makes the arrangement workable: Prometheus provides detailed time series, Kubernetes exposes container output when needed, and WireGuard connects distributed nodes privately. The setup shows that teams can design routine observability around operational measurements instead of retaining every application event. The key decision comes before deployment: define which signals are necessary, prevent sensitive values from entering them, and treat verbose logging as a controlled incident tool.

Source: Daniil Romashov · cloudnativenow.com

Comments

Sign in to comment.
No comments yet. Be the first to comment.

Related Articles

Running Production With Zero Stored Logs