ThisCalculator

Everyday

Grafana Tempo Redaction Before Storage: What to Know

Updated · Standard formula, shown below

Grafana Tempo Redaction Before Storage: What to Know

Yes, Grafana Tempo can keep selected sensitive data out of stored traces, but the best place to redact it is usually before spans reach Tempo—for example, in application instrumentation or a telemetry collector. Tempo is a trace backend, not a universal content-scrubbing layer, so do not assume that enabling Tempo automatically detects or removes secrets and personal data.

What Tempo stores and why redaction matters

Tempo stores distributed traces: collections of spans describing work across services. A span can contain a name, timestamps, trace and span identifiers, status, resource attributes, span attributes, and events. The exact data depends on the instrumentation and configuration that produced it. Attribute values might include an HTTP route, a user identifier, a database statement, or—in a poorly designed system—a password or authorization header.

Tracing data can be valuable for diagnosing latency and failures, but it can also reveal information that should not be broadly accessible or retained. Examples include email addresses, session tokens, customer identifiers, query parameters, and request or response bodies. Whether a particular value is sensitive depends on the data and the organization’s policies.

Redaction means removing or transforming a value before it is written to the trace store. It is different from hiding a column in a dashboard or restricting who can query traces: those measures may limit visibility, but the original value can still remain stored. Deletion after ingestion is also not a substitute for preventing future sensitive data from being written.

Where to redact data

Choose the earliest reliable point in the telemetry path. A common flow is application instrumentation, an optional OpenTelemetry Collector or other processing layer, and then Tempo as the backend. The precise components vary by deployment.

  • At instrumentation: Do not create sensitive attributes in the first place. Capture an approved route template such as /customers/{id} instead of a raw URL containing an identifier, and avoid recording credentials or full request bodies.
  • In a collector: A configured processor can remove, modify, or restrict attributes before exporting telemetry. This is useful when several services share a centrally managed pipeline, provided every route to Tempo passes through that processor.
  • In application or gateway middleware: Filter data at the point where it is collected or enriched. This can be necessary for fields that a general-purpose processor cannot reliably recognize or transform.
  • At Tempo: Tempo’s role is principally to receive and store traces. Do not treat it as a general-purpose, automatic detector that finds arbitrary secrets in every span field.

Collector capabilities and configuration syntax depend on the collector distribution and version. Consult the documentation for the exact processor you deploy, and test its behavior with representative trace data. A processor that removes one attribute name does not necessarily catch the same value copied into an event, another attribute, or a span name.

How to design a reliable redaction policy

Start with a data inventory rather than a broad rule that might silently discard useful telemetry. Identify which services emit traces, which fields they produce, who can query them, and where the data is exported. Then define what may be collected, what must be removed, and what can be retained in a less identifying form.

  1. List risky fields. Review instrumentation and sample traces for headers, query strings, request bodies, database statements, user identifiers, and exception messages.
  2. Prevent collection where possible. Excluding a secret at its source is generally simpler and safer than relying on a later pattern match.
  3. Apply transformations consistently. Use allowlists for attributes when practical, or remove known sensitive keys. If replacing values with a placeholder, consider whether the remaining context is still identifying.
  4. Cover every export path. Check direct application exporters, sidecars, gateways, and secondary pipelines. A collector policy protects only the data that passes through it.
  5. Test before rollout. Send synthetic examples containing known test values and inspect the exported trace and the data visible in Tempo. Include nested events and error paths, not just ordinary spans.
  6. Monitor and review. Recheck the policy when services, instrumentation libraries, processors, or attribute conventions change.

For example, an HTTP instrumentation library might record a full URL such as /reset?email=alex@example.test&token=sample. A safer design is to record a route template and omit the query string, rather than hoping a downstream expression recognizes every possible token format. If request bodies are not needed for tracing, do not capture them.

What redaction can and cannot guarantee

Rules that match attribute names are usually more predictable than rules that search arbitrary text for secret-like patterns. Pattern-based detection can miss unfamiliar formats, encoded values, or data split across fields; it can also remove harmless text. Redaction should therefore be layered with careful instrumentation, access controls, retention settings, and appropriate handling of telemetry exports.

Also distinguish removal from masking. Replacing a value with a fixed marker prevents the original value from being stored only if the transformation happens before storage. Hashing is not automatically anonymous: predictable identifiers can sometimes be guessed, and a stable hash can still link activity over time. Use such transformations only when they meet the intended privacy and operational requirements.

If sensitive data has already reached Tempo, a new processing rule generally affects incoming telemetry, not historical traces already stored. The required response depends on the storage backend, retention and deletion capabilities, backups, and organizational procedures. Do not assume that changing instrumentation instantly removes old data.

Tempo redaction versus access controls

Redaction and access control solve different problems. Redaction reduces what is stored. Authentication, authorization, network controls, and careful query access limit who can reach stored traces. Both matter: a restricted trace backend can still contain excessive sensitive data, while a redacted backend can still expose operational details that should be controlled.

Likewise, hiding a sensitive attribute in a Grafana visualization does not prove that it was removed from the stored trace. A user with another query path may still retrieve it. Verify the actual exported and stored trace data rather than relying on dashboard presentation.

Frequently Asked Questions

Does Grafana Tempo automatically redact passwords or tokens?
No. Do not assume Tempo automatically detects arbitrary credentials in span names, attributes, events, or other trace content. Prevent collection or configure a tested processing step before export.

Can an OpenTelemetry Collector redact traces before Tempo receives them?
Yes, when the chosen collector distribution and configured processor support the required transformation and all relevant traces pass through it. Verify the processor’s documented behavior for your version and data structure.

Does hiding an attribute in Grafana remove it from Tempo?
No. Hiding a field in a dashboard changes what is displayed, not necessarily what was exported or stored. Check the trace itself and the ingestion path.

Can I redact traces that are already stored?
A new ingestion rule does not normally rewrite historical traces. Whether existing data can be deleted depends on the storage setup and its retention or deletion mechanisms; review those controls separately.

Should I redact at the application or collector?
Prefer not to emit sensitive values at all when possible. A collector can provide a shared enforcement layer, but only if it is on every path to Tempo and its rules cover the relevant fields.

Practical takeaway

Tempo can store traces that have already been sanitized, but it should not be treated as an automatic privacy filter. Keep secrets and unnecessary personal data out of instrumentation, add a tested collector or middleware policy where needed, confirm that every export path is covered, and separately manage access and previously stored traces. That approach makes the boundary clear: redaction happens before storage, while Tempo stores the trace data it receives.

Related

Method: the standard formula for this calculation, rounded as each result states. See how the calculators work or report an error.