Tracing Pipeline (APM)

Use Settings → Pipeline (Beta) → Traces to control which traces Middleware ingests. The tracing pipeline lets you:

• Drop traces that match one or more conditions (e.g., a specific service).
• Sample traces from selected services by percentage (e.g., keep/drop a fraction to reduce volume).

Create a Pipeline#

• Go to Settings → Pipeline (Beta) → Traces.
  
• Click Create (or open an existing pipeline), give it a Name & Description (e.g., “Drop service”).
• The Conditioning step has two sub-sections:
  • Create Rules — define what to drop.
  • Sampling Rate — set per-service percentages.
• Click Save Configuration, then enable/disable the pipeline from the list view with the toggle.
  

You’ll see Last Edited and Created By details on the list page so you can audit changes.

Create Rules: Drop by Conditions#

The Rules builder generates a Boolean expression and shows it in PREVIEW.

Example:

1([serviceName] == "app-socket-server-v2")

This means: drop all traces where serviceName equals the given value.

How to add a rule#

• Attribute: choose the field to check (e.g., serviceName).
• Operator: choose how to compare.
• Value(s): type and press Enter to add as a chip (supports multiple values for IN/NOT IN).
• Use Add Rule to add more rows.
• Use the AND / OR switch (shown between rule rows) to combine conditions.
• Delete any row with the red trash icon.
• The copy icon next to "PREVIEW" lets you copy the generated condition for future reference.

Examples#

Drop a single service:

1([serviceName] == "checkout-api")

Drop multiple services

1([serviceName] IN ["worker", "cron-runner", "staging-frontend"])

Drop paths that match a pattern (case-insensitive)

1([http.route] ILIKE "/health%")    // drops /health, /healthz, /health/check

Drop by compound condition

1([serviceName] == "app-socket-server-v2" && [env] == "staging")

Sampling Rate — Drop by Percentage#

Use this section when you don’t want to block a service entirely but reduce its trace volume.

• Click Add New Service, select a Service, and set a Sampling Rate (%).
• A value of 100 % means drop all traces for that service (functionally equivalent to a full service drop).
• Values between 1–99 % drop that fraction of incoming traces for the listed service.
• Add multiple services with different percentages if needed.

The Sampling Rate acts at the service level. If you also define Rules, any trace that matches a drop rule is removed; in addition, listed services have the specified percentage of remaining traces dropped. In short, rules target “what” to drop; sampling controls “how much”.

Recommended Set-ups (common use cases)#

• Silence a noisy service temporarily
  • Rule: (none)
  • Sampling: app-socket-server-v2 → 80%
  • Effect: dramatically lowers volume while keeping a 20% slice for visibility.
• Remove non-actionable traffic
  • Rule: [http.route] ILIKE "/health%" OR [spanName] ILIKE "%metrics%"
  • Effect: drops health checks and metrics scrapes that clutter APM.
• Cut staging volume
  • Rule: [env] == "staging"
  • Optional Sampling: additional 50% on particular staging services to cut deeper.
• Decommissioned service
  • Rule: [serviceName] == "legacy-billing"
  • Or Sampling: set 100% for legacy-billing.

Notes#

• Preview is the truth: it reflects the exact Boolean the backend evaluates (e.g., && for AND, || for OR, parentheses for grouping).
• Value chips: for IN/NOT IN, add multiple values as separate chips.
• LIKE/ILIKE: use % as wildcard; avoid leading wildcards unless necessary to keep eval efficient.
• Order of operations: the builder adds parentheses so your AND/OR intent is preserved.
• Scope: This page controls Traces. Similar controls exist on other tabs (Logs, Metrics, RUM), but are independent.

Troubleshooting (quick, no-table)#

• My rule isn’t dropping anything
  • Confirm the exact attribute name (e.g., serviceName vs service.name) and check casing.
  • Check the operator (e.g., LIKE needs % wildcards).
  • Look at PREVIEW to ensure the rule logic (AND/OR) matches your intent.
• Too much got dropped
  • Lower the Sampling Rate (e.g., from 100% to 50%) or temporarily disable the pipeline from the list toggle.
  • Remove the broadest rule first (e.g., env‐wide) and re-add narrower conditions.
• Multiple values not working
  • Use IN/NOT IN and add each value as a separate chip.
• Performance considerations
  • Prefer exact matches (=/IN) over wide LIKE "%foo%".
  • Use ILIKE only if you truly need case-insensitive matches.