OpenTelemetry

Compatibility

Languages & SDKs (recommended baselines): Node.js ≥ 18, Python ≥ 3.8, Java (8+/11+), .NET ≥ 6, Go ≥ 1.20, PHP ≥ 8.1, Ruby ≥ 3.0, Rust ≥ 1.70. Auto‑instrumentation exists for most of these; use manual APIs for custom spans/metrics.

Instrument Your Application (quick start)

Configure via environment variables first. Add manual code only where needed.

1. Environment Variables

• Kubernetes: put the same values under env: in your Deployment.
• Docker Compose: use environment: in your service.
• Windows PowerShell: use $env:OTEL_EXPORTER_OTLP_ENDPOINT = 'https://<uid>.middleware.io:443' etc.

2. Node.js (auto + manual)

The given command installs the Node SDK, auto‑instrumentations, and OTLP exporters. Prefer gRPC unless your network requires HTTP.

1npm i @opentelemetry/sdk-node @opentelemetry/auto-instrumentations-node @opentelemetry/exporter-trace-otlp-grpc @opentelemetry/exporter-metrics-otlp-grpc

Create otel.js and run your app with node -r ./otel.js server.js:

1// otel.js
2import { NodeSDK } from '@opentelemetry/sdk-node';
3import { getNodeAutoInstrumentations } from '@opentelemetry/auto-instrumentations-node';
4import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-grpc';
5import { OTLPMetricExporter } from '@opentelemetry/exporter-metrics-otlp-grpc';
6import { PeriodicExportingMetricReader } from '@opentelemetry/sdk-metrics';
7
8const traceExporter = new OTLPTraceExporter();
9const metricExporter = new OTLPMetricExporter();
10
11const sdk = new NodeSDK({
12  traceExporter,
13  metricReader: new PeriodicExportingMetricReader({ exporter: metricExporter }),
14  instrumentations: [getNodeAutoInstrumentations()],
15});
16
17sdk.start();

Add a custom span:

1import { trace } from '@opentelemetry/api';
2const tracer = trace.getTracer('app');
3
4app.get('/checkout', async (req, res) => {
5  await tracer.startActiveSpan('charge-card', async (span) => {
6    try { /* ... */ } finally { span.end(); }
7  });
8  res.send('ok');
9});

3. Python

The following instructions will help you install the OTel distro and OTLP exporter, then bootstrap auto‑instrumentation.

1pip install opentelemetry-distro opentelemetry-exporter-otlp
2opentelemetry-bootstrap -a install

Run with auto‑instrumentation: The auto-instrumentation allows your process to collect common libraries automatically.

1export OTEL_PYTHON_LOGGING_AUTO_INSTRUMENTATION_ENABLED=true
2export OTEL_PYTHON_LOG_CORRELATION=true
3
4opentelemetry-instrument \
5  --traces_exporter otlp \
6  --metrics_exporter otlp \
7  --logs_exporter otlp \
8  --service_name my-service \
9  python app.py

4. Java (SDK + autoconf)

• Add dependencies (Gradle):

  1dependencies {
  2  implementation("io.opentelemetry:opentelemetry-api:1.43.0")
  3  runtimeOnly("io.opentelemetry:opentelemetry-sdk:1.43.0")
  4  runtimeOnly("io.opentelemetry:opentelemetry-exporter-otlp")
  5  runtimeOnly("io.opentelemetry.instrumentation:opentelemetry-javaagent:2.9.0")
  6}

• Run with the Java agent:

  1java \
  2-javaagent:/path/opentelemetry-javaagent.jar \
  3-Dotel.service.name=my-service \
  4-Dotel.exporter.otlp.endpoint=https://<uid>.middleware.io:443 \
  5-Dotel.exporter.otlp.headers=authorization=<MW_API_KEY> \
  6-jar app.jar

5. .NET

It adds the OTLP exporter; tracing/metrics are configured via DI in your app.

• First, use the following for installation:

  1dotnet add package OpenTelemetry.Exporter.OpenTelemetryProtocol

• Additionally, you can add the packages if required to your workflow:

  1dotnet add package OpenTelemetry.Extensions.Hosting
  2dotnet add package OpenTelemetry.Instrumentation.AspNetCore
  3dotnet add package OpenTelemetry.Instrumentation.Http
  4dotnet add package OpenTelemetry.Exporter.OpenTelemetryProtocol

• Programmatic setup: It registers ASP.NET Core and runtime instrumentation and exports over OTLP.

  1builder.Services.AddOpenTelemetry()
  2  .ConfigureResource(r => r.AddService("my-service"))
  3  .WithTracing(t => t
  4      .AddAspNetCoreInstrumentation()
  5      .AddHttpClientInstrumentation()
  6      .AddOtlpExporter())
  7  .WithMetrics(m => m
  8      .AddAspNetCoreInstrumentation()
  9      .AddRuntimeInstrumentation()
  10      .AddOtlpExporter());

• Wire logs: By wiring logs, you get traces + metrics + logs over OTLP with consistent service.name.

  1using OpenTelemetry;
  2using OpenTelemetry.Logs;
  3using OpenTelemetry.Resources;
  4
  5builder.Logging.AddOpenTelemetry(o =>
  6{
  7    o.IncludeScopes = true;
  8    o.ParseStateValues = true;
  9    o.SetResourceBuilder(ResourceBuilder.CreateDefault().AddService("my-service"));
  10    o.AddOtlpExporter(); // picks up OTEL_* env vars
  11});

Other languages follow the same pattern: set OTEL_* env vars, add auto‑instrumentation where available, fall back to manual spans/metrics for critical paths.

Sampling

Head‑based sampling (SDK side): Choose what percentage of new root traces to record; child spans follow the parent decision.

1# Sample ~20% of new root traces; children follow the parent decision
2export OTEL_TRACES_SAMPLER=traceidratio
3export OTEL_TRACES_SAMPLER_ARG=0.2
4
5# Defaults often map to: parentbased_always_on
6# Other common values: always_on, always_off, parentbased_always_off

Tail‑based sampling (Collector side): Samples after the full trace is seen, which is great for “errors or slow requests only.” Use this in a central Collector.

1receivers:
2  otlp:
3    protocols:
4      http:
5      grpc:
6processors:
7  tail_sampling:
8    decision_wait: 10s
9    policies:
10      - name: errors
11        type: status_code
12        status_code:
13          status_codes: [ ERROR ]
14      - name: 25percent
15        type: probabilistic
16        probabilistic:
17          sampling_percentage: 25
18  batch: {}
19exporters:
20  otlphttp:
21    endpoint: https://<MW_UID>.middleware.io:443/v1/traces
22    headers:
23      authorization: <MW_API_KEY>
24service:
25  pipelines:
26    traces:
27      receivers: [otlp]
28      processors: [tail_sampling, batch]
29      exporters: [otlphttp]

Correlation & Context Propagation

• Goal: one click from a user session (RUM) to the exact backend trace and logs.
  1. Pick a propagator and make it consistent everywhere.
  • If you use Middleware RUM, prefer b3 (or b3multi). Set in backend via OTEL_PROPAGATORS=b3.
  • For pure OpenTelemetry defaults, use tracecontext,baggage.
  2. Enable log correlation so logs carry trace_id/span_id. Most SDKs auto‑inject; if not, add your logger’s OTel layout/MDC enricher.
  3. Forward incoming headers through your API gateway/reverse proxy.
• RUM snippet (example – React/Browser): Include this in your SPA’s HTML to capture page loads and XHR/fetch; set the propagation format to match your backend. The Middleware RUM SDK can emit B3 or W3C headers and make sure your backend expects the same.

1<script src="https://cdnjs.middleware.io/browser/libs/<ver>/middleware-rum.min.js" crossorigin="anonymous"></script>
2<script>
3  window.Middleware.track({
4    projectName: "web",
5    serviceName: "web",
6    accountKey: "<MW_API_KEY>",
7    target: "https://<uid>.middleware.io",
8    tracePropagationFormat: "b3", // or "w3c"
9    tracePropagationTargets: [/api\.mydomain\.com/i],
10  });
11</script>

• Backend: Match the propagator with your frontend to keep a single distributed trace.

  1export OTEL_PROPAGATORS=b3     # match the frontend
  2# or: export OTEL_PROPAGATORS=tracecontext,baggage

Expected outcome: clicking a RUM event shows the correlated distributed trace; from a backend span, you can pivot back to the user/session.

Integrations & Deployment Patterns

• Direct‑to‑Middleware (simple): applications export via OTLP/HTTP or OTLP/gRPC to https://<uid>.middleware.io:443.
• Through an OpenTelemetry Collector (recommended for fleets): receive on 0.0.0.0:4317 (gRPC) / :4318 (HTTP), process (tail sampling, attributes, batch, memory limiter), then export to Middleware’s OTLP endpoints (see §2.1). Supports multiple exporters if you mirror data.
• Edge/sidecar: ship from egress‑restricted networks via a local Collector, then forward over TLS 443.

Minimal Collector “gateway” example (all signals): run once per cluster/region to receive from many apps and export to Middleware.

1receivers:
2  otlp:
3    protocols: { http: {}, grpc: {} }
4processors:
5  memory_limiter: {}
6  batch: {}
7exporters:
8  otlphttp/traces:
9    endpoint: https://<MW_UID>.middleware.io:443/v1/traces
10    headers: { authorization: <MW_API_KEY> }
11  otlphttp/metrics:
12    endpoint: https://<MW_UID>.middleware.io:443/v1/metrics
13    headers: { authorization: <MW_API_KEY> }
14  otlphttp/logs:
15    endpoint: https://<MW_UID>.middleware.io:443/v1/logs
16    headers: { authorization: <MW_API_KEY> }
17service:
18  pipelines:
19    traces:  { receivers: [otlp], processors: [memory_limiter, batch], exporters: [otlphttp/traces] }
20    metrics: { receivers: [otlp], processors: [memory_limiter, batch], exporters: [otlphttp/metrics] }
21    logs:    { receivers: [otlp], processors: [memory_limiter, batch], exporters: [otlphttp/logs] }

Custom Metrics (per language)

1import { MeterProvider } from '@opentelemetry/sdk-metrics';
2import { OTLPMetricExporter } from '@opentelemetry/exporter-metrics-otlp-grpc';
3const exporter = new OTLPMetricExporter();
4const meter = new MeterProvider({
5  readers: [new PeriodicExportingMetricReader({ exporter })],
6}).getMeter('app');
7
8const checkoutCounter = meter.createCounter('checkout.count', {
9  description: 'Number of checkouts',
10});
11
12checkoutCounter.add(1, { currency: 'INR', channel: 'web' });

1from opentelemetry.sdk.metrics import MeterProvider
2from opentelemetry.exporter.otlp.proto.http.metric_exporter import OTLPMetricExporter
3from opentelemetry.sdk.metrics.export import PeriodicExportingMetricReader
4
5exporter = OTLPMetricExporter()
6meter = MeterProvider(readers=[PeriodicExportingMetricReader(exporter)]).get_meter("app")
7
8latency = meter.create_histogram("checkout.latency", unit="ms")
9latency.record(123.4, {"route": "/checkout", "status_code": 200})

1Meter meter = GlobalMeterProvider.get().meterBuilder("app").build();
2LongCounter items = meter.counterBuilder("checkout.items").build();
3items.add(3, Attributes.of(stringKey("channel"), "web"));

Metric hygiene: choose base units (seconds, bytes, metres), stable names, and low‑cardinality attributes. Prefer Histograms for latencies and sizes.

Troubleshooting Checklist

No data appears

• Verify DNS/TLS egress: curl -sv https://<uid>.middleware.io:443/health || true (HTTP 404 is fine; you’re checking TLS reachability).
• Confirm headers: OTEL_EXPORTER_OTLP_HEADERS includes authorization=<MW_API_KEY>.
• Mismatched transport: if you set /v1/traces, ensure you’re using an OTLP/HTTP exporter. For gRPC exporters, use the root endpoint without /v1/*.
• Set OTEL_LOG_LEVEL=debug and check application logs for exporter errors.
• For Collectors, ensure batch and memory_limiter are present and exporters point to the correct endpoints; look for exporter/otlphttp errors.

Broken Correlation

• Frontend uses b3, but backend expects tracecontext (or vice‑versa). Make them match.
• API gateway strips propagation headers; allowlist b3* or traceparent/tracestate/baggage.

High cost / high volume

• Reduce OTEL_TRACES_SAMPLER_ARG (e.g., 0.1). Add Collector tail sampling to keep errors/slow traces while cutting noise.

.NET specific

• Prefer setting endpoint in code if your runtime ignores the env var. If needed, pass configuration via IConfiguration and AddOtlpExporter().