Kubernetes Auto-instrumentation

The OpenTelemetry (OTel) Kubernetes Operator supports injecting and configuring auto-instrumentation libraries for .NET, Java, Node.js, Python, and Go services.

Middleware Application Monitoring is built on OpenTelemetry and OTel Kubernetes Operator can be used to auto-instrument your applications implemented in supported programming languages.

Prerequisites

1 Kubernetes Version

Kubernetes version 21.0.0 or above. Check with the following command:

Shell

shell

kubectl version

2 Middleware Kubernetes Agent

Install Middleware Kubernetes agent using these instructions. Check with the following command:

Shell

shell

kubectl get pods -n mw-agent-ns

Install

In most of cases, you will need to install cert-manager along with the Operator, unless you already have cert-manager installed or have alternative methods of generating certificates in your Kubernetes cluster.

Bash script

MW_INSTALL_CERT_MANAGER=true MW_API_KEY="<MW_API_KEY>" bash -c "$(curl -L https://install.middleware.io/scripts/mw-kube-auto-instrumentation-install.sh)"

Helm

helm repo add jetstack https://charts.jetstack.io --force-update

helm install cert-manager jetstack/cert-manager --namespace cert-manager --create-namespace --version v1.14.5 --set installCRDs=true

helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts

helm install opentelemetry-operator-controller-manager  open-telemetry/opentelemetry-operator --namespace opentelemetry-operator-system --create-namespace --values https://install.middleware.io/manifests/autoinstrumentation/helm-values.yaml

Configuration

Instrumentation Custom Resource

In order to auto-instrument your applications, the OTel Kubernetes Operator needs to be configured to know which Kubernetes pods to instrument and which automatic instrumentation configuration, called Instrumentation Custom Resource (CR), to use for those pods.

The OTel Kubernetes Operator installation steps described in the Installation section also installs Instrumentation CR in mw-agent-ns namespace.

To confirm that Instrumentation CR is installed, issue the folowing command

kubectl get otelinst mw-otel-auto-instrumentation -n mw-agen-ns

The Instrumentation CR looks like below

apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
  name: mw-otel-auto-instrumentation
  namespace: mw-agent-ns
spec:
  exporter:
    endpoint: http://mw-service.mw-agent-ns:9319
  propagators:
    - tracecontext

This Instrumentation CR will be used to annotate pods specifications. Below is the list of supported programming languages and the annotations used for auto-instrumentation.

Java

To enable auto-instrumentation for a Java application, add the following annotation to your pod specification:

instrumentation.opentelemetry.io/inject-java: "mw-agent-ns/mw-otel-auto-instrumentation"

Node.js

To enable auto-instrumentation for a Node.js application, add the following annotation to your pod specification:

instrumentation.opentelemetry.io/inject-nodejs: "mw-agent-ns/mw-otel-auto-instrumentation"

Python

To enable auto-instrumentation for a Python application, add the following annotation to your pod specification:

instrumentation.opentelemetry.io/inject-python: "mw-agent-ns/mw-otel-auto-instrumentation"

.NET

To enable auto-instrumentation for a .NET application and specify the runtime identifier (RID), use the following annotations:

For Linux glibc based images (default):

instrumentation.opentelemetry.io/inject-dotnet: "mw-agent-ns/mw-otel-auto-instrumentation"
instrumentation.opentelemetry.io/otel-dotnet-auto-runtime: "linux-x64"  # Optional as it's the default value

For Linux musl based images:

instrumentation.opentelemetry.io/inject-dotnet: "mw-agent-ns/mw-otel-auto-instrumentation"
instrumentation.opentelemetry.io/otel-dotnet-auto-runtime: "linux-musl-x64"

Go

To enable auto-instrumentation for a Go application, you need to set the path to the executable and ensure the container has elevated permissions. Use the following annotations in the pod specifications:

instrumentation.opentelemetry.io/inject-go: "mw-agent-ns/mw-otel-auto-instrumentation"
instrumentation.opentelemetry.io/otel-go-auto-target-exe: "/path/to/container/executable"

Additionally, set the required security context for the container in the pod specification (spec.containers) as shown below:

securityContext:
  privileged: true
  runAsUser: 0

The annotations described above for all the supported programming languages used mw-agent-ns/mw-otel-auto-instrumentation  as the value for the annotation. This is because the OTel Kubernetes Operator installed using Middleware installation instructions creates mw-otel-auto-instrumentation Instrumentation CR in mw-agent-ns namespace as described before.

You can also create your own Instrumentation CR in any namespace and use them in the auto-instrumentation annotations.

Below are the all possible values for the auto-instrumentation annotation:

• "true": Inject an Instrumentation CR from the current namespace. It is expected that you will have only one Instrumentation CR defined in the current namespace. The behavior could be unpredictable if you have multiple Instrumentation CRs defined in the current namespace.
• "my-instrumentation": Inject a specific Instrumentation CR with name my-instrumentation from the current namespace.
• "my-other-namespace/my-instrumentation": Inject a specific Instrumentation CR with name my-instrumentation from namespace my-other-namespace. This is the option used in all our examples above with Instrumentation CR name “mw-otel-auto-instrumentation” and namespace mw-agent.
• "false": Do not inject an Instrumentation CR. This is useful for temporarily stopping instrumentation.

Explore your data on Middleware

Once the auto-instrumentation is complete and the relevant pods are restarted, the data should start flowing to Middleware for any new requests made to your applications. You can explore APM traces data by going to the APM section of your Middleware account.

Uninstall

Bash

MW_UNINSTALL_CERT_MANAGER=true MW_API_KEY="your-initial-token"
bash -c "$(curl -L https://install.middleware.io/scripts/mw-kube-auto-instrumentation-uninstall.sh)"

Helm

helm uninstall opentelemetry-operator-controller-manager --namespace opentelemetry-operator-system

If you want to uninstall cert-manager, issue command below

helm uninstall cert-manager --namespace cert-manager

Additional Resources

• OpenTelemetry Operator documentation

• OpenTelemetry Operator README on GitHub