Troubleshooting issues with the Python APM agent

Telemetry-related troubleshooting and debugging

To help troubleshoot and debug issues with your application's telemetry, you can enable specific settings in the mw_tracker function or use environment variables. These settings allow you to export telemetry data to the console, save them to log files, or debug basic information logs.

#in the `mw_tracker` function from middleware
log_level = "DEBUG"

#as an env variable
MW_LOG_LEVEL = "DEBUG"

#in the `mw_tracker` function from middleware
console_exporter = true

#as an env variable
MW_CONSOLE_EXPORTER = true

#in the `mw_tracker` function from middleware
debug_log_file = true

#as an env variable
MW_DEBUG_LOG_FILE = true

yum -y install python3-devel
yum -y install gcc-c++

apt install -y python3-dev
apt install -y build-essential

apk add python3-dev
apk add build-base

OTEL_BSP_SCHEDULE_DELAY = 100

Data isn't appearing in Middleware dashboard

If you don't see data appearing in your Middleware.io dashboard, here are common troubleshooting steps.

1. Check Environment Variables:

   • Verify that MW_TRACKER is True if you used the function-based instrumentation method.
   • Confirm MW_API_KEY and MW_TARGET are exactly correct (no typos, extra spaces).
   • Ensure variables are properly exported in your shell, Dockerfile, or Kubernetes manifest.

2. Verify middleware-run Usage:

   • Crucially, confirm your application is started using the middleware-run wrapper (e.g., middleware-run python your_app.py). Starting directly (python your_app.py) will prevent proper agent initialization.

3. Inspect middleware-bootstrap Results:

   • Run pip list | grep opentelemetry to check if opentelemetry-instrumentation- packages (like opentelemetry-instrumentation-flask) are installed for your dependencies.
   • If any are missing, re-run middleware-bootstrap -a install and ensure it has access to your requirements.txt or installed packages.

4. Increase Agent Logging Level:

   • Set MW_LOG_LEVEL=DEBUG in your environment variables before running your application.
   • Examine your application's console output for opentelemetry or middleware-io messages, especially ERROR or WARNING messages, or lines indicating "Middleware Agent initialized successfully."

5. Network Connectivity:

   • From your application's host/container, try to ping or curl your MW_TARGET URL to confirm network reachability.
   • Check firewall rules, security groups, or proxy configurations that might be blocking outbound traffic on port 443.

6. Application Activity:

   • Ensure your application is actively receiving traffic or performing operations that generate telemetry data. If your web app isn't getting requests, there won't be any traces or request metrics. Manually make some requests to test.

7. Resource Limits (Kubernetes/Containers):

   • In containerized environments, confirm your pods have sufficient CPU and memory resources. A resource-starved agent may fail to process and export data.

8. Time Synchronization:

   • Ensure the system clock on your application host is synchronized. Significant clock drift can lead to issues with timestamping and data correlation in observability platforms.