Troubleshooting the Middleware Go APM

If you encounter issues or don't see data appearing in your Middleware.io dashboard, here are common troubleshooting steps.

Check Configuration:
- API Key (accessToken) and Target URL (target): Double-check that your MW_API_KEY (or accessToken in track.Track) and MW_TARGET (or target) are correctly set and free of typos. These are critical for connecting to Middleware.io.
- Service Name (service): Ensure MW_SERVICE_NAME (or service in track.Track) is set. This is how your application is identified in the Middleware.io UI.
- Environment Variables vs. Code: Remember that environment variables take precedence. If you're setting values in code and they don't seem to apply, check if an environment variable is overriding them.
Verify Agent Initialization:
- Ensure track.Track() is called exactly once at the beginning of your main function.
- Confirm that defer config.Shutdown() is present to properly flush telemetry data on exit. Without this, some data might be lost.
Check Go Module Installation:
- Verify dependencies: After go get commands, ensure your go.mod file has entries like github.com/middleware-labs/golang-apm/tracker.
- Run go mod tidy: This command cleans up your go.mod and go.sum files by removing unused dependencies and adding any missing ones necessary for your project. Run it after adding new go get dependencies.
```
go mod tidy
```
  Explanation: go mod tidy ensures that go.mod accurately reflects the project's imports and that go.sum contains correct checksums for all dependencies. Expected Output (if changes are made):
```
go: added <some_dependency> vX.Y.Z
go: removed <some_unused_dependency>
```
  (No output if everything is already tidy.)
- Run go mod vendor (if using vendoring): If your project uses Go modules' vendoring feature (where dependencies are copied into a vendor directory within your project), ensure you run this command.
```
go mod vendor
```
  Explanation: go mod vendor copies all necessary dependencies into a local vendor/ directory, allowing your build to be completely isolated from network access to module proxies. This is common in some CI/CD environments. Expected Output (similar to):
```
go: vendoring modules in <your_project_path>/vendor
```
Increase Agent Logging Level:
- Set the MW_LOG_LEVEL environment variable to DEBUG (e.g., export MW_LOG_LEVEL=DEBUG) or use track.WithConfigTag("logLevel", "DEBUG") during initialization.
- Examine your application's console output for any log messages from middleware-io or opentelemetry. Look for ERROR or WARN messages that indicate connection issues, authentication failures, or processing problems. You should see messages indicating successful initialization and data export.
Network Connectivity:
- From the host or container where your Go application is running, try to ping or curl your MW_TARGET URL to confirm network reachability on port 443.
```
curl -v -o /dev/null -s -w "%{http_code}\n" https://your_middleware_uid.middleware.io:443
```
  Explanation: This command attempts to make an HTTPS connection. A 200 or 30X HTTP status code indicates a successful connection to the endpoint. Expected Output (for successful connection):
```
* Trying <IP_ADDRESS>:443...
* Connected to your_middleware_uid.middleware.io (<IP_ADDRESS>) port 443
* ALPN: offers h2
* ALPN: offers http/1.1
...
< HTTP/1.1 200 OK
...
200
```
  (If you see "Connection refused", "Operation timed out", or similar, it indicates a network issue.)
- Check firewall rules, security groups, or proxy configurations that might be blocking outbound HTTPS traffic to the Middleware.io ingestion endpoint.
Application Activity:
- Ensure your application is actively handling requests or performing operations that would generate traces, metrics, and logs. If your Go web server isn't receiving requests, there won't be any APM data. Manually make some requests to your application's endpoints using curl or a web browser.
```
curl http://localhost:8080/hello
```
  Expected Output:
```
{"message":"Hello from Go APM!"}
```
  (And you should see corresponding logs in your application's console if MW_CONSOLE_EXPORTER is true.)
- Resource Limits (Kubernetes/Containers): In containerized environments, confirm that your pods have sufficient CPU and memory resources. A resource-starved application or agent might struggle to process and export telemetry data. Review your Kubernetes resource limits and requests.
Time Synchronization:
- Ensure the system clock on your application host is synchronized using NTP. Significant clock drift can lead to issues with timestamping and data correlation in observability platforms.
OpenTelemetry Compatibility:
- Middleware.io's agent is OpenTelemetry-native. If you're using other OpenTelemetry libraries directly, ensure they are compatible and correctly configured to use the global TracerProvider and MeterProvider initialized by middleware-io/golang-apm/tracker.