Log collection

Middleware Log Monitoring brings logs from infrastructure, applications, cloud services, serverless environments, and third party log collectors to a single place to search and investigate workflow.

Choose your log source below and follow the corresponding setup guide to start sending logs to Middleware.

After ingestion, Log Explorer provides search, filtering, and event inspection so you can investigate errors and activity across your configured sources.

Setup Log collection#

Choose a collection method based on where your logs are generated.

Host logs#

Install the Middleware Agent to collect logs from your hosts.

Application logs#

For application-generated logs, use one of the following methods:

Auto-instrument your application to collect supported application logs without modifying your application code.
Install the Middleware APM SDK for your application's language or framework.

Windows Event Logs#

Use the Windows Event Logs integration to collect Windows Event Logs.

Custom logs#

Use the Custom Logs API to send custom or programmatically generated logs directly to Middleware.

OpenTelemetry#

If your application or infrastructure already uses OpenTelemetry, configure the OpenTelemetry SDK to send logs to Middleware.

Serverless applications#

For serverless workloads, use the collection method supported by your platform:

Use the APM package to collect logs from supported serverless applications.
For AWS Lambda, use the OpenTelemetry Lambda Extension to collect logs through the AWS Lambda Telemetry API.

Cloud integrations#

Use the corresponding cloud integration to collect logs from supported cloud services:

Third-party log collectors#

if you already use a log collector, configure it to forward logs to Middleware:

Other Log Sources#

If your log source is not listed, contact [email protected] for the appropriate collection method.

View Collected Logs#

After logs are collected and ingested, go to: Log Monitoring → Log Explorer

Use Log Explorer to search, filter, and inspect logs collected from your configured sources.

Troubleshooting#

FAQ#

Logs are not visible after configuration, what to do?

If logs are not appearing in Log Explorer after configuration, check the following based on your log source.

Infrastructure logs

If logs are collected through the Middleware Host Agent or Kubernetes Agent, enable the debug exporter in otel-config.yaml. This can help identify common configuration issues, such as:

Permission issues
Missing log paths
Incorrect or mismatched log paths

Application logs

Middleware application instrumentation is based on OpenTelemetry. If application logs are missing, set OTEL_LOG_LEVEL=debug to enable detailed OpenTelemetry logging.

Also verify that the logger used by your application is supported by the OpenTelemetry SDK for your language. Refer to the relevant Middleware APM SDK repository or contact the Middleware team if you need help confirming support.

How long does it take for logs to appear?

Once the log collection configuration is complete, logs should appear in the Log Explorer in near real time.

How often are logs collected?

Logs are collected continuously rather than at a fixed polling interval.

When using the Middleware Agent, logs are read as they are written and forwarded through the OpenTelemetry Collector pipeline. Batching and export behavior depend on the Collector configuration, but logs typically reach Middleware within seconds.

What is the default log collection rate?

By default, Middleware collects 100% of logs and does not drop logs at the collection layer.

To control which logs are sent or retained, use Log Pipelines to filter or drop logs based on your requirements.

Need assistance or want to learn more about Middleware? Get in touch with us via our Contact Us or join our Slack channel.