Ingestion Controls

The Ingestion Controls section is where users can tailor their log tracking infrastructure. Easily toggle your Middleware Agent logs, metrics, and trace ingestion on and off. The Middleware Host Agent will search for changes in your account configuration and automatically update the configuration YAML file and reload itself.

Fine-tune your log collection process by selecting your preferred hosts for tracking infrastructure file-based logs, Middleware Agent Logs, and specify custom directory paths for monitoring your log collection.

Infrastructure File-Based Logs

Quickly enable OR disable the Middleware Agent from monitoring file-based logs for collection. Infrastructure filed-based logs are collected by default.

To disable the collection of file-based logs, navigate to Settings, select Ingestion Controls, and press the Infrastructure File-based Logs radio button.

Middleware Agent Logs

Quickly enable or disable Middleware Agent log collection. Middleware Agent pod-logs are generated by the Middleware Agent.

Middleware Agent Log Paths

You can dynamically select the log files you want to monitor. Add directory paths in the Middleware Agent Log Paths section.

Below is the accepted regex pattern:

'^(\\/(?:[\\w_\\-^* ]+\\/)*(?:[\\w_\\-*]+(?:\\**\\/)?)?[\\w_\\-.]+[^.]$)'

The follwing are examples of available paths:

/home/ubuntu/abc.log

/home/ubuntu/*.log

/home/ubuntu/**/*.log

Custom Attribute Controls

Middleware supports standard regex rules to allow extraction of attributes and their values from log messages. The attributes scraped will be available in filters, search bars, and expanded log attributes.

Log Extraction Example

Using the below log message as an example, we are going to extract four key attributes: date, process_id, level, and content.

[2024-02-26 09:15:41 -0500] [671097] [ERROR] Shutting down: Master

We can extract the desired attributes with the below regex:

\[(?P<date>.+)\] \[(?P<process_id>.+)\] \[(?P<level>.*)\] (?P<content>.+)

Given the original log message and above regex, we receive the following key and value pairs:

date = 2024-02-26 09:15:41 -0500
process_id = 671097
level = ERROR
content =Shutting down: Master

The following image shows how this looks from within the Middleware Application:

Sensitive Data Masking for Logs

Middleware provides users the ability to mask data at time of ingestion. This means that all data will be masked before it leaves your systems, allowing full control over security regulations and company policies.

Similar to the above attribute extraction, Data Masking is also based on standard regex rules.

The setting will expect 2 inputs from users as shown in the image below:

1. A Regex Pattern for the keywords that users want to hide
2. A Replacement Value, e.g. *****, HIDDEN, etc.

If your log content looks like: Transaction ID: 675-98-3434

\\d{3}-?\\d{2}-?\\d{4} modifies the log before sending: Transaction ID: *****

Metrics Ingestion

By default, MW Agent metrics are automatically ingested. To disable this feature, press the radio button and click Save. Disabling metrics will limit your overall understanding of your infrastructure vitals and performance.

Trace Ingestion

By default, MW Agent traces are automatically ingested. To disable this feature, press the radio button and click Save. Disabling traces will limit your ability to isolate issues within your infrastructure.