Amazon ECS

The Middleware Agent collects task and container telemetry from Amazon Elastic Container Service (ECS) workloads on Amazon EC2 or AWS Fargate. Log collection and application tracing provide additional context when configured.

Use ECS telemetry to investigate task performance and container resource utilization, and application behavior across your ECS clusters..

For example, task metrics and logs can help explain why a workload slows down or stops unexpectedly.

Note: Amazon ECS Anywhere and database integrations are not currently supported through this installation.

How it works#

Middleware supports two instrumentation methods:

Manual instrumentation: Configure the Middleware Agent in your ECS task definitions. On Fargate, the Agent runs as a sidecar container. On EC2, it runs as a Daemon service on each container instance.
Auto-instrumentation: Use the mw-ecs CLI to automatically add the Middleware Agent, OpenTelemetry instrumentation libraries, and Fluent Bit log routing to existing ECS task definitions.

The Middleware Agent collects ECS metrics through the ECS Task Metadata Endpoint. Container metrics on EC2 can also be collected through Docker.

Prerequisites#

Before installing the Middleware Agent, verify the following:

An AWS account with an existing ECS cluster.
ECS workloads running on EC2 or Fargate.
Permissions to create, register, and update ECS task definitions and services.
A Middleware account with an API key (MW_API_KEY) and target URL (MW_TARGET).
Network connectivity from the Agent to your Middleware target URL over HTTPS.

For auto-instrumentation, additional AWS credentials and CLI requirements are listed in the relevant section.

Installation#

Choose the instrumentation method that matches your environment.

Method	Use case
Manual — Fargate	Deploy the Middleware Agent alongside individual ECS application tasks
Manual — EC2	Monitor ECS workloads by running one Agent per EC2 container instance
Auto-instrumentation	Add monitoring to existing ECS task definitions using the `mw-ecs` CLI

Manual instrumentation#

Method 1: Install on ECS Fargate#

For Fargate, the Middleware Agent runs as a sidecar container within each ECS task.

A task configured for metrics and log collection contains three containers:

Application container: Runs your application.
Middleware Agent: Collects and forwards telemetry to Middleware.
Fluent Bit: Routes application logs to the Middleware Agent using FireLens.

1 Add the Middleware Agent container#

Open your ECS task definition and add the following container to the containerDefinitions array.

1{
2  "name": "mw-agent",
3  "image": "ghcr.io/middleware-labs/mw-host-agent:master",
4  "cpu": 256,
5  "portMappings": [
6    {
7      "name": "8006-tcp",
8      "containerPort": 8006,
9      "hostPort": 8006,
10      "protocol": "tcp",
11      "appProtocol": "http"
12    }
13  ],
14  "essential": true,
15  "environment": [
16    {
17      "name": "MW_API_KEY",
18      "value": "<MW_API_KEY>"
19    },
20    {
21      "name": "MW_TARGET",
22      "value": "https://<MW_UID>.middleware.io:443"
23    }
24  ],
25  "mountPoints": [],
26  "volumesFrom": [],
27  "logConfiguration": {
28    "logDriver": "awslogs",
29    "options": {
30      "awslogs-group": "<YOUR_LOGS_GROUP>",
31      "mode": "non-blocking",
32      "awslogs-create-group": "true",
33      "max-buffer-size": "25m",
34      "awslogs-region": "<YOUR_REGION>",
35      "awslogs-stream-prefix": "ecs"
36    }
37  }
38}

Replace the placeholders with your Middleware credentials, AWS Region, and CloudWatch Logs group.

Note: For production deployments, use AWS Secrets Manager or another supported secret-management method instead of storing the API key in plaintext.

2 Collect ECS metrics#

After adding the Middleware Agent sidecar, ECS task and container metrics are collected automatically.

The Agent uses the ECS Task Metadata Endpoint to retrieve task and container resource information.

No additional metrics configuration is required for this step.

3 Configure task log collection#

To forward application logs to Middleware, add a Fluent Bit sidecar container and configure AWS FireLens.

Add the following container to your ECS task definition:

1{
2  "name": "log_router",
3  "image": "amazon/aws-for-fluent-bit:stable",
4  "cpu": 0,
5  "portMappings": [],
6  "essential": true,
7  "environment": [],
8  "mountPoints": [],
9  "volumesFrom": [],
10  "user": "0",
11  "firelensConfiguration": {
12    "type": "fluentbit",
13    "options": {
14      "enable-ecs-log-metadata": "true"
15    }
16  }
17}

Next, add the following logConfiguration to your primary application container:

1"logConfiguration": {
2  "logDriver": "awsfirelens",
3  "options": {
4    "Host": "127.0.0.1",
5    "Name": "forward",
6    "Port": "8006"
7  }
8}

FireLens forwards application logs to the Middleware Agent on port 8006.

Make sure the Middleware Agent sidecar is running before verifying log collection.

4 Parse structured JSON logs (optional)#

By default, FireLens forwards JSON logs as a structured payload, which Middleware displays in the log body.

To display the Message field as the log body and keep other JSON fields as attributes, update the Fluent Bit configuration:

1"firelensConfiguration": {
2  "type": "fluentbit",
3  "options": {
4    "enable-ecs-log-metadata": "true",
5    "config-file-type": "file",
6    "config-file-value": "/fluent-bit/configs/parse-json.conf"
7  }
8}

The application must emit structured JSON logs containing a Message field for this parsing configuration.

5 Deploy the updated task definition#

After configuring the containers:

Register a new revision of the ECS task definition.
Update your ECS service to use the new revision.
Deploy the updated task.
Verify that the application, Middleware Agent, and Fluent Bit containers are running.

Complete task definition example

{Retain the complete Fargate task definition JSON from the existing documentation, including the application container, Middleware Agent, and Fluent Bit log router.}

Method 2: Install on ECS EC2#

For ECS workloads running on EC2, deploy the Middleware Agent as a Daemon service.

This runs one Middleware Agent task on each eligible EC2 container instance in the ECS cluster.

1 Create the Middleware Agent task definition#

Create a task definition named mw-agent-task.

Choose the network mode based on your ECS environment:

Network mode	Description
`host`	Uses the EC2 host network directly
`bridge`	Uses Docker bridge networking
`awsvpc`	Assigns an independent network interface to the Agent task

The Agent task definition must include:

Middleware Agent image: ghcr.io/middleware-labs/mw-host-agent:master
MW_API_KEY and MW_TARGET environment variables
Required host filesystem mounts
Container ports 9319, 9320, and 8006
EC2 launch compatibility

Use the appropriate task definition for your network mode.

Host network mode

{Retain the complete Host Network Mode JSON from the existing documentation.}

Bridge network mode

{Retain the complete Bridge Network Mode JSON from the existing documentation.}

AWSVPC network mode

{Retain the complete AWSVPC Network Mode JSON from the existing documentation.}

Note: The EC2 configuration requires host-level mounts and privileged container access. Review these permissions before deploying the Agent.

2 Register the task definition#

You can register the Middleware Agent task definition using the AWS CLI or AWS Management Console.

Using AWS CLI

Save your task definition as mw-ecs-agent.json and run:

1aws ecs register-task-definition \
2  --cli-input-json file://mw-ecs-agent.json

Using AWS Management Console

Open Amazon ECS → Task Definitions.
Click Create new task definition.
Select the JSON configuration editor.
Paste your Middleware Agent task definition.
Register the task definition.

3 Deploy the Agent as a Daemon service#

Open Amazon ECS → Clusters.
Select the ECS cluster you want to monitor.
Click Create service.
Under Compute options, select Launch type and choose EC2.
Under Deployment configuration, select:
- Application type: Service
- Task definition: mw-agent-task
- Revision: Latest registered revision
- Service type: DAEMON
Enter a service name and create the service.

The Daemon scheduling strategy runs one Middleware Agent task per eligible EC2 container instance.

4 Configure service discovery for AWSVPC (optional)#

If you use awsvpc network mode, configure service discovery so application tasks can communicate with the Middleware Agent.

Enable Service discovery in the ECS service configuration.
Create or select an existing namespace.
Set the service discovery name to mw-agent.
Enable ECS task health propagation.
Configure DNS records as required by your VPC.

For example, if the service name is mw-agent and the namespace is mw-agent-ns, the service discovery endpoint is:

1mw-agent.mw-agent-ns

This endpoint is used by application tasks to send telemetry to the Middleware Agent.

5 Configure application traces and logs#

Update the task definition for each application you want to monitor.

For Host or Bridge network mode

Set the following environment variable in your application container:

1MW_AGENT_SERVICE=172.17.0.1

For AWSVPC network mode

Use the Middleware Agent service discovery endpoint:

1MW_AGENT_SERVICE=mw-agent.mw-agent-ns

For log collection, configure your application's logging settings to forward logs to the Agent on port 8006.

Use the following destinations:

Network mode	Log destination
Host / Bridge	`172.17.0.1:8006`
AWSVPC	`mw-agent.mw-agent-ns:8006`

After updating the application task definition, register a new revision and update the ECS service to use it.

6 Verify the Agent installation#

Go to Amazon ECS → Clusters → [Your Cluster] → Services.

Verify that:

The Middleware Agent Daemon service is running.
One Agent task is scheduled on each eligible EC2 container instance.
The Agent tasks report a healthy running status.
Application tasks can communicate with the Agent.

Auto-instrumentation#

Middleware provides the mw-ecs CLI to add observability components to existing ECS task definitions without manually configuring sidecars.

Auto-instrumentation supports Java, Node.js, and Python, with automatic language detection.

The CLI adds:

Middleware Agent sidecar: Collects and forwards telemetry.
APM init container: Provides OpenTelemetry instrumentation libraries.
Fluent Bit sidecar: Routes application logs using FireLens.

Prerequisites#

Before using auto-instrumentation, make sure you have:

An existing ECS task definition.
AWS credentials configured through environment variables, an AWS CLI profile, or an IAM role.
Middleware API key and target URL.
An active Middleware AWS integration if you want to discover task definitions from the Middleware UI.
Go 1.21 or later, only if building the mw-ecs CLI from source.

1 Install the mw-ecs CLI#

Run the following command on the machine used to manage your ECS deployments:

1MW_API_KEY=<MW_API_KEY> \
2MW_TARGET=https://<MW_UID>.middleware.io:443 \
3bash -c "$(curl -L https://install.middleware.io/scripts/mw-ecs-install.sh)"

The installation script downloads the mw-ecs CLI and configures it with your Middleware credentials.

You can also build the CLI from the mw-ecs GitHub repository.

2 Discover ECS task definitions#

List active ECS task definition families and their instrumentation status:

1mw-ecs discover --region <YOUR_REGION>

The output includes:

Task definition family
Latest available revision
MW agent sidecar enabled status
APM enabled status
Logs enabled status

3 Instrument task definitions#

To instrument a specific task definition:

1mw-ecs instrument \
2  --task-definition <FAMILY_NAME>:<REVISION> \
3  --enable-apm \
4  --enable-logs \
5  --register \
6  --region <YOUR_REGION>

This command adds the required Middleware components and registers a new ECS task definition revision.

Instrument multiple task definitions#

You can instrument multiple task definitions in a single command:

1mw-ecs instrument \
2  --task-definition my-api-service:3 \
3  --task-definition my-worker-service:5 \
4  --task-definition my-web-app:2 \
5  --enable-apm \
6  --enable-logs \
7  --register \
8  --region us-east-1

Instrument all active task definitions#

To instrument all active task definition families:

1mw-ecs instrument \
2  --all \
3  --enable-apm \
4  --enable-logs \
5  --register \
6  --region <YOUR_REGION>

Note: Use --dry-run to review the changes before registering new task definition revisions.

Instrumentation options#

Flag	Description
`--task-definition`	Task definition family and revision, or full ARN
`--all`	Instrument all active task definition families
`--language`	Override language detection (`java`, `node`, `python`)
`--libc`	Override C library detection (`glibc`, `musl`)
`--enable-apm`	Enable application trace collection
`--enable-logs`	Add FireLens log routing
`--register`	Register a new ECS task definition revision
`--run`	Run the task after registration
`--dry-run`	Preview changes without applying them
`--region`	AWS Region
`--cluster`	ECS cluster name
`--subnets`	Subnets for AWSVPC networking
`--security-groups`	Security groups for AWSVPC networking

4 Verify instrumentation#

After registering the instrumented task definition, run:

1mw-ecs discover --region <YOUR_REGION>

Verify that the task definition reports Middleware instrumentation.

Next, update your ECS service to use the newly registered revision.

Note: Registering a task definition does not automatically update an existing ECS service. Deploy the new revision to start collecting application telemetry.

Auto-instrument from the Middleware UI#

You can also generate instrumentation commands directly from the Middleware UI.

Go to Installation → Amazon ECS.
Select Auto-Instrumentation.
Configure the AWS integration if it is not already connected.
Review the discovered ECS task definitions.
Select the task definitions you want to instrument.
Enable the required options:
- Enable APM
- Enable Logs
- Register
Copy the generated mw-ecs instrument command.
Run the command on the machine where the mw-ecs CLI is installed.

After registration, update the ECS service to use the new task definition revision.

Verify data collection#

After deploying the Middleware Agent and updating your ECS services, allow a few minutes for telemetry to appear.

Go to Infrastructure Monitoring → ECS and verify that your ECS resources are reporting data.

After data collection starts, you can:

View ECS clusters, services, tasks, task definitions, and containers.
Monitor container CPU, memory, and other collected resource metrics.
View application and container logs in Log Monitoring.
View application traces in APM, if instrumentation is enabled.
Create alerts for collected metrics.
Create custom dashboard or view predefined dashboard, go to Dashboard Builder and select ECS Metrics Dashboard.

Roll back instrumentation#

To revert an instrumented task definition to a previous revision, run:

1mw-ecs rollback --task-definition <FAMILY_NAME>:<REVISION>

Verify the resulting task definition configuration and update the ECS service to use the intended revision.

Uninstall#

Remove the mw-ecs CLI#

To uninstall the mw-ecs CLI:

1bash -c "$(curl -L https://install.middleware.io/scripts/mw-ecs-uninstall.sh)"

This removes the CLI tool from your machine.

Note: Removing the CLI does not remove Middleware components from task definitions that were previously instrumented.

To stop collecting telemetry, roll back the instrumented task definitions and deploy the corresponding revisions.

Remove manually installed Agents#

For Fargate deployments, remove the Middleware Agent and Fluent Bit containers from the task definition, register a new revision, and deploy it.

For EC2 deployments, remove the Middleware Agent Daemon service after confirming that application tasks no longer depend on its telemetry endpoints.

Troubleshooting#

Middleware Agent is not running#

If the Agent task fails to start:

Verify that the ECS task definition contains the required Agent configuration.
Check the ECS task stopped reason and container exit codes.
Verify that MW_API_KEY and MW_TARGET are configured correctly.
Review Agent logs in the configured CloudWatch Logs group.
Confirm that the task has sufficient CPU and memory resources.

No ECS metrics in Middleware#

If the Agent is running but ECS metrics are not available:

Verify outbound connectivity to your Middleware target URL on port 443.
Check that the Agent can access the ECS Task Metadata Endpoint.
Verify that the Agent is deployed using the correct launch type and network mode.
Confirm that ECS tasks are running and reporting resource data.

Application logs are missing#

If application logs do not appear:

Verify that the Fluent Bit container is running.
Check that the application uses the awsfirelens log driver.
Verify that the configured log destination points to the Middleware Agent on port 8006.
Check Fluent Bit logs for forwarding or connection errors.

Application traces are missing#

If auto-instrumentation is enabled but traces do not appear:

Verify that the application language is supported.
Confirm that the instrumented task definition was registered successfully.
Verify that the ECS service is running the new task definition revision.
Check that the APM init container completed successfully.
Review the application and Middleware Agent logs for instrumentation errors.

Task definitions are not discovered#

If task definitions do not appear in the Middleware UI:

Verify that the AWS integration is connected.
Check that the AWS account and Region are correct.
Confirm that the integration has permission to discover ECS task definitions.
Verify that the task definitions are active.

If you cannot access the ECS installation page, your user role may not have the required Installation permission.

Contact your Middleware administrator to update your permissions in Settings.

If the issue persists, contact Middleware Support at [email protected].

Need assistance or want to learn more about Middleware? Get in touch with us via our Contact Us or join our Slack channel.

Amazon ECS

How it works#

Prerequisites#

Installation#

Manual instrumentation#

Method 1: Install on ECS Fargate#

1 Add the Middleware Agent container#

2 Collect ECS metrics#

3 Configure task log collection#

4 Parse structured JSON logs (optional)#

5 Deploy the updated task definition#

Method 2: Install on ECS EC2#

1 Create the Middleware Agent task definition#

2 Register the task definition#

3 Deploy the Agent as a Daemon service#

4 Configure service discovery for AWSVPC (optional)#

5 Configure application traces and logs#

6 Verify the Agent installation#

Auto-instrumentation#

Prerequisites#

1 Install the mw-ecs CLI#

2 Discover ECS task definitions#

3 Instrument task definitions#

Instrument multiple task definitions#

Instrument all active task definitions#

Instrumentation options#

4 Verify instrumentation#

Auto-instrument from the Middleware UI#

Verify data collection#

Roll back instrumentation#

Uninstall#

Remove the mw-ecs CLI#

Remove manually installed Agents#

Troubleshooting#

Middleware Agent is not running#

No ECS metrics in Middleware#

Application logs are missing#

Application traces are missing#

Task definitions are not discovered#

Installation menu is missing#

Ask AI about this page