Amazon CloudWatch is a monitoring and observability service that provides real-time insights into AWS resources and applications. While CloudWatch provides many default metrics, sometimes you need custom metrics to monitor specific aspects of your infrastructure or applications.
This guide covers everything you need to know about CloudWatch custom metrics, from basics to advanced use cases.
What Are CloudWatch Custom Metrics?
Amazon CloudWatch provides default metrics for AWS services like EC2, S3, and Lambda, but sometimes, these built-in metrics are not enough. Custom metrics allow you to track application-specific data that CloudWatch does not automatically provide.
For example, if you are running a web application, you might want to monitor:
- Number of active users at any given time
- API response times to track performance
- Failed login attempts for security monitoring
- Business KPIs like total purchases or user sign-ups
Custom metrics give you control over what data to collect, how often, and in what format, helping you gain better insights into your application's health and performance.
Why Use CloudWatch Custom Metrics?
There are several reasons why businesses and developers rely on custom metrics in CloudWatch:
1. Monitor Application-Specific Events and Data
AWS provides basic infrastructure-level monitoring, but custom metrics let you track what's happening inside your application. If you run an e-commerce site, you may want to monitor:
- Total number of orders placed per hour
- Inventory levels to ensure products donβt go out of stock
- Customer support requests volumes to track demand
These insights help with troubleshooting and decision-making.
2. Track Business-Related KPIs
Metrics are not just for technical monitoringβthey can also help track business performance indicators. Examples include:
- User sign-ups per day to measure growth
- Revenue per hour to monitor sales trends
- Churn rate (how many users stop using your service)
3. Gain Deeper Insights Into System Health and Performance
CloudWatch's built-in metrics might tell you how many requests your server is handling, but custom metrics let you dig deeper:
- Request latency β How long does it take for your API to respond?
- Error rates β How many failed API calls are happening?
- Database query times β Are slow queries causing bottlenecks?
With these insights, engineers can optimize performance and ensure smooth user experiences.
4. Create Custom Alarms for Unique Scenarios
AWS CloudWatch Alarms can notify you when a metric crosses a threshold, but default metrics may not cover your needs. With custom metrics, you can:
- Set up an alert when too many failed login attempts occur (possible security threat).
- Get notified if checkout failures spike (possible payment system issue).
- Trigger an auto-scaling event if API response times slow down.
These custom alarms help businesses stay proactive rather than reacting after an issue has already impacted users.
Types of CloudWatch Metrics
Amazon CloudWatch helps you monitor your AWS resources by collecting and displaying different types of data, called metrics.
These metrics act as real-time health indicators for your cloud setup. Below are the main types of CloudWatch metrics and what they track.
1. AWS Service Metrics
AWS automatically collects and provides metrics for its services. If you're using AWS resources like EC2 (virtual machines), S3 (storage), or RDS (databases), CloudWatch will track performance indicators such as:
- EC2 (Virtual Machines) β CPU usage, disk activity, and network traffic.
- RDS (Databases) β Query response time, active connections, and storage usage.
- S3 (Storage Buckets) β File storage count, access frequency, and request counts.
- Lambda (Serverless Functions) β Invocation count, execution duration, and error rates.
These built-in metrics are available with no additional setup, and many are free up to certain limits.
2. Custom Metrics
Sometimes, AWS's default metrics are not enough for your specific needs. Custom metrics allow you to track application-specific data, such as:
- User activity β Track daily active users, logins, or customer sessions.
- Business KPIs β Monitor total orders, revenue per hour, or abandoned carts.
- Custom performance indicators β Track request latency, error rates, or queue depths.
You can send custom metrics to CloudWatch using the AWS SDK, CLI, or CloudWatch Agent and then set up dashboards and alarms to track changes over time.
3. Server and Host Metrics
For EC2 instances or on-premises servers, CloudWatch can monitor system-level details, such as:
- Memory usage (not enabled by default, but can be added).
- Disk space to prevent running out of storage.
- Running processes to ensure key applications remain active.
To collect these metrics, install the CloudWatch Agent, which provides deeper insights into server performance.
4. Application Metrics
Application performance is critical to user experience. CloudWatch allows you to track:
- API response time β How long it takes your app to respond to requests.
- Error rates β The percentage of failed requests or system crashes.
- User activity β Number of active users, transactions, or real-time session counts.
By monitoring these, you can detect slowdowns or failures before they impact customers.
5. Log-Based Metrics
CloudWatch Logs store system and application logs, but you can also extract key insights by turning logs into metrics. Examples include:
- Failed login attempts to identify potential security threats.
- HTTP 500 errors to detect website crashes.
- Database errors to pinpoint application issues.
These log-based metrics help with troubleshooting and security monitoring.
6. Billing and Cost Metrics (Avoid Unexpected Bills!)
AWS costs can add up quickly, but CloudWatch can help you track:
- Your total AWS bill in real time to avoid surprises.
- Usage of free-tier services to prevent accidental charges.
- Cost alerts to notify you when spending exceeds a set threshold.
Setting up cost monitoring ensures you stay within your budget and avoid unexpected charges.
7. Event-Driven Metrics
These metrics track events triggered by changes in your AWS environment, such as:
- Auto Scaling changes β When AWS adds or removes EC2 instances based on demand.
- Failed notifications β If an email or SMS alert from SNS fails to send.
- Step Functions executions β Monitoring the success or failure of automation workflows.
These event-driven metrics provide insight into dynamic system changes, allowing for faster response times.
How to Implement CloudWatch Custom Metrics?
There are three main ways to send custom metrics to CloudWatch:
- Using the PutMetricData API β The simplest way to send a custom metric via AWS CLI or SDKs.
- Using CloudWatch Agent β Collects system-level metrics from servers.
- Using Embedded Metric Format (EMF) β Logs structured JSON data that CloudWatch extracts automatically.
Each method is useful for different scenarios, and businesses often use a combination of them to get a complete view of their systems.
Step-by-Step Process to Set Up Custom Metrics in CloudWatch
Custom metrics in Amazon CloudWatch allow you to track specific data points that are not provided by AWS by default. This section explains the prerequisites and different methods for sending custom metrics to CloudWatch.
Prerequisites
Before setting up custom metrics, ensure you have the following:
- AWS Account with CloudWatch Permissions β You need an AWS account with access to CloudWatch.
- AWS CLI or SDK Installed β Required if you plan to send metrics programmatically.
- IAM Role with CloudWatch Permissions β The IAM role or user must have
cloudwatch:PutMetricData
permission to send custom metrics.
Methods to Send Custom Metrics
You can send custom metrics to CloudWatch using the following methods:
1. AWS CLI
The AWS CLI allows you to send custom metrics using the put-metric-data
command.
Example Command:
aws cloudwatch put-metric-data --namespace "Custom/MyApp" --metric-name "PageLoadTime" --value 1.25
--namespace
β Groups related metrics (e.g.,Custom/MyApp
).--metric-name
β The name of the metric (e.g.,PageLoadTime
).--value
β The data point being recorded (e.g.,1.25
seconds).
2. AWS SDK (Python Example using Boto3)
You can send custom metrics programmatically using AWS SDKs. Below is a Python example using Boto3:
import boto3
cloudwatch = boto3.client('cloudwatch')
response = cloudwatch.put_metric_data(
Namespace='Custom/MyApp',
MetricData=[
{
'MetricName': 'PageLoadTime',
'Value': 1.25,
'Unit': 'Seconds'
}
]
)
print(response)
This sends a metric named PageLoadTime
under the namespace Custom/MyApp
with a value of 1.25
seconds.
3. CloudWatch Agent
The CloudWatch Agent can collect both system-level and custom application metrics.
Steps to Set Up the CloudWatch Agent:
- Install the Agent on your EC2 instance or on-premises server:
sudo yum install amazon-cloudwatch-agent -y
- (For Ubuntu/Debian, use
sudo apt install amazon-cloudwatch-agent
.)
- Configure the Agent by creating or editing the
amazon-cloudwatch-agent.json
file:
{
"metrics": {
"append_dimensions": {
"InstanceId": "${aws:InstanceId}"
},
"metrics_collected": {
"cpu": {
"measurement": ["cpu_usage_idle"],
"metrics_collection_interval": 60
}
}
}
}
- Start the Agent:
sudo amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -c file:/opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json -s
Once started, the agent will continuously send custom metrics to CloudWatch.
Choosing the Right Method
Method | Best For | Setup Complexity |
---|---|---|
AWS CLI | Quick one-time metric updates | Easy |
AWS SDK | Programmatic metric submission | Medium |
CloudWatch Agent | Continuous system-level monitoring | Moderate |
What are CloudWatch Namespaces, Dimensions, and Units?
When working with CloudWatch, understanding Namespaces, Dimensions, and Units is essential for organizing and interpreting your custom metrics. These elements help you structure your data in a way that makes it easy to monitor and analyze.
Namespaces
A namespace is essentially a container that groups related CloudWatch metrics. It helps organize metrics into logical categories, such as AWS service-specific metrics (e.g., AWS/EC2
) or custom metrics that you define.
- Predefined Namespaces: AWS provides default namespaces for its services. For example,
AWS/EC2
is the namespace for EC2 instance metrics. - Custom Namespaces: If you're collecting custom metrics, you need to create a unique namespace. For example, you might use
Custom/MyApp
as a namespace for metrics related to your application.
Example:
aws cloudwatch put-metric-data --namespace "Custom/MyApp" --metric-name "PageLoadTime" --value 2.5
In this example, the metric PageLoadTime
is under the custom namespace Custom/MyApp
.
Dimensions
Dimensions are name/value pairs that provide additional context to a metric. They allow you to categorize and filter your metrics by specific attributes. For example, you might want to track the RequestCount
metric for different instances, regions, or environments.
Example:
aws cloudwatch put-metric-data --namespace "Custom/MyApp" --metric-name "RequestCount" --value 5 --dimensions "InstanceId=i-1234567890abcdef0"
Here, the RequestCount
metric is tagged with a Dimension
of InstanceId=i-1234567890abcdef0
, allowing you to monitor this metric for a specific EC2 instance.
Dimensions are powerful for filtering metrics when creating dashboards or setting up alarms. You can add multiple dimensions to give even more context to your data.
Units
Units help specify the kind of data the metric represents, ensuring that the values are interpreted correctly. When sending custom metrics, you can define a unit for each metric, such as seconds, bytes, or percent, depending on what the metric measures.
Here are some common units in CloudWatch:
- Seconds: Used for time-based measurements like response times or latency.
- Bytes: Used for size-based measurements like memory usage or file size.
- Percent: Used for percentage-based metrics like CPU usage or error rates.
- Count: Used for metrics that measure the number of occurrences, like transactions or API requests.
Example:
aws cloudwatch put-metric-data --namespace "Custom/MyApp" --metric-name "RequestLatency" --value 1.25 --unit "Seconds"
In this case, RequestLatency
is measured in seconds, meaning the value 1.25
represents the time taken to process a request.
How to Query and Visualize Custom Metrics in CloudWatch
Once you have set up custom metrics in CloudWatch, you can easily query and visualize them to gain valuable insights into your application's performance and system health. Here's how you can do that.
Viewing Metrics in the CloudWatch Console
To view your custom metrics, follow these steps:
- Navigate to the CloudWatch Console
Open the AWS Management Console and go to the CloudWatch service. - Click on Metrics in the Left Panel
In the CloudWatch dashboard, click on Metrics from the left-hand navigation menu. - Select Custom Metrics and Choose Your Namespace
Under the Browse tab, click on Custom Metrics. You'll see a list of namespaces that you have created or have access to.
Select the namespace where your custom metrics are located (e.g., Custom/MyApp). - Select Metrics to View Graphs and Statistics
Once you select the namespace, CloudWatch will display a list of metrics under that namespace.
Click on the checkbox next to a metric (e.g., PageLoadTime, RequestCount) to view the corresponding graph.
You can see the graph, statistical data (like average, minimum, maximum), and any trends in the data.
Creating CloudWatch Dashboards
CloudWatch Dashboards provide a way to consolidate and visualize multiple metrics in one place, making it easy to monitor key performance indicators at a glance.
- Go to CloudWatch Dashboards
From the CloudWatch console, click on Dashboards in the left panel. - Click Create Dashboard and Add Widgets
Click the Create dashboard button.
You will be prompted to give your dashboard a name and start adding widgets. - Select Metrics, Then Choose Your Custom Namespace
For each widget, click on Add widget, and choose the Metrics option.
Under the Browse tab, select your custom namespace (e.g., Custom/MyApp) and pick the relevant metric(s) you want to track. - Customize Graphs to Monitor Trends
After selecting your metrics, you can customize the visualization type (e.g., line graph, bar chart, etc.) and adjust the time range.
For example, you might set a period of the last 24 hours or a rolling 7-day window to monitor trends.
You can also modify how the graph displays, such as using percentile statistics, thresholds, or grouping metrics.
How to Set Alarms for Custom Metrics
CloudWatch Alarms allow you to take automated actions based on metric thresholds. This helps you proactively monitor your application's performance and respond to issues in Metric.
Creating an Alarm
Follow these steps to set up an alarm for your custom metric:
- Navigate to CloudWatch Alarms
- Open the AWS Management Console and go to the CloudWatch service.
- Click on Alarms in the left-hand navigation panel.
- Click Create Alarm
- Click the Create alarm button to start the setup process.
- Choose the Metric, Namespace, and Dimensions
- Click Select Metric and navigate to Custom Metrics.
- Choose the namespace where your custom metric is stored (e.g.,
Custom/MyApp
). - Select the metric (e.g.,
PageLoadTime
) and apply any relevant dimensions (e.g., instance ID, region).
- Set Conditions and Threshold Values
- Choose the statistic (e.g.,
Average
,Sum
,Minimum
,Maximum
) that best represents the metric behavior. - Define the threshold condition (e.g., trigger an alarm if the metric exceeds
2.0
seconds). - Specify the evaluation period and data points needed to trigger the alarm.
- Choose the statistic (e.g.,
- Configure Notifications Using SNS (Simple Notification Service)
- Under Actions, select an SNS topic to send notifications when the alarm state changes.
- If you donβt have an SNS topic, create a new one and subscribe to a recipient (email, Lambda function, etc.).
- Review and Create the Alarm
- Verify all settings and click Create an alarm to activate it.
Example: Alarm for High Latency
The following AWS CLI command creates an alarm that triggers when the PageLoadTime
metric exceeds 2.0
seconds:
aws cloudwatch put-metric-alarm --alarm-name "HighLatencyAlarm" \
--metric-name "PageLoadTime" --namespace "Custom/MyApp" \
--statistic Average --period 60 --threshold 2.0 \
--comparison-operator GreaterThanThreshold --evaluation-periods 1 \
--alarm-actions arn:aws:sns:us-east-1:123456789012:MyNotificationTopic
Breakdown of parameters:
--alarm-name "HighLatencyAlarm"
β The name of the alarm.--metric-name "PageLoadTime"
β The custom metric being monitored.--namespace "Custom/MyApp"
β The namespace where the metric is stored.--statistic Average
β Evaluate the average value of the metric.--period 60
β Checks the metric every 60 seconds.--threshold 2.0
β Alarm triggers if the value exceeds 2.0 seconds.--comparison-operator GreaterThanThreshold
β Alarm activates if the metric is greater than the threshold.--evaluation-periods 1
β Requires one consecutive period of data to trigger.--alarm-actions arn:aws:sns:us-east-1:123456789012:MyNotificationTopic
β Sends a notification via SNS when triggered.
Best Practices for Custom Metrics
Implementing CloudWatch custom metrics efficiently can improve monitoring while optimizing cost and performance. Follow these best practices to get the most value from your custom metrics.
Use Consistent Namespaces for Better Organization
- Define clear and structured namespaces (e.g.,
Custom/MyApp
,Custom/Backend
,Custom/Frontend
). - Avoid overly generic names like
Custom/App
to prevent confusion when scaling monitoring efforts.
Minimize the Number of Unique Dimensions to Reduce Costs
- Each unique dimension combination creates a separate time series, increasing storage costs.
- Use only essential dimensions (e.g., instance ID, region) instead of excessive metadata.
Use Appropriate Data Aggregation Techniques
- Choose the right statistic for analysis:
- Average for trends and performance tracking.
- Sum for counting total occurrences (e.g., API calls).
- Max/Min for identifying peak values (e.g., highest latency).
- Reduce data noise by sending aggregated values at a reasonable interval (e.g., every minute instead of every second).
Implement Log Filtering to Send Only Necessary Data
- Use Amazon CloudWatch Logs Insights or AWS Lambda to process logs before sending metrics.
- Avoid sending raw logs as metricsβextract key values instead (e.g., error rates, transaction counts).
Automate Metric Collection
- Use AWS Lambda to automatically collect and send custom metrics based on event triggers.
- Configure CloudWatch Agent to send system-level metrics from EC2 instances, containers, or on-premises servers.
- Write scripts using AWS SDKs (e.g., Python Boto3, AWS CLI) to automate metric reporting.
CloudWatch Pricing Breakdown
CloudWatch pricing is based on the number of custom metrics, API calls, alarms, and dashboards. Understanding these costs can help optimize your monitoring strategy.
Custom Metrics Pricing Breakdown
- Number of stored metrics β $0.30 per metric per month (first 10,000 metrics).
- API requests to send data β Charges apply based on request volume.
- Alarms and dashboards β Each custom metric used in an alarm or dashboard adds to costs.
Cost Optimization Strategies
- Use fewer dimensions to avoid unnecessary time series.
- Aggregate data before sending (e.g., compute averages locally rather than sending raw data).
- Use CloudWatch Embedded Metrics Format (EMF) to batch multiple data points, reducing API calls.
Conclusion
AWS CloudWatch custom metrics help you track what mattersβyour appβs performance, system health, and key business metrics. This guide will show you exactly how to set them up and make the most of them.