How to Send Application Logs from EC2 to CloudWatch Logs

Table of Contents

1. Introduction
2. Prerequisites
3. Create a New Custom Policy
4. Install the CloudWatch Logs Agent
5. Conclusion

Introduction

This blog shows how to send application logs from an Amazon Linux 2 EC2 instance to AWS CloudWatch Logs, including how to set up and configure the agent.

Let's assume you have a running web server on an EC2 instance, and the application has written logs to the web server. Now you want to move these logs to CloudWatch for further analysis. The following are the steps to accomplish this.

Prerequisites

• Set up an AWS account

• EC2 instance is running and has application logs at /var/app/my-app/error.log

• EC2 instance has an attached role, my-ec2-role

Create a New Custom Policy

Here, we'll create the following new custom policy for IAM and assign it to the role my-ec2-role for EC2. Once we attach this policy to the role the EC2 instance is using, the instance will have permission to create logs in CloudWatch Logs.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "logs:CreateLogGroup",
                "logs:CreateLogStream",
                "logs:PutLogEvents",
                "logs:DescribeLogStreams"
            ],
            "Resource": [
                "*"
            ]
        }
    ]
}

Install the CloudWatch Logs Agent

Update the Amazon Linux 2 instance with the latest repositories.

sudo yum update -y

Install the AWS Logs agent on the EC2 instance.

sudo yum install -y awslogs

You'll need to modify the following file to change the AWS region if you're not using the default region.

/etc/awslogs/awscli.conf

Open the awslogs.conf file to modify the configuration for CloudWatch Logs.

cd /etc/awslogs/
nano awslogs.conf

After opening the awslogs.conf file, you need to modify it to specify the path of the logs, so the CloudWatch Logs agent will look at that file and collect the logs into CloudWatch. Next, update the log_group_name so that when you go to the CloudWatch Logs dashboard, the name you enter will appear there.

datetime_format = %b %d %H:%M:%S
file = /var/www/my-app/error.log
buffer_duration = 5000
log_stream_name = {instance_id}
initial_position = start_of_file
log_group_name = my-web-logs

After modifying the file, you can start the awslog service with the following command.

sudo systemctl start awslogsd

If you want to enable the awslog service to start after Amazon Linux 2 reboots, you can run the following command.

sudo systemctl enable awslogsd.service

Now, that's it. Your application logs should be pushed into CloudWatch Logs. You can go to AWS CloudWatch Logs and check the log group you configured on the EC2 instance.

Conclusion

Logs are important for applications, especially when something goes wrong. You can use the logs to identify the root cause and figure out what's going on.

What's next: there is a newer way to use the CloudWatch Agent for both CloudWatch Logs and metrics, so we will create a blog about that next time. Also, this blog covers how to set up logging on a running instance. We will also look into how to install the CloudWatch Logs agent when initially setting up the EC2 instance.