How to Send Application Logs from EC2 to CloudWatch Logs

This blog shows how to send application logs from Amazon Linux 2 EC2 Instance to AWS CloudWatch logs, it'll include how to set up and configure.

Let's assume you have a running web server on EC2 instance, and the application had written logs to this web server, now you want to move this logs to the CloudWatch for further analysis, the following are steps how you can accomplish. 

Pre requirement

• Set up an AWS account
• EC2 Instance is running and has application logs to /var/app/my-app/error.log
• EC2 Instance had attached a role, my-ec2-role

Create new custom Policy

Here, we'll create the following new custom policy for IAM and assign it to the role my-ec2-role for EC2. Once we attached this policy to the role EC2 is using, so our EC2 has permission to create logs into CloudWatch logs.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "logs:CreateLogGroup",
                "logs:CreateLogStream",
                "logs:PutLogEvents",
                "logs:DescribeLogStreams"
            ],
            "Resource": [
                "*"
            ]
        }
    ]
}

Install old CloudWatch logs agent

Update Amazon Linux 2 instance with the latest repositories.

sudo yum update -y

Install AWS logs to EC2 instance.

sudo yum install -y awslogs

You'll need to cd to the following file to modify the AWS region if you're not using the default region

/etc/awslogs/awscli.conf

open the awslogs.conf file to modify the configuration for the CloudWatch logs.

cd /etc/awslogs/
nano awslogs.conf

After opening the awslogs.conf file, you need to modify the file to specify the path of the logs, so the CloudWatch logs agent will look into that file and collect it into CloudWatch logs. Next, update the log_group_name, so when you go to the CloudWatch logs dashboard, the name you enter will appear. 

datetime_format = %b %d %H:%M:%S
file = /var/www/my-app/error.log
buffer_duration = 5000
log_stream_name = {instance_id}
initial_position = start_of_file
log_group_name = my-web-logs

After modifying the file, you can start the awslog service by the following command.

sudo systemctl start awslogsd

If you want to enable the awslog service after Amazon Linux 2 reboot, you can add the following command.

sudo systemctl enable awslogsd.service

Now, that's it, your application logs should be able to push into the CloudWatch logs. You. can go to the AWS CloudWatch Logs and check the log group you added at the EC2 instance. 

Conclusion

Logs are important to the applications, especially when something went wrong, you can use the logs to identify what's the root cause and figure out what's going on. 

What's next, seems like there is a new way to use CloudWatch Agent for both CloudWatch logs and metric, so will create a blog about that next time. Also, this blog is about how to set up in running instance, will check how to install the CloudWatch Logs when set up the EC2 instance.