Using AWS CloudWatch Logs and AWS ElasticSearch for log aggregation and visualization

If you run your infrastructure in AWS, then you can use CloudWatch Logs and AWS ElasticSearch + Kibana for log aggregation/searching/visualization as an alternative to either rolling your own ELK stack, or using a 3rd party SaaS solution such as Logentries, Loggly, Papertrail or the more expensive Splunk, Sumo Logic etc.

Here are some pointers on how to achieve this.

1) Create IAM policy and role allowing read/write access to CloudWatch logs

I created a IAM policy called cloudwatch-logs-access with the following content:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "logs:CreateLogGroup",
                "logs:CreateLogStream",
                "logs:PutLogEvents",
                "logs:DescribeLogStreams"
            ],
            "Resource": [
                "arn:aws:logs:*:*:*"
            ]
        }
    ]
}


Then I create an IAM role called cloudwatch-logs-role and attached the cloudwatch-logs-access policy to it.

2) Attach IAM role to EC2 instances

I attached the cloudwatch-logs-role IAM role to all EC2 instances from which I wanted to send logs to CloudWatch (I went to Actions --> Instance Settings --> Attach/Replace IAM Role and attached the role)

3) Install and configure CloudWatch Logs Agent on EC2 instances

I followed the instructions here for my OS, which is Ubuntu.

I first downloaded a Python script:

# curl https://s3.amazonaws.com/aws-cloudwatch/downloads/latest/awslogs-agent-setup.py -O

Then I ran the script in the region where my EC2 instances are:

# python awslogs-agent-setup.py --region us-west-2 Launching interactive setup of CloudWatch Logs agent ... Step 1 of 5: Installing pip ...DONE Step 2 of 5: Downloading the latest CloudWatch Logs agent bits ... DONE Step 3 of 5: Configuring AWS CLI ... AWS Access Key ID [None]: AWS Secret Access Key [None]: Default region name [us-west-2]: Default output format [None]: Step 4 of 5: Configuring the CloudWatch Logs Agent ... Path of log file to upload [/var/log/syslog]: Destination Log Group name [/var/log/syslog]: Choose Log Stream name: 1. Use EC2 instance id. 2. Use hostname. 3. Custom. Enter choice [1]: 2 Choose Log Event timestamp format: 1. %b %d %H:%M:%S (Dec 31 23:59:59) 2. %d/%b/%Y:%H:%M:%S (10/Oct/2000:13:55:36) 3. %Y-%m-%d %H:%M:%S (2008-09-08 11:52:54) 4. Custom Enter choice [1]: 3 Choose initial position of upload: 1. From start of file. 2. From end of file. Enter choice [1]: 1 More log files to configure? [Y]:

I continued by adding more log files such as apache access and error logs, and other types of logs.

You can start/stop/restart the CloudWatch Logs agent via:

# service awslogs start

The awslogs service writes its logs in /var/log/awslogs.log and its configuration file is in /var/awslogs/etc/awslogs.conf.

4) Create AWS ElasticSearch cluster

Not much to say here. Follow the prompts in the AWS console :)

For the initial Access Policy for the ES cluster, I chose an IP-based policy and specified the source CIDR blocks allowed to connect:

 "Statement": [

    {

      "Effect": "Allow",

      "Principal": {

        "AWS": "*"

      },

      "Action": "es:*",

      "Resource": "arn:aws:es:us-west-2:accountID:domain/my-es-cluster/*",

      "Condition": {

        "IpAddress": {

          "aws:SourceIp": [

            "1.2.3.0/24",

            "4.5.6.7/32"

          ]

        }

      }

    }

5) Create subscription filters for streaming CloudWatch logs to ElasticSearch

First, make sure that the log files you configured with the AWS CloudWatch Log agent are indeed sent to CloudWatch. For each log file name, you should see a CloudWatch Log Group with that name, and inside the Log Group you should see multiple Log Streams, each Log Stream having the same name as the hostname sending those logs to CloudWatch.

I chose one of the Log Streams, went to Actions --> Stream to Amazon Elasticsearch Service, chose the ElasticSearch cluster created above, then created a new Lambda function to do the streaming. I had to create a new IAM role for the Lambda function. I created a role I called lambda-execution-role and associated with it the pre-existing IAM policy AWSLambdaBasicExecutionRole.

Once this Lambda function is created, subsequent log subscription filters for other Log Groups will reuse it for streaming to the same ES cluster.

One important note here is that you also need to allow the role lambda-execution-role to access the ES cluster. To do that, I modified the ES access policy and added a statement for the ARN of this role:

    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::accountID:role/lambda-execution-role"
      },
      "Action": "es:*",
      "Resource": "arn:aws:es:us-west-2:accountID:domain/my-es-cluster/*"
    }

6) Configure index pattern in Kibana

The last step is to configure Kibana to use the ElasticSearch index for the CloudWatch logs. If you look under Indices in the ElasticSearch dashboard, you should see indices of the form cwl-2017.11.24. In Kibana, add an Index Pattern of the form cwl-*. It should recognize the @timestamp field as the timestamp for the log entries, and create the Index Pattern correctly.

Now if you go to the Discover screen in Kibana, you should be able to visualize and search your log entries streamed from CloudWatch.