DZHG  / Introducing go-awslogs

At my current job (Arena Solutions), we have 2 production environments. One of them is in AWS.

Migrating a legacy application to AWS is not an easy job. I was the tech lead of that migration project. It was a time consuming and frustrating experience. Luckily, we eventually conquered most of the challenges and now we have the AWS production instance running smoothly.

After the migration, another challenge fell on us. We had to push the entire engineering team to adopt the public cloud infrastructure and deployment model. The very usual complain I heard from the team was: “How can I easily get the logs for our services?”

I used to gave a response like: “there is an open source tool called awslogs. It allows you to tail the AWS CloudWatch log streams”.

You might think it would end like that. But it didn’t. I got various follow-up questions, like below:

I don’t have python installed. How can I use it? (Yes, we do have developers using non-Unix-like systems without python)

I use a remote server as my main working station. It does not have python3 installed. How can I use it?

I have python3 and awslogs installed but it fails with some error. Can you help me?

What’s the date time formats I can use with awslogs?

Don’t get me wrong. awslogs is a great tool. It helped me a lot in the migration project. However, it has a few issues when trying to get broader adoption across different teams:

  • It requires Python3 (not easy to adopt when someone is using Windows without Python3)
  • In some cases, it fails to run if LD_LIBRARY_PATH is modified
  • Lacks of clear documentation about date time formats

The first 2 issues are essentially caused by Python. There is no workaround.

go is a perfect solution for the issues above. go binaries are statically linked. It means the compiled go binaries can run without any 3rd party dependencies.

So, now, I’m introducing go-awslogs. This project is inspired by awslogs. go-awslogs is still in early stage but it covers most if not all features of original awslogs.

For example, you can retrieve all log groups from AWS CloudWatch:

go-awslogs groups

And, retrieve all log streams of a group:

go-awslogs streams my-log-group

Fetch logs by starting and ending time:

go-awslogs get my-log-group stream-1 -s 2020-07-23T08:23:00-07:00 -e 2020-07-25T12:15:00-07:00

Watch the log stream (tail -f)

go-awslogs get my-log-group stream-1 --watch

The README has more detailed usages.

go-awslogs supports 3 kinds of time formats when specifying the start or end flags:

  • Relative: like 5min ago, 1.5hours ago, etc.
  • RFC3339: 2020-07-23T15:23:35Z or 2020-07-23T08:23:35-07:00
  • Human: 01/02/2006 15:04:05 -07:00

All timezone must be defined by offset due to go timezone handling inside of standard library. I talked about it in the other post.

As a single binary after compile, go-awslogs can be installed effortlessly.

I hope I will get fewer questions about how to access AWS CloudWatch logs going forward.