© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.

Container logging: Best practices for consistency

Carmen Puccio

G P S T E C 3 2 9

Principal Solutions Architect

AWS Partner Program

Shashi Prabhakar

Manager, Partner Solutions Architects, Modernization & Linux

AWS Partner Program

Before we begin, let’s categorize the logs

Typically developers are concerned with this because these logs give insight into application performance

• Example application: /var/log/myapp.log

• Web logs: journalctl -u nginx.service --since today

• Transaction logs

• Others

Typically operations teams are concerned with this because these logs give insight into

the stability of the platform and the availability of services

• Syslog and other OS logs

• Audit logs

• Performance metrics

It’s all over the place

# Apache access log

127.0.0.1 - - [28/Jul/2019:10:22:04 -0300] "GET / HTTP/1.0" 200 2216

# Apache error log

127.0.0.1 - - [28/Jul/2019:10:27:32 -0300] "GET /hidden/ HTTP/1.0" 404 7218[Thu Jun 22 11:35:48 2019] [notice] caught SIGTERM, shutting down

# typical expressjs log output

[Mon, 21 Nov 2019 20:52:11 GMT] 200 GET /foo (1ms)

# /var/log/secure

Sep 27 16:48:13 ip-192-168-2-203 sshd[6181]: Invalid user super from 34.80.19.95 port 53100

# .NET core

Unhandled Exception: System.NullReferenceException: Object reference not set to an instance of an object.

at error.Program.Main(String[] args) in /Users/puccio/Desktop/myApp/Program.cs:line 11

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.

But you said you want to run containers?

What about logging?

all the things!

Log

What about logging? In reality

all the things . . .

almost

Log

Application logging considerations• Implement a standard and easy-to-configure framework: Log4j2 / Log4net, NLog, Winston

• Log to STDOUT/STDERR

• If files are a must, think about the log names: mylog.yyyy-MM-dd HH:mm:ss.SSS.log

• Use logrotate or configure the docker-daemon.json

• {"log-driver":"json-file","log-opts":{"max-size":"10m","max-file":"3"}}

• Use log severity levels: DEBUG, INFO, WARN, and ERROR

• Use a standard structure such as JSON

• Custom formats and raw text logs need custom parsing rules in order to extract data for analysis

• Multiline messages are not your friend; dump them to JSON, even in this case

• This also applies to the schema for the fields. Use consistent names and label your services.

• {"time":"2019-11-04 12:34:56.0000","level":"ERROR","message":"HelloWorld","service":"foo","event":{"error":{"name":"RuntimeError","message":"MissingClass is undefined"}}}

• {"time":"2019-11-04 12:35:56.0000","level":"ERROR","message":"HelloWorld","service":"bar","event":{"error":{"exception":"RuntimeError","message":"MissingClass is undefined"}}}

• Use trace ID, for example, X-Amzn-Trace-Id for HTTP headers via the AWS X-Ray SDK

• X-Amzn-Trace-Id: Root=1-5759e988-bd862e3fe1be46a994272793;Sampled=1

What just happened?

Frontend

service

AWS Fargate

Backend service

v2

AWS Fargate

Backend service

v1

AWS Fargate

Error

Logging challenges with containers

• No permanent storage• Containers are not pets. If your container dies,

your logs will most likely be gone too.

• No fixed location from a network

address standpoint• You should not care about the IP addresses of

your containers

• No fixed location from a placement

perspective• Even though you can target nodes with things

such as label selectors, it’s all up to the

scheduler

Log collection modes: Sidecar

app-container Sidecar

Core functionality Peripheral tasks

such as

• Logging

• Proxy

• Configuration

• Others

Instance

AWS Fargate

Amazon Elastic

Kubernetes Service

(Amazon EKS)

Amazon Elastic Container

Service (Amazon ECS)

Log collection modes: DaemonSet

app-container

DaemonSet

Core functionality

Peripheral tasks

such as:

• Logging

• Node monitoring

• Others

Instance

app-container

DaemonSet

Core functionality

Peripheral tasks

such as:

• Logging

• Node monitoring

• Others

Instance

AWS Fargate Amazon EKSAmazon ECS

DaemonSet versus SidecarWho is using what for logging?

DaemonSet Sidecar

Collecting logs Files Stdout & stderr/files

Deployment Standard manifest/deployment Standard manifest/deployment

Multitenant isolation Average? Strong?

ScaleRequests/limits

Scheduling conflicts?

Can lead to significant resource

consumption (requests/limits)

Resource usageLow (each node runs a

container)High (each pod runs a container)

Ability to customizeLow (1 DaemonSet covering all

nodes per node)

High (each pod is configured

separately)

Cluster size? Clusters with few functions? Large clusters? PaaS? SaaS?

Anything else? ? ?

How to consolidate? Enter Fluent Bit

Amazon Elastic Container

Registry (Amazon ECR)

Amazon EKS

Amazon ECS

Fluent Bit

DaemonSet

Amazon Simple Storage

Service (Amazon S3)

Amazon Kinesis

Data FirehoseAmazon Athena

Fluent Bit

DaemonSet

app-container

app-container

Related breakouts

CON309 FireLens

ANT312 Amazon Elasticsearch Service: Everything ingest

ANT332 Use Amazon ES to visualize and monitor containerized applications

Thank you!

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.

Carmen Puccio

@CarmenAPuccio

Shashi Prabhakar

@shashikantprabhakar

Please complete the session survey in the mobile app.

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.

Let’s start with the basics

$: docker run --name test -d busybox sh -c "while true; do $(echo date); sleep 1; done”

$: docker ps -aCONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES

3bedb2d04f06 busybox "sh -c 'while true; …" 23 seconds ago Up 22 seconds test

$: docker logs test

Tue Sep 24 13:56:52 UTC 2019

Tue Sep 24 13:56:53 UTC 2019

Tue Sep 24 13:56:54 UTC 2019

Tue Sep 24 13:56:55 UTC 2019

Tue Sep 24 13:56:56 UTC 2019

But what just happened?$: docker logs test

Tue Sep 24 13:56:52 UTC 2019

Tue Sep 24 13:56:53 UTC 2019

Tue Sep 24 13:56:54 UTC 2019

Tue Sep 24 13:56:55 UTC 2019

Tue Sep 24 13:56:56 UTC 2019

• Docker supports different logging drivers used to store and/or stream container stdout and stderr logs of the main container process (pid 1)

• The easiest and most used logging method for containerized applications is to write to the stdout and stderr streams

• By default, the Docker daemon uses the json-file logging driver

$: docker info --format '{{.LoggingDriver}}'

json-file

• When you start a container, you can configure it to use a different logging driver than the Docker daemon’s default, using the --log-driver flag

$: docker run --log-driver none --name test2 -d busybox sh -c "while true; do $(echo date); sleep 1; done”

$: docker inspect -f '{{.HostConfig.LogConfig.Type}}' test2

none

Logging driversDriver Description

none No logs are available for the container, and docker logs do not return any output

local Logs are stored in a custom format designed for minimal overhead

json-file The logs are formatted as JSON, the default logging driver for Docker

syslog Writes logging messages to the syslog facility; the syslog daemon must be running on the host machine

journald Writes log messages to journald; the journald daemon must be running on the host machine

gelf Writes log messages to a Graylog extended log format (GELF) endpoint such as Graylog or Logstash

fluentd Writes log messages to fluentd (forward input); the fluentd daemon must be running on the host machine

awslogs Writes log messages to Amazon CloudWatch Logs

splunk Writes log messages to splunk using the HTTP Event Collector

etwlogs Writes log messages as event tracing for Windows (ETW) events (only available on Windows platforms)

gcplogs Writes log messages to Google cloud platform (GCP) logging

logentries Writes log messages to Rapid7 Logentries

The awslogs log driver

$: docker run --log-driver=awslogs \

--log-opt awslogs-region=us-east-1 \

--log-opt awslogs-group=reinvent \

--log-opt awslogs-create-group=true \

--name test3 -d busybox sh -c "while true; do $(echo date); sleep 1; done"

Amazon ECS & AWS Fargate log configuration

"logConfiguration": {

logDriver": "awslogs","fluentd","gelf","json-file","journald","logentries","splunk","syslog","awsfirelens",

"options": {

"string": "string"

...

},

"secretOptions": [{

"name": "string",

"valueFrom": "string"

}]

}

AWS FargateAmazon ECS

/var/log/containers/POD-NAME_NAMESPACE_CONTAINER-NAME-CONTAINER-ID.log

What about Kubernetes?

• Logs should have a separate storage and lifecycle independent of nodes, pods, or containers

• This concept is called cluster-level logging

• Cluster-level logging requires a separate backend to store, analyze, and query logs

• Kubernetes provides no native storage solution for log data, but you can integrate many existing logging solutions into your Kubernetes cluster

stdout

stderr

app-container

my-pod

Instance

Deleted on pod eviction

Fluent Bit on Amazon EKS

/var/log/containers/POD-NAME_NAMESPACE_CONTAINER-NAME-CONTAINER-ID.log

stdout

stderr

app-container

my-pod

Instance

Fluent Bit

DaemonSet

Read logs Look up

metadata

Kubernetes API Server

Analyze the tag and extract

the following metadata:• Pod name

• Namespace

• Container name

• Container ID

Query Kubernetes API Server in order to obtain extra metadata for the POD in question• Pod ID

• Labels

• Annotations