Logs are great. They are easy when you’re running on one or two boxes. However, they get harder when you’re running in a modest distributed environment and even more so if you’ve embraced micro-services and containers. I’ve compiled some notes and thoughts on getting Logentries running in a Kubernetes environment.
At Fairwinds, we use Kubernetes for our clients. Centralized logs are critical for them to understand their environments and for us to assist in troubleshooting. Sometimes, a client will have existing logging tools or services they would like us to use. This is the case with my current client, who utilizes Logentries.
Getting it running in Kubernetes was not very difficult, but some things were learned. Since Kubernetes is all about containers, it makes sense that a log collector for Logentries should also run as a container. The folks at Logentries clearly feel the same way and have made the logentries/docker-logentries image available on docker hub. The instructions discuss how to run in a container, but it is limited to straight docker environment. Kubernetes is based on docker, but you do run things differently.
I run the Logentries collector as a DaemonSet in its own ServiceAccount under the kube-system namespace. The reason for running it as DaemonSet is to ensure we run one of these containers on every node. A dedicated ServiceAccount isn’t necessary, but helps to isolate things. You can also use a different namespace if you’d rather not add to kube-system, but it fits pretty well since this is for logging everything.
Logentries allows you to send logs to different locations named Log Sets. In order to get logs to show up in Log Sets, you will need to create the Log Set and then an access token. Since this is a secret, we’ll store it as such. Here is an example secret.yml to create the logentries_token in Kubernetes:
---
apiVersion: v1
data:
logentries_token: <your token | base64 -w0>
kind: Secret
metadata:
name: logentries
namespace: kube-system
Apply this with kubectl apply -f secret.yml. Next we will create the the ServiceAccount. The serviceaccount.yml looks like this:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: logentries
namespace: kube-system
Apply this with kubectl apply -f serviceaccount.yml. Now we are ready for the creation of the actual work horses. The daemonset.yml looks like this:
---
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
name: logentries
namespace: kube-system
spec:
selector:
matchLabels:
app: logentries
template:
metadata:
name: logentries
labels:
app: logentries
spec:
serviceAccount: logentries
containers:
- name: logentries
image: logentries/docker-logentries
imagePullPolicy: Always
securityContext:
privileged: true
args:
- "-t $LOGENTRIES_TOKEN"
- "-j"
# - "--no-stats"
# - "--no-logs"
# - "--no-dockerEvents"
# - "-a <a special string>"
# - "--stats-interval <STATSINTERVAL>
# - "--matchByName REGEXP"
# - "--matchByImage REGEXP"
# - "--skipByName REGEXP"
# - "--skipByImage REGEXP"
volumeMounts:
- name: socket
mountPath: /var/run/docker.sock
env:
- name: LOGENTRIES_TOKEN
valueFrom:
secretKeyRef:
name: logentries
key: logentries_token
volumes:
- name: socket
hostPath:
path: /var/run/docker.sock
Which you apply with kubectl apply -f daemonset.yml.
The above should work if you have a standard type setup, similar to the one you get from kops. If you’ve done custom installation work, you may need to adjust some things, like the location of the socket etc.
The above is the default configuration which will send container logs, container statistics, and docker events to Logentries. Should you wish to alter the logging that happens via the flags, I’ve added them in commented fashion. Details on those flags can be found on docker hub: logentries/docker-logentries/.
In general, it works as advertised. Once the DaemonSet is running, the logs will flow. Memory utilization of each container or Pod is under 100MB and CPU consumption has been modest thus far.
The logs produced by the containers are in JSON form and the -j switch makes sure that things pass through properly. The JSON that arrives appears clean and fully functional.
The first thing I noticed was that it can be tricky to find the logs for a single Deployment. For example, depending on how your images or containers are named, you will likely need to use regular expressions. The simple search will not find substrings. If we have a name of pod-staging_nginx-worker searching for nginx, it will not return results, but name=/.*nginx.*/ will.
Now that we have logs for only the pod or container we’re interested in, there is the issue of all the stats that are also part of the found results. To reduce the results to only logs from the container itself, you can add another clause: AND line. Every line logged will be stored in a JSON attribute named line, thus forcing the search to also find only those items gets us what we need: name=/.*nginx.*/ AND line.
Another issue still in search of an answer is the use case where you’d like a Log Set for application X and application Y. With a classic Logentries collector, which accesses the log files directly, you can configure which Log Set the logs from a given file will go. In a Kubernetes environment, where the logs are harvested via a docker API, that is not possible. Unfortunately, Logentries does not offer a way to slice and tag once the logs are received from docker. One possible option would be to run a Logentries container alongside each pod and utilize --matchByName or --matchByImage and --no-stats and --no-dockerEvents. Clearly, that will lead to a large number logging containers. I will be investigating this a bit more, but the default is likely the only reasonable approach for now.
In conclusion, Logentries is perfectly usable and easy to set up with Kubernetes.
