“Astro takes advantage of new Kubernetes-native patterns to help engineers declaratively manage Datadog monitors with the agility to match the ephemeral nature of these environments. We're excited to partner with Fairwinds and support their efforts in making this tool available to benefit the entire community."
— Michael Gerstenhaber, Director of Product Management at Datadog
In a Kubernetes environment, one of the challenges we see is accurate monitoring management that reflects the state of clusters and workloads. Monitoring is often an afterthought: as workloads change, monitoring updates are seldom made. Existing tools rely on manually configuring monitoring state, which introduces toil to SRE teams. It also increases availability and performance risks because monitors may not be present or accurate enough to trigger changes in KPIs (Key Performance Indicators). This can result in SLA breaches due to undetected issues and/or noisy pagers due to incorrect monitor settings.
To tackle this problem, Fairwinds has introduced a new open source software project called Astro. Astro is a Kubernetes operator that watches objects in your cluster for defined patterns and manages Datadog monitors based on this state.
Astro provides three key elements to greatly simplify Datadog monitor management:
-
Automated lifecycle management of Datadog monitors: Given configuration parameters, the utility will automatically manage defined monitors for all relevant objects within the Kubernetes cluster. As objects change, monitors are updated to reflect that state, eliminating the need for manual configuration.
-
Correlation between logically bound objects: Astro has the ability to manage monitors for all objects within a given namespace. This ensures greater consistency across monitor configurations.
-
Templating of values from Kubernetes objects into managed monitors: Any data from a managed Kubernetes object can be inserted into a managed monitor. This produces more informative and contextualized alerts, making it easier to triage issues.
Getting Started
You can run Astro in your cluster with 2 simple steps: configuring your desired monitors, and deploying the helm chart.
Monitors are configured via 1 or more yaml configuration files. These files can be a local file path or remotely accessed via URL. These configuration files are reloaded periodically so any changes you make will eventually propagate to your monitors. Here’s an example configuration:
In this example, a monitor would be created for any Kubernetes deployments that have the annotations listed in match_annotations (in this case, astro/owner: astro). This particular monitor will alert you when a deployment has no pods available. Astro uses go templating and you can insert any variables from the Kubernetes object or from the cluster_variables section of your config file into monitors. Specific details about the config file syntax is available in Astro’s readme.
Once your config file is in place, you’re ready to deploy with helm! The helm chart is available on GitHub. If you’re installing with Reckoner, a tool to declaratively install and manage multiple Helm chart releases, here’s a snippet to install Astro:
In case you haven’t tried Reckoner, it’s available on GitHub.
Enjoy Astro
Now, every time a deployment is created with the astro/owner annotation, Astro will automatically create a Datadog monitor for it, and we’ll get an alert if it goes down. Without Astro, we would have had to rely on the engineering team to manually create those monitors, which is time consuming and error-prone.
Astro is a great open source project that can be used to keep your monitors in sync, prevent outages due to misconfigured monitors, and fight pager fatigue. We hope you enjoy automated management of your Datadog monitors!