Push Custom Metrics to Datadog using CloudWright

January 14, 2020

Every business has metrics worth monitoring.

Some companies track active users. Some track file uploads. Others track product engagement. These metrics provide real-time insight into the health of a product — just like infrastructure metrics track the health of a machine.

Datadog is a popular monitoring and alerting service — we use and love it at CloudWright. In addition to hardware monitoring, Datadog is a great platform for capturing and tracking custom business metrics — active users, uploaded files, and delivery backlogs.

Pushing business-logic metrics to Datadog can be a hassle, because it requires both dedicated infrastructure and authentication to internal APIs and databases. Luckily, quickly combining these resources is the type of problem which CloudWright makes easy.

In this post, we'll use CloudWright to extract data from an (example) internal company database and push the metric to Datadog for monitoring and alerting.

Tracking Active Website Users

"Active Website Users" is a valuable product metric, often measured as "users who have logged in within the past week." Internally, this data is accessible in our MySQL database as last_login:

mysql> describe customers;
+------------+--------------+------+-----+---------+-------+
| Field      | Type         | Null | Key | Default | Extra |
+------------+--------------+------+-----+---------+-------+
| name       | varchar(225) | YES  |     | NULL    |       |
| last_login | date         | YES  |     | NULL    |       |
+------------+--------------+------+-----+---------+-------+

We'll set up our CloudWright application with two Modules — our Datadog integration and our MySQL database (for more details about how to create and use CloudWright Modules, check out the Modules section in our docs).

Setting up the Datadog and MySQL Modules

We'll show how to these modules up from scratch, but remember — you'll usually be able to re-use modules your team has already created.

Setting up the Datadog Module is easy — you'll just need your Datadog API key and Application keys (see the Datadog docs for more details):

Datadog

MySQL needs a hostname, username, database, and password. We'll connect to this database directly — if your database is behind a firewall, you'll need to set up a Deployment Zone with access to your resouces:

MySQL

Once those these Modules are created, we can move on to writing our Application.


Once our Modules are attached to a new Application, querying the database and forwarding the data to Datadog is just a few lines of Python:

import sqlalchemy as sql
import datetime as DT

mysql = CloudWright.get_module("mysql")
datadog = CloudWright.get_module("datadog")

# Query for users active within the past 7 days
WEEK_AGO = DT.date.today() - DT.timedelta(days=7)

# The name of our datadog metric
METRIC = "cloudwright.example.active.users"

# Query for count of users who have logged in since cutoff, get count
q = sql.text("SELECT COUNT(*) from customers WHERE last_login > :cutoff")
count = mysql.execute(q, cutoff=WEEK_AGO).first()[0]

# Pipe our metric into datadog
datadog.Metric.send(metric = METRIC, points = count)

# For debugging — print the count we just wrote
CloudWright.response.set_value({'count_written': count})

We can use the CloudWright dry-run console to test that our script is pushing the counts we expect:

Dry Run

and we can confirm that Datadog has indeed picked up our custom metric:

Metric

Last but not least, we'll set up a trigger to run this script on a schedule. CloudWright cron triggers can run on any schedule we want — this application we'll choose to run daily:

Metric

Once we publish our application, we're live — that's really all it takes to push internal metrics to Datadog using CloudWright.

If you'd like to learn more, see a live demo, or start using CloudWright to improve your own metric monitoring story, let's get in touch.