Instrumentation is a critical part of any application. Along with system counters like cpu, heap, free disk, etc--
it's important to create application-level metrics to make sure health is measured closer to your customer's experience.

Example metrics could be user-registration, password-change, profile-change,
etc. If you see a major spike or dip in these metrics, a wider problem could
be indicated.

For this example a custom metric was needed, and no infrastructure was in
place for harvesting it (e.g. collectd). Golang is handy for creating an
easy-to-install daemon which performs the measurement and periodically
harvests the data into stackdriver.

The overall flow for custom metrics in stackdriver goes like this:

1. Create a custom metric with the stackdriver Metric Descriptor API -- this assigns a data type, dimensions, name & resource category to the metric.
2. Push metric values to the metric, recording it's value, resources and additional dimensions.

In the following example, we wanted to measure the number of files being
uploaded by a backup agent. The number of files was indicated by lines in a
logfile, which we used as our indicator (see Line Counter and WatchFile).
By sending the line count to stackdriver every 5 s, we were able to create a
dashboard and alert monitoring backup activity.

1. Create the Metric Descriptor

For your first metric, it's easiest to use the API explorer which will auto complete the definition. As the project matures, call mectricDescriptors.create via gcloud

1. Visit Explorer
2. name: backup-count
3. valueType: INTEGER
4. remove labels. We will assign a relevant resource using the writeMetric call

Read the complete article here...