Labels, objects & strings¶
A family often has many values: one per result, per worker, per interface. systemd's report framework gives each value two kinds of context, and the provider exposes both.
| Context | On the wire | In the provider | In OpenTelemetry |
|---|---|---|---|
| Fields: dimensions that tell values apart | "fields":{"result":"ok"} |
Labels(...) and With(...) |
point attributes |
| Object: the thing a value is about | "object":"worker-a" |
On(...) |
systemd.object attribute |
The List blocks on this page use a compact notation for the replies:
name{object}{"field":"value"} value.
Labels¶
Declare label names at registration, then pick a series with With, one
value per label, in order:
reqs := p.Counter("Requests", provider.Help("HTTP requests"), provider.Labels("method", "code"))
reqs.With("GET", "200").Inc()
reqs.With("POST", "500").Inc()
io.example.Api.Requests{"code":"200","method":"GET"} 1
io.example.Api.Requests{"code":"500","method":"POST"} 1
Label names follow the metric naming rules: a letter, then letters, digits
and single underscores. Declaring an invalid or duplicate name panics, and so
does calling With with the wrong number of values.
Keep handles
With finds the series without a lock or an allocation, but it still
hashes the values. On hot paths, look the series up once:
Label values are unbounded strings. As with any metrics system, keep their cardinality low; the scraper keeps at most 16,384 values per source and counts the rest as dropped.
Objects¶
On(object) returns the series about one thing, such as a unit, a device or
a worker. systemd uses objects heavily: io.systemd.Manager.NRestarts has one
value per unit, with the unit's name as the object.
restarts := p.Counter("WorkerRestarts", provider.Help("Per worker: restarts"))
restarts.On("worker-a").Inc()
restarts.On("worker-b").Add(3)
With and On compose, in either order: With keeps the receiver's object
and On keeps its label values.
errs := p.Counter("WorkerErrors", provider.Labels("kind"))
errs.On("worker-a").With("timeout").Inc()
errs.With("timeout").On("worker-b").Inc()
A series appears once it is written. Intermediate handles, such as
errs.On("worker-a") before .With(...), are never published.
Field or object?
Use the object when the value is about something with its own identity, especially a systemd unit, and there is one natural subject per value. Use fields for dimensions you would group or sum by, such as a result code. A value can have both.
String metrics¶
A string metric publishes a string value, such as a state or a version. Series without a value are not published.
state := p.String("WorkerState", provider.Help("Per worker: state"))
ws := state.On("worker-a")
ws.Set("idle")
// later
ws.Set("busy")
In OpenTelemetry a string becomes an info-style metric: a Gauge with value
1 and the string in the systemd.value attribute. In Prometheus that is
Set allocates (it stores a new string); numeric updates do not.
Deleting series¶
Delete stops publishing a series, for example when the worker or unit it
describes goes away:
Using the handle afterwards does not bring the series back; call With or
On again for a fresh one. On the root series of a family (no labels, no
object), Delete resets the value and hides it until it is next written.
A complete example¶
examples/queued is the provider SDK's sample daemon. It uses every feature
on this page, plus callback metrics and socket activation:
// Command queued is the sample daemon for the provider SDK: a toy job
// queue that publishes its metrics through systemd's report framework.
//
// Run it directly (it binds /run/systemd/report/io.example.Queue, so it
// needs write access there; use -dir to pick another directory):
//
// go run ./examples/queued -dir /tmp/report
// go run ./cmd/sdmetrics-exporter -once -dir /tmp/report
//
// Or install it with the unit files next to this file, socket-activated,
// and read it with systemd-report:
//
// sudo install -m755 queued /usr/local/bin/
// sudo cp queued.socket queued.service /etc/systemd/system/
// sudo systemctl enable --now queued.socket
// systemd-report metrics io.example.Queue
package main
import (
"context"
"flag"
"log"
"math/rand/v2" // nosemgrep: go.lang.security.audit.crypto.math_random.math-random-used
"os"
"os/signal"
"runtime"
"syscall"
"time"
"github.com/rossedman/sdmetrics/provider"
)
func main() {
dir := flag.String("dir", provider.DefaultDir, "report directory, when not socket-activated")
workers := flag.Int("workers", 3, "worker goroutines")
flag.Parse()
p, err := provider.New("io.example.Queue",
provider.WithDir(*dir),
provider.FromActivation(), // adopt queued.socket when started by it
provider.WithSocketMode(0o666), // like systemd's own sources: readable by any local user
)
if err != nil {
log.Fatal(err)
}
depth := p.Gauge("Depth", provider.Help("Jobs waiting"))
jobs := p.Counter("JobsProcessed", provider.Help("Jobs finished, by result"), provider.Labels("result"))
busy := p.Gauge("WorkersBusy", provider.Help("Workers running a job"))
state := p.String("WorkerState", provider.Help("Per worker metric: state"))
p.GaugeFunc("Goroutines", func() float64 { return float64(runtime.NumGoroutine()) }, provider.Help("Goroutines in the daemon"))
p.StringFunc("GoVersion", runtime.Version, provider.Help("Go release the daemon was built with"))
ok, failed := jobs.With("ok"), jobs.With("failed") // keep handles on hot paths
ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM)
defer stop()
queue := make(chan time.Duration, 1000)
go func() { // producer
for ctx.Err() == nil {
select {
case queue <- time.Duration(rand.IntN(300)) * time.Millisecond:
depth.Set(float64(len(queue)))
default:
}
time.Sleep(time.Duration(rand.IntN(200)) * time.Millisecond)
}
}()
for w := 0; w < *workers; w++ {
name := "worker-" + string(rune('a'+w))
ws := state.On(name)
ws.Set("idle")
go func() {
for {
select {
case <-ctx.Done():
return
case job := <-queue:
depth.Set(float64(len(queue)))
busy.Inc()
ws.Set("busy")
time.Sleep(job)
if rand.IntN(10) == 0 {
failed.Inc()
} else {
ok.Inc()
}
busy.Dec()
ws.Set("idle")
}
}
}()
}
log.Printf("queued: publishing %s", p.Name())
if err := p.Serve(ctx); err != nil {
log.Fatal(err)
}
}
Run it and read it back: