Introducing keyload: Rotate TLS Certificates Without a Restart

Short-lived certificates are a good idea right up until you have to run them. cert-manager will happily renew a certificate every day, and the kubelet will happily put the new files in your pod. Then nothing happens, because your Go service read tls.crt once at startup and has been holding on to it ever since. A day later the certificate expires and every handshake fails.

The usual fixes all have a catch. A reloader sidecar that restarts the pod whenever the Secret changes works, but now a certificate renewal is a rolling restart. A GetCertificate callback that re-reads the file on each handshake works for the server’s own certificate, but it doesn’t help with the CA bundles, and with mTLS those rotate too. And the hand-rolled version that reloads on a timer will, sooner or later, read a key file halfway through a write and start serving garbage.

So I wrote keyload. It’s a Go library that watches your certificate, key and CA bundle files, validates them when they change, and swaps them in atomically. Servers and clients pick up the new material on their next handshake. If the new files are bad, the old ones keep working.

What You Get #

• Both directions. Your certificate and key, and the CA bundles you use to verify servers and clients. With mTLS, both sides follow a rotation.
• Bad material never goes live. An expired certificate, a key that doesn’t match, or a half-written file is rejected, and the last good one keeps serving.
• Kubernetes Secret volumes work out of the box, including the kubelet’s ..data symlink swap.
• CA rotation without an outage. A removed CA stays trusted for an overlap window you choose.
• crypto/tls still does the verification for servers, HTTP clients and gRPC. No InsecureSkipVerify.
• Prometheus metrics and reload events, so you hear about a bad certificate before it expires.
• No dependencies outside the standard library. gRPC support is its own module.

go get github.com/rossedman/keyload

Quick Start #

You point keyload at your files and ask it for a tls.Config. That’s the whole setup.

r, err := keyload.New(keyload.Files{
	Cert:      "/tls/tls.crt",
	Key:       "/tls/tls.key",
	Roots:     "/tls/ca.crt", // CAs to verify servers
	ClientCAs: "/tls/ca.crt", // CAs to verify clients
})
if err != nil {
	log.Fatal(err) // the first load must be valid
}
defer r.Close()

// Server: the certificate and client CAs follow reloads.
srv := &http.Server{TLSConfig: r.ServerConfig(&tls.Config{
	ClientAuth: tls.RequireAndVerifyClientCert,
})}

// Client: every new connection uses the current certificate and roots.
tr := &http.Transport{}
if err := r.ConfigureTransport(tr, nil); err != nil {
	log.Fatal(err)
}

The tls.Config you pass in is optional. It’s where your other settings go (ClientAuth, MinVersion, ServerName and so on), and keyload fills in the certificates and CAs from whatever is current.

Watching It Work #

The repo has a gencerts example that writes a throwaway CA and a certificate laid out like a cert-manager Secret, and a server example that serves mTLS with keyload. Start them up and make a request:

➜ go run ./examples/gencerts -dir /tmp/keyload
➜ go run ./examples/server -dir /tmp/keyload &
➜ curl --cacert /tmp/keyload/ca.crt --cert /tmp/keyload/tls.crt \
       --key /tmp/keyload/tls.key https://localhost:8443/
hello, leaf

Now rotate the certificate the way a renewal would, with a new one from the same CA:

➜ go run ./examples/gencerts -dir /tmp/keyload -rotate

The server picks it up within a second, without being told:

level=INFO msg="TLS material loaded" generation=1 trigger=initial expires=2026-10-15T00:44:32Z
level=INFO msg="TLS material loaded" generation=2 trigger=watch expires=2026-10-15T00:44:33Z

And now the interesting part. Break the key file on purpose:

➜ echo oops > /tmp/keyload/tls.key

level=ERROR msg="TLS reload rejected; still serving the previous certificate" generation=2 trigger=watch
  err="keyload: invalid material (key /tmp/keyload/tls.key): truncated or malformed PEM block"

The server is still on generation 2, and requests still work. Nobody gets paged because a file was bad for a few seconds. You still find out, though, which is the point of the next section.

What Gets Rejected #

Every reload is validated before it’s used. If any check fails, the current certificates stay in service and you get an event with the reason.

There’s one subtle case worth calling out. A lot of certificate managers write tls.crt and tls.key one at a time, so for a moment you can have a new certificate sitting next to the old key. That looks exactly like ErrKeyMismatch. Instead of raising a false alarm, keyload retries a half-written-looking reload 3 times with a short backoff before it gives up on it.

You can hook into every reload, good or bad:

r, err := keyload.New(files, keyload.OnReload(func(ev keyload.Event) {
	if ev.Err != nil {
		log.Printf("TLS reload rejected, still serving generation %d: %v", ev.Generation, ev.Err)
	}
}))

Events name the file and the reason on a rejection, and the subjects and serials that changed on success. They never include key material.

Rotating a CA #

Rotating a leaf certificate is the easy case. Rotating the CA is where outages come from. If you swap the CA bundle on the server before every client has a certificate from the new CA, the clients that haven’t rotated yet get locked out.

OverlapWindow handles this. A CA that’s removed from a bundle stays trusted for that long after it disappears:

r, err := keyload.New(files, keyload.OverlapWindow(24*time.Hour))

So the rotation becomes three boring steps. Add the new CA to the bundle. Move certificates over to it. Remove the old CA. Peers that haven’t rotated yet keep working until the window ends, and then the old CA is dropped on its own.

This also applies to session resumption. A resumed session’s client certificate is checked against the current client CAs, so once a CA is really gone, its clients can’t sneak back in on an old session ticket.

In Kubernetes #

This is the setup I built it for. cert-manager renews a Certificate into a Secret, the Secret is mounted into the pod, and keyload notices the change:

spec:
  containers:
    - name: api
      image: ghcr.io/example/api:1.0.0 # your app, using keyload
      volumeMounts:
        - name: tls
          mountPath: /tls
          readOnly: true
  volumes:
    - name: tls
      secret:
        secretName: api-tls

The kubelet doesn’t edit the files in place. It writes a new directory and swaps a ..data symlink to point at it. keyload watches the directories with inotify and handles that swap, so the change shows up within moments. No reloader sidecar and no restart.

On NFS, FUSE and other filesystems where inotify misses changes, and on macOS and Windows, keyload checks every 30 seconds instead. You can also reload on a signal or a timer:

r, err := keyload.New(files,
	keyload.OnSignal(syscall.SIGHUP),  // kill -HUP to reload now
	keyload.PollEvery(5*time.Minute),  // and on a timer
)

If your certificates don’t live in files at all, say they come from Vault or a secrets API, implement Source and return PEM. keyload.SourceFunc turns a plain function into one.

Knowing Before It Expires #

The worst certificate outage is the quiet one. Renewal broke three weeks ago, nobody noticed, and today the certificate expired. keyload exports metrics so that can’t happen quietly. keyload.MetricsHandler(r) serves Prometheus text format without pulling in a client library:

➜ curl -s localhost:9090/metrics | grep -v '^#'
keyload_generation{reloader="api"} 1
keyload_reload_total{reloader="api",result="success"} 1
keyload_reload_total{reloader="api",result="invalid"} 0
keyload_reload_total{reloader="api",result="read_error"} 0
keyload_cert_not_after_seconds{reloader="api",role="leaf"} 1.792025086e+09
keyload_cert_not_after_seconds{reloader="api",role="roots"} 1.823474686e+09
keyload_last_success_timestamp_seconds{reloader="api"} 1.791938686286792e+09

Two alerts cover almost everything:

- alert: CertificateExpiresSoon
  expr: keyload_cert_not_after_seconds - time() < 7 * 86400
- alert: CertificateReloadFailing
  expr: increase(keyload_reload_total{result!="success"}[15m]) > 0

The first catches renewal that stopped. The second catches someone pushing bad files. Pair that with MinRemaining(time.Hour) and keyload won’t even accept a certificate that’s about to expire, so you get the rejection and the alert instead of a surprise.

HTTP, gRPC and Everything Else #

Different libraries want TLS handed to them in different ways, so there are a few entry points:

gRPC lives in its own module, github.com/rossedman/keyload/grpccreds, so the core library doesn’t drag grpc-go into projects that don’t use it:

creds := grpccreds.New(r, grpccreds.WithTLSConfig(&tls.Config{
	ClientAuth: tls.RequireAndVerifyClientCert,
}))
srv := grpc.NewServer(grpc.Creds(creds))
conn, err := grpc.NewClient(addr, grpc.WithTransportCredentials(grpccreds.New(r)))

That last row of the table deserves a little honesty. Some libraries only take a static *tls.Config on the client side, and there’s no hook in crypto/tls to swap the trusted roots on one of those. So in “Config mode” keyload sets InsecureSkipVerify and does the verification itself in VerifyConnection, using the same options crypto/tls would. I don’t love writing that sentence, which is why it’s the most heavily tested part of the library. CI generates 2,000 chains (wrong host, IP SANs, expired intermediates, wrong key usage, name constraints), verifies each one with both crypto/tls and Config mode, and fails the build on any disagreement. Use the other modes when you can. Use this one when a library leaves you no choice.

Is It Fast? #

It sits in the handshake path, so it has to stay out of the way. ServerConfig hands each handshake a config that was built once per reload, so the hot path takes no locks and makes no allocations.

Benchmarks run on every push, and a test fails the build if the handshake path ever starts allocating.

The tests I care about more are the ones that look like real life. CI rotates certificates and CAs on both sides in the middle of traffic over HTTP/1.1, HTTP/2 and gRPC, and allows zero failed requests. It runs every kind of bad file through a reload and checks that the old certificates stay and the right reason comes back. It reproduces the kubelet’s ..data symlink swap on a real filesystem. And it fuzzes the PEM parsing.

Try It #

Every example is a small runnable program with the commands to try it at the top of the file:

git clone https://github.com/rossedman/keyload
cd keyload
go run ./examples/gencerts -dir /tmp/keyload
go run ./examples/server -dir /tmp/keyload

There’s also a docs site with an interactive reload simulator, where you can push good and bad files at a reloader and watch what it does without running anything.

This is the first release, v0.1.0, and it’s MIT licensed. Until 1.0 the API may still change. I’ve written before about reloading on SIGHUP, and this is really the same idea applied to the one piece of config that will take you down on a schedule if you don’t reload it. If you try it and a rotation surprises you, or there’s a library it doesn’t plug into yet, open an issue on GitHub. I’d love to hear about it.