# Introducing keyload: Rotate TLS Certificates Without a Restart > keyload is a small Go library that reloads TLS certificates, keys and CA bundles while a server or client keeps running, and refuses to swap in anything that's broken. - Author: Ross Edman - Published: 2026-10-13 - Tags: go, kubernetes - URL: https://rossedman.io/blog/computers/introducing-keyload/ **TL;DR:** `keyload` reloads your TLS certificate, key and CA bundles from disk while your Go server or client keeps running. When cert-manager rotates a Secret, the next handshake uses the new files. When the new files are bad (expired, mismatched key, half-written), it keeps serving the last good ones and tells you why. Standard library only, with a separate module for gRPC. Short-lived certificates are a good idea right up until you have to run them. cert-manager will happily renew a certificate every day, and the kubelet will happily put the new files in your pod. Then nothing happens, because your Go service read `tls.crt` once at startup and has been holding on to it ever since. A day later the certificate expires and every handshake fails. The usual fixes all have a catch. A reloader sidecar that restarts the pod whenever the Secret changes works, but now a certificate renewal is a rolling restart. A `GetCertificate` callback that re-reads the file on each handshake works for the server's own certificate, but it doesn't help with the CA bundles, and with mTLS those rotate too. And the hand-rolled version that reloads on a timer will, sooner or later, read a key file halfway through a write and start serving garbage. So I wrote [keyload](https://github.com/rossedman/keyload). It's a Go library that watches your certificate, key and CA bundle files, validates them when they change, and swaps them in atomically. Servers and clients pick up the new material on their next handshake. If the new files are bad, the old ones keep working. --- ## What You Get - **Both directions.** Your certificate and key, *and* the CA bundles you use to verify servers and clients. With mTLS, both sides follow a rotation. - **Bad material never goes live.** An expired certificate, a key that doesn't match, or a half-written file is rejected, and the last good one keeps serving. - **Kubernetes Secret volumes work out of the box**, including the kubelet's `..data` symlink swap. - **CA rotation without an outage.** A removed CA stays trusted for an overlap window you choose. - **crypto/tls still does the verification** for servers, HTTP clients and gRPC. No `InsecureSkipVerify`. - **Prometheus metrics and reload events**, so you hear about a bad certificate before it expires. - **No dependencies** outside the standard library. gRPC support is its own module. ```shell go get github.com/rossedman/keyload ``` --- ## Quick Start You point keyload at your files and ask it for a `tls.Config`. That's the whole setup. ```go r, err := keyload.New(keyload.Files{ Cert: "/tls/tls.crt", Key: "/tls/tls.key", Roots: "/tls/ca.crt", // CAs to verify servers ClientCAs: "/tls/ca.crt", // CAs to verify clients }) if err != nil { log.Fatal(err) // the first load must be valid } defer r.Close() // Server: the certificate and client CAs follow reloads. srv := &http.Server{TLSConfig: r.ServerConfig(&tls.Config{ ClientAuth: tls.RequireAndVerifyClientCert, })} // Client: every new connection uses the current certificate and roots. tr := &http.Transport{} if err := r.ConfigureTransport(tr, nil); err != nil { log.Fatal(err) } ``` The `tls.Config` you pass in is optional. It's where your other settings go (`ClientAuth`, `MinVersion`, `ServerName` and so on), and keyload fills in the certificates and CAs from whatever is current. --- ## Watching It Work The repo has a `gencerts` example that writes a throwaway CA and a certificate laid out like a cert-manager Secret, and a `server` example that serves mTLS with keyload. Start them up and make a request: ```shell ➜ go run ./examples/gencerts -dir /tmp/keyload ➜ go run ./examples/server -dir /tmp/keyload & ➜ curl --cacert /tmp/keyload/ca.crt --cert /tmp/keyload/tls.crt \ --key /tmp/keyload/tls.key https://localhost:8443/ hello, leaf ``` Now rotate the certificate the way a renewal would, with a new one from the same CA: ```shell ➜ go run ./examples/gencerts -dir /tmp/keyload -rotate ``` The server picks it up within a second, without being told: ```text level=INFO msg="TLS material loaded" generation=1 trigger=initial expires=2026-10-15T00:44:32Z level=INFO msg="TLS material loaded" generation=2 trigger=watch expires=2026-10-15T00:44:33Z ``` And now the interesting part. Break the key file on purpose: ```shell ➜ echo oops > /tmp/keyload/tls.key ``` ```text level=ERROR msg="TLS reload rejected; still serving the previous certificate" generation=2 trigger=watch err="keyload: invalid material (key /tmp/keyload/tls.key): truncated or malformed PEM block" ``` The server is still on generation 2, and requests still work. Nobody gets paged because a file was bad for a few seconds. You still find out, though, which is the point of the next section. --- ## What Gets Rejected Every reload is validated before it's used. If any check fails, the current certificates stay in service and you get an event with the reason. | Check | Error | |---|---| | The key matches the certificate | `ErrKeyMismatch` | | Every certificate in the chain is inside its validity window | `ErrExpired`, `ErrNotYetValid` | | At least `MinRemaining(d)` is left before the chain expires | `ErrExpiresTooSoon` | | The certificate chains to the new roots (with `RequireChain()`) | `ErrChainVerification` | | Each CA bundle has at least one certificate | `ErrEmptyBundle` | | No truncated PEM | `ErrTruncatedPEM` | There's one subtle case worth calling out. A lot of certificate managers write `tls.crt` and `tls.key` one at a time, so for a moment you can have a new certificate sitting next to the old key. That looks exactly like `ErrKeyMismatch`. Instead of raising a false alarm, keyload retries a half-written-looking reload 3 times with a short backoff before it gives up on it. You can hook into every reload, good or bad: ```go r, err := keyload.New(files, keyload.OnReload(func(ev keyload.Event) { if ev.Err != nil { log.Printf("TLS reload rejected, still serving generation %d: %v", ev.Generation, ev.Err) } })) ``` Events name the file and the reason on a rejection, and the subjects and serials that changed on success. They never include key material. --- ## Rotating a CA Rotating a leaf certificate is the easy case. Rotating the CA is where outages come from. If you swap the CA bundle on the server before every client has a certificate from the new CA, the clients that haven't rotated yet get locked out. `OverlapWindow` handles this. A CA that's removed from a bundle stays trusted for that long after it disappears: ```go r, err := keyload.New(files, keyload.OverlapWindow(24*time.Hour)) ``` So the rotation becomes three boring steps. Add the new CA to the bundle. Move certificates over to it. Remove the old CA. Peers that haven't rotated yet keep working until the window ends, and then the old CA is dropped on its own. This also applies to session resumption. A resumed session's client certificate is checked against the *current* client CAs, so once a CA is really gone, its clients can't sneak back in on an old session ticket. --- ## In Kubernetes This is the setup I built it for. cert-manager renews a `Certificate` into a Secret, the Secret is mounted into the pod, and keyload notices the change: ```yaml spec: containers: - name: api image: ghcr.io/example/api:1.0.0 # your app, using keyload volumeMounts: - name: tls mountPath: /tls readOnly: true volumes: - name: tls secret: secretName: api-tls ``` The kubelet doesn't edit the files in place. It writes a new directory and swaps a `..data` symlink to point at it. keyload watches the directories with inotify and handles that swap, so the change shows up within moments. No reloader sidecar and no restart. > **Mount the whole Secret, not a subPath.** A `subPath` mount is a copy that never gets updated, so keyload has nothing to see. Mount the Secret as a directory like the example above. On NFS, FUSE and other filesystems where inotify misses changes, and on macOS and Windows, keyload checks every 30 seconds instead. You can also reload on a signal or a timer: ```go r, err := keyload.New(files, keyload.OnSignal(syscall.SIGHUP), // kill -HUP to reload now keyload.PollEvery(5*time.Minute), // and on a timer ) ``` If your certificates don't live in files at all, say they come from Vault or a secrets API, implement `Source` and return PEM. `keyload.SourceFunc` turns a plain function into one. --- ## Knowing Before It Expires The worst certificate outage is the quiet one. Renewal broke three weeks ago, nobody noticed, and today the certificate expired. keyload exports metrics so that can't happen quietly. `keyload.MetricsHandler(r)` serves Prometheus text format without pulling in a client library: ```shell ➜ curl -s localhost:9090/metrics | grep -v '^#' keyload_generation{reloader="api"} 1 keyload_reload_total{reloader="api",result="success"} 1 keyload_reload_total{reloader="api",result="invalid"} 0 keyload_reload_total{reloader="api",result="read_error"} 0 keyload_cert_not_after_seconds{reloader="api",role="leaf"} 1.792025086e+09 keyload_cert_not_after_seconds{reloader="api",role="roots"} 1.823474686e+09 keyload_last_success_timestamp_seconds{reloader="api"} 1.791938686286792e+09 ``` Two alerts cover almost everything: ```yaml - alert: CertificateExpiresSoon expr: keyload_cert_not_after_seconds - time() < 7 * 86400 - alert: CertificateReloadFailing expr: increase(keyload_reload_total{result!="success"}[15m]) > 0 ``` The first catches renewal that stopped. The second catches someone pushing bad files. Pair that with `MinRemaining(time.Hour)` and keyload won't even accept a certificate that's about to expire, so you get the rejection and the alert instead of a surprise. --- ## HTTP, gRPC and Everything Else Different libraries want TLS handed to them in different ways, so there are a few entry points: | You have | Use | Who verifies | |---|---|---| | Any `crypto/tls` server (`http.Server`, raw listeners) | `r.ServerConfig(base)` | crypto/tls | | An `http.Transport` or your own TLS dials | `r.ConfigureTransport(tr, base)` or `r.DialTLSContext(base)` | crypto/tls | | grpc-go | `grpccreds.New(r)` | crypto/tls | | A library that only accepts a `*tls.Config` | `r.ClientConfig(base)` | keyload, matching crypto/tls | gRPC lives in its own module, `github.com/rossedman/keyload/grpccreds`, so the core library doesn't drag grpc-go into projects that don't use it: ```go creds := grpccreds.New(r, grpccreds.WithTLSConfig(&tls.Config{ ClientAuth: tls.RequireAndVerifyClientCert, })) srv := grpc.NewServer(grpc.Creds(creds)) conn, err := grpc.NewClient(addr, grpc.WithTransportCredentials(grpccreds.New(r))) ``` That last row of the table deserves a little honesty. Some libraries only take a static `*tls.Config` on the client side, and there's no hook in crypto/tls to swap the trusted roots on one of those. So in "Config mode" keyload sets `InsecureSkipVerify` and does the verification itself in `VerifyConnection`, using the same options crypto/tls would. I don't love writing that sentence, which is why it's the most heavily tested part of the library. CI generates 2,000 chains (wrong host, IP SANs, expired intermediates, wrong key usage, name constraints), verifies each one with both crypto/tls and Config mode, and fails the build on any disagreement. Use the other modes when you can. Use this one when a library leaves you no choice. > **Two gotchas worth knowing.** For HTTP/2 servers, set `NextProtos: []string{"h2", "http/1.1"}` in the base config you pass to `ServerConfig`. And `http.Transport` skips `DialTLSContext` for HTTPS through a proxy, so `ConfigureTransport` returns `ErrProxyConfigured` there. Use `ClientConfig` behind a proxy. --- ## Is It Fast? It sits in the handshake path, so it has to stay out of the way. `ServerConfig` hands each handshake a config that was built once per reload, so the hot path takes no locks and makes no allocations. - **~2.5 ns**: Server handshake overhead (0 allocations) - **~260 ns**: Client connection overhead (2 allocations) - **~2.4 µs**: Reload of unchanged files (3 allocations) Benchmarks run on every push, and a test fails the build if the handshake path ever starts allocating. The tests I care about more are the ones that look like real life. CI rotates certificates and CAs on both sides in the middle of traffic over HTTP/1.1, HTTP/2 and gRPC, and allows zero failed requests. It runs every kind of bad file through a reload and checks that the old certificates stay and the right reason comes back. It reproduces the kubelet's `..data` symlink swap on a real filesystem. And it fuzzes the PEM parsing. --- ## Try It Every example is a small runnable program with the commands to try it at the top of the file: - examples/ - gencerts/ — a demo CA and certificate, rotated on demand - server/ — an mTLS server whose certificate and client CAs reload - client/ — an mTLS client whose certificate and trusted CAs reload - proxy/ — an HTTPS client behind a proxy, using Config mode - metrics/ — Prometheus metrics, reload logs and alert rules - kubernetes/ — a cert-manager Certificate mounted into a Deployment ```shell git clone https://github.com/rossedman/keyload cd keyload go run ./examples/gencerts -dir /tmp/keyload go run ./examples/server -dir /tmp/keyload ``` There's also a [docs site with an interactive reload simulator](https://rossedman.github.io/keyload/), where you can push good and bad files at a reloader and watch what it does without running anything. [rossedman/keyload](https://github.com/rossedman/keyload) This is the first release, `v0.1.0`, and it's MIT licensed. Until 1.0 the API may still change. I've written before about [reloading on `SIGHUP`](https://rossedman.io/blog/computers/hot-reload-sighup-with-go/), and this is really the same idea applied to the one piece of config that will take you down on a schedule if you don't reload it. If you try it and a rotation surprises you, or there's a library it doesn't plug into yet, open an issue on GitHub. I'd love to hear about it.