Introducing brownout: Turn Off Go Endpoints Without a Deploy

Most incidents I’ve been part of have a moment where somebody says “can we just turn that endpoint off?” The database is on fire because of one expensive report, or a downstream service is falling over and every request to it is making things worse. The answer is usually yes, but how is the problem. You ship a code change, flip a feature flag that may or may not exist for that route, or edit a load balancer config by hand under pressure. All of those are slow, and slow is the last thing you want in the middle of an incident.

So I wrote brownout, a small Go library that does one thing: it lets you shed some or all of the traffic to selected HTTP routes while the server is running. You add a rule, the very next request sees it, and when things recover you remove the rule.

The name comes from the power grid. A blackout is everything off. A brownout is reducing the load on purpose so the whole system doesn’t go down.

What You Get #

• Plain net/http middleware. It works with any router built on http.Handler.
• Partial brownouts. Let a percentage of traffic through instead of all or nothing.
• Method filters. Shed only POST, PUT, PATCH and DELETE to make an API read-only.
• Rules that expire. A forgotten brownout doesn’t stay on forever.
• Three ways to change rules: the Go API, a built-in JSON admin API, or a JSON file it watches for changes.
• A lock-free request path. Rules are swapped atomically, so checking them costs almost nothing.
• No dependencies outside the standard library.

go get github.com/rossedman/brownout

Quick Start #

You create a Controller, wrap your app with its middleware, and expose its admin API somewhere only you can reach.

package main

import (
	"fmt"
	"log"
	"net/http"

	"github.com/rossedman/brownout"
)

func main() {
	c := brownout.New()

	app := http.NewServeMux()
	app.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
		fmt.Fprintf(w, "hello from %s\n", r.URL.Path)
	})

	// The admin API goes on an internal-only listener.
	go func() {
		log.Fatal(http.ListenAndServe("127.0.0.1:9090", c.AdminHandler()))
	}()

	log.Fatal(http.ListenAndServe(":8080", c.Middleware(app)))
}

With no rules, the middleware does nothing:

➜ curl localhost:8080/reports/q3
hello from /reports/q3

Now let’s pretend the reports endpoint is hammering the database. Shed all of it for ten minutes and ask clients to come back in a minute:

➜ curl -X PUT localhost:9090/rules/reports \
    -d '{"path": "/reports/*", "ttl": "10m", "retry_after_seconds": 60}'
{"id":"reports","path":"/reports/*","retry_after_seconds":60,"expires_at":"2026-10-13T10:29:31Z"}

The very next request gets a 503:

➜ curl -i localhost:8080/reports/q3
HTTP/1.1 503 Service Unavailable
Content-Type: text/plain; charset=utf-8
Retry-After: 60

Service Unavailable

When the database recovers, delete the rule. If you forget, it lifts by itself when the ten minutes are up.

➜ curl -X DELETE localhost:9090/rules/reports
➜ curl localhost:8080/reports/q3
hello from /reports/q3

Partial Brownouts #

Turning an endpoint all the way off is often too much. Search might be expensive, but it’s also how people find things. allow_percent lets a share of matching requests through and sheds the rest:

➜ curl -X PUT localhost:9090/rules/search -d '{"path": "/search", "allow_percent": 20}'

I sent 100 requests at it to check:

➜ for i in $(seq 100); do curl -s -o /dev/null -w '%{http_code}\n' localhost:8080/search; done | sort | uniq -c
     18 200
     82 503

About one in five got through, which is what we asked for. Each request is decided on its own, so it won’t be exactly 20 every time.

Making an API Read-Only #

Rules can be limited to certain HTTP methods. This one sheds every write to the API and leaves reads alone, which is handy during a database failover or a migration:

➜ curl -X PUT localhost:9090/rules/readonly \
    -d '{"path": "/api/*", "methods": ["POST", "PUT", "PATCH", "DELETE"]}'

How Rules Match #

A rule’s path is either an exact path (/search), a prefix ending in * (/api/*), or * on its own to match everything. Rules are checked in order and the first match decides. If the first matching rule lets a request through, later rules aren’t checked. That’s how you carve out exceptions:

c.Set(brownout.Rule{ID: "health", Path: "/api/health", AllowPercent: 100}) // always allowed
c.Set(brownout.Rule{ID: "api", Path: "/api/*"})                            // everything else shed

Here’s everything a rule can set:

Three Ways to Change Rules #

The admin API is the one you’ll reach for in an incident, but it isn’t the only way in.

c.Set(rule)          // add or update one rule
c.Remove("reports")  // remove one rule
c.Replace(rules)     // swap the whole set atomically
c.Clear()            // remove every rule
c.Rules()            // list the active rules

curl localhost:9090/rules                     # list
curl -X PUT localhost:9090/rules/reports -d '{"path": "/reports/*"}'
curl -X PUT localhost:9090/rules -d '{"rules": [...]}'   # replace all
curl -X DELETE localhost:9090/rules/reports   # remove one
curl -X DELETE localhost:9090/rules           # remove all

go c.WatchFile(ctx, "/etc/brownout/rules.json", 5*time.Second, func(err error) {
	log.Printf("brownout: %v", err)
})

Replace is all or nothing. If any rule in the set is invalid, nothing changes.

Rules From a Kubernetes ConfigMap #

Rules live in memory, one copy per process. If you run several replicas, calling the admin API on each one gets old fast. Instead, mount a ConfigMap as a file and let every replica watch it:

{
  "rules": [
    {
      "id": "reports",
      "path": "/reports/*",
      "retry_after_seconds": 120,
      "message": "Reports are temporarily disabled while we recover from an incident."
    },
    { "id": "search-partial", "path": "/search", "allow_percent": 25 }
  ]
}

kubectl edit configmap, wait for the kubelet to sync the file, and every pod picks up the change on its next poll. If the file is half-written or isn’t valid JSON, the reload fails and the current rules stay in place, so a typo won’t wipe out your brownout.

Automatic Brownouts #

Because rules can be set from Go, nothing says a human has to set them. Here’s a loop that watches a dependency’s health. After three failed probes in a row it sheds the routes that need that dependency, and after three good ones it restores them:

func watchDependency(ctx context.Context, c *brownout.Controller, dep *dependency) {
	var failures, successes int
	ticker := time.NewTicker(time.Second)
	defer ticker.Stop()

	for {
		select {
		case <-ctx.Done():
			return
		case <-ticker.C:
		}

		if err := dep.Ping(ctx); err != nil {
			failures, successes = failures+1, 0
		} else {
			failures, successes = 0, successes+1
		}
		_, active := c.Rule("auto-recommendations")

		switch {
		case failures >= 3:
			c.Set(brownout.Rule{
				ID:                "auto-recommendations",
				Path:              "/recommendations/*",
				RetryAfterSeconds: 30,
				ExpiresAt:         time.Now().Add(time.Minute),
			})
		case successes >= 3 && active:
			c.Remove("auto-recommendations")
		}
	}
}

The ExpiresAt is a safety net. The loop sets the rule again on every failed probe, which keeps pushing the expiry out. If the loop itself ever gets stuck, the brownout still lifts on its own within a minute.

Degrading Instead of Rejecting #

Sometimes a 503 is the wrong answer. If a product page calls an expensive personalization service, you’d rather show the plain page than an error. Check gives you the decision without the middleware’s response, so you can do whatever you want with it:

app.HandleFunc("/products/{id}", func(w http.ResponseWriter, r *http.Request) {
	id := r.PathValue("id")

	if rule, degraded := features.Check(r); degraded {
		w.Header().Set("X-Degraded", rule.ID)
		fmt.Fprintf(w, "Product %s\n", id) // the cheap version
		return
	}
	fmt.Fprintln(w, personalize(id)) // the expensive version
})

features here is its own Controller, separate from the one wrapping the app, so rules meant to degrade a page never trigger the hard 503. Set {"path": "/products/*"} on it and the product page keeps returning 200, just without the personalized parts.

Metrics and Custom Responses #

You’ll want to know how much traffic a brownout is actually shedding, and your API clients probably expect JSON errors rather than plain text. Two options cover that:

c := brownout.New(
	brownout.WithOnShed(func(r *http.Request, rule brownout.Rule) {
		shedCounter.WithLabelValues(rule.ID).Inc()
	}),
	brownout.WithResponder(func(w http.ResponseWriter, r *http.Request, rule brownout.Rule) {
		w.Header().Set("Content-Type", "application/json")
		w.WriteHeader(rule.StatusCode)
		json.NewEncoder(w).Encode(map[string]string{
			"error": "temporarily unavailable",
			"rule":  rule.ID,
		})
	}),
)

WithOnShed runs for every shed request, which is where you’d bump a Prometheus counter or write a log line. WithResponder replaces the response body entirely. There’s also WithDefaultStatus if you’d rather send 429 than 503 by default.

Try It #

Every example above is a small runnable server in the examples/ folder, with the curl commands to try it at the top of each file:

git clone https://github.com/rossedman/brownout
cd brownout
go run ./examples/basic

There’s also a docs site with an interactive rule playground where you can try rules against paths without running anything.

The next time someone asks “can we just turn that off?” in an incident channel, I’d like the answer to be one curl. If you try it out, or there’s a rule you wish it could express, open an issue on GitHub. I’d love to hear about it.