Hot Reload: Respecting SIGHUP With Go

It’s the details that matter when you operate and run a piece of software. One of the most important details I find when operating applications is how it respects SIGHUP signals. Why? Because SIGHUP makes it easy to do configuration changes without disruption and config changes are the source of most major incidents. So what is SIGHUP and why should you care?

SIGHUP is a signal sent to a process when its controlling terminal is closed. However, when we are running a daemonized process (which we do a lot if you work below the application layers), we don’t ever expect this signal to get sent. Somewhere along the way it became common practice for daemonized processes to use SIGHUP for configuration reload. This allows us to not terminate the process while making changes!

In this article, I will show you a simple way to respect SIGHUP and “hot reload” (doing it live, its hot!) a config. We are going to create a simple web server that uses a config value to display a message. During this process we are going to send a SIGHUP and have the message update without downtime or restarting our process.

Setup

First, let’s setup a simple http server. I am just working in a directory and have a single file called main.go. Nothing crazy here, just a single endpoint that says “Hello, World!”

package main

import (
	"log"
	"net/http"
)

func main() {
    http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
        w.Write([]byte("Hello, World!"))
    })
    log.Fatal(http.ListenAndServe(":8080", nil))
}

To test this open two terminals and run go run main.go to start the server and in the other window run

➜ curl localhost:8080
Hello, World!

Wow. Much amaze.

Signal Handling

At this point we have a server but we only want our application to respond to one OS signal, SIGHUP. Go provides an os/signals package to help us with this! Let’s rub some of that on it.

package main

import (
	"log"
	"net/http"
+   "os"
+   "os/signals"
+   "syscall"
)

func main() {
    http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
        w.Write([]byte("Hello, World!"))
    })
+   sigs := make(chan os.Signal)
+   signal.Notify(sigs, syscall.SIGHUP)
    log.Fatal(http.ListenAndServe(":8080", nil))
}

Cool. Wait, what just happened? First, we load the os and syscall packages. In the main function we create a new channel that accepts values of the type os.Signal. We then pass that channel to signal.Notify as well as define the types of signals we want to notify on, in this case only SIGHUP.

There is a problem with this though. We aren’t doing anything with the signal. Let’s incrementally approach this. First goal is to log when a signal is thrown, that’s all. To do this, we need to check the values on the channel.

func main() {
    http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
        w.Write([]byte("Hello, World!"))
    })
+   sigs := make(chan os.Signal)
+   signal.Notify(sigs, syscall.SIGHUP)
    log.Fatal(http.ListenAndServe(":8080", nil))

+    for {
+        select {
+        case <-sigs:
+            log.Println("hot reload")
+        }
+    }
}

This is great but if we run this it still doesn’t work. Why? Because the http server is running before the select statement and so it never reaches that point until we exit because its blocking. We need the server to run in parallel so we can operate around it. To do this we will put this in its own goroutine.

func main() {
    http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
        w.Write([]byte("Hello, World!"))
    })
+   sigs := make(chan os.Signal)
+   signal.Notify(sigs, syscall.SIGHUP)

+   go func() {
+       log.Println("starting up....")
+       log.Fatal(http.ListenAndServe(":8080", nil))
+   }()

+   for {
+       select {
+       case <-sigs:
+           log.Println("hot reload")
+       }
+   }
}

Now we are getting somewhere. If we run this, we should be able to send SIGHUP signals to it and get the message “hot reload”. Let’s test it out! In one terminal start your server (go run main.go) and in another run ps to find your process number

# in terminal one
➜ go run main.go
2021/03/30 15:44:36 starting up....

Once the application is running, we need to find the process of the compiled executable. In the below output process 21371 is the one we want to send signals to not 21351. This will kill the go run main.go process but leave your server running.

# in terminal two
➜ ps
  PID TTY           TIME CMD
18507 ttys000    0:00.62 -zsh
 3162 ttys001    0:00.95 -zsh
21351 ttys001    0:00.92 go run main.go
21371 ttys001    0:00.01 /var/folders/ny/sp_8f52x4hj_lrkl5cqz4ndw0000gp/T/go-build725126617/b001/exe/main

# send the signal!
➜ kill -SIGHUP 21371

In terminal one you should see something similar to this

2021/03/30 15:44:36 starting up....
2021/03/30 15:46:06 hot reload

Configuration

Now all we need is a configuration value to reload! Let’s add on to our criteria now that logging is working. Next, we expect when a SIGHUP signal is received for the configuration to be reloaded. Let’s add some configuration values!

package main

import (
    "log"
    "net/http"
    "os"
    "os/signal"
    "syscall"
)

+var conf = &Config{Message: "Hello, World!"}

+type Config struct {
+    Message string
+}

func main() {
    http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
+        w.Write([]byte(conf.Message))
    })

    sigs := make(chan os.Signal, 1)
    signal.Notify(sigs, syscall.SIGHUP)

    go func() {
        log.Println("starting up....")
        log.Fatal(http.ListenAndServe(":8080", nil))
    }()

    for {
        select {
        case <-sigs:
            log.Println("hot reload")
+            conf.Message = "Go To Hell!"
        }
    }
}

So above we have added a Config struct, set a default with the Message value of Hello World! and we have altered our handler to use this configuration value. We have also altered our select area to set a new message whenever we receive a SIGHUP. This message is less welcoming.

Let’s run this!

# in terminal one
➜ go run main.go
2021/03/30 15:44:36 starting up....

In terminal two, test the endpoint to make sure our default settings work

# in terminal two
➜ curl localhost:8080
Hello, World!

Now send a SIGHUP just like we did before and then attempt to curl again

➜ curl localhost:8080
Go To Hell!

Conclusion

Congratulations, you just learned one of my favorite patterns in computers! This is a powerful pattern that scales really well. If I was going to continue working on this I would load configuration values from a file or a datastore (or both!). You could then use any number of tools to send a SIGHUP to all of your processes to do a config reload.