docker go

BuildKit As A Library: A CI Pipeline In Go

8 min read
TL;DR
BuildKit isn’t just what runs docker build. It’s a Go library. Describe your CI steps as a build graph, connect to the BuildKit inside Docker, and you get parallel steps, caching and cross-compiled artifacts in ./dist from a plain go run.

In the last post we wrote a BuildKit frontend: a plugin that runs inside docker build and decides what a build file means. This time we’ll flip it around. BuildKit is a set of Go libraries, and nothing says you need a Dockerfile, a frontend or even docker build to use them.

We’ll write a small Go program that is our whole CI pipeline. It vets and tests a project, cross-compiles it for three platforms, and drops the binaries in ./dist on your machine. Everything runs in containers, in parallel, with caching, using the BuildKit that already ships inside Docker. This is the same idea tools like Dagger and Earthly are built on. Let’s dive into it.

The Project #

The thing we’re building is about as small as a Go project gets: a greeting function, a test for it, and a main. The pipeline lives in its own module under ci/, so BuildKit’s dependencies never touch the app’s go.mod.

  • greet/
    • go.mod
    • main.go
    • main_test.go
    • ci/
      • go.mod
      • main.go
shell 
➜ cd ci
➜ go mod init example.com/ci
➜ go get github.com/moby/buildkit@v0.34.0 github.com/moby/moby/client

The Pipeline #

Here’s all of it. We’ll walk through it after.

ci/main.go go 
package main

import (
	"context"
	"fmt"
	"log"
	"net"
	"os"
	"path/filepath"

	"github.com/moby/buildkit/client"
	"github.com/moby/buildkit/client/llb"
	gateway "github.com/moby/buildkit/frontend/gateway/client"
	"github.com/moby/buildkit/util/progress/progressui"
	docker "github.com/moby/moby/client"
	ocispecs "github.com/opencontainers/image-spec/specs-go/v1"
	"github.com/tonistiigi/fsutil"
	"golang.org/x/sync/errgroup"
)

// The platforms we ship binaries for.
var targets = []ocispecs.Platform{
	{OS: "linux", Architecture: "amd64"},
	{OS: "linux", Architecture: "arm64"},
	{OS: "darwin", Architecture: "arm64"},
}

func main() {
	src := "."
	if len(os.Args) > 1 {
		src = os.Args[1]
	}
	if err := run(context.Background(), src); err != nil {
		log.Fatal(err)
	}
}

func run(ctx context.Context, src string) error {
	c, err := connect(ctx)
	if err != nil {
		return err
	}
	defer c.Close()

	srcFS, err := fsutil.NewFS(src)
	if err != nil {
		return err
	}
	opt := client.SolveOpt{
		// Files from this machine, exposed to the build as llb.Local("src").
		LocalMounts: map[string]fsutil.FS{"src": srcFS},
		// Write the final state to ./dist instead of making an image.
		Exports: []client.ExportEntry{{
			Type:      client.ExporterLocal,
			OutputDir: filepath.Join(src, "dist"),
		}},
	}

	ch := make(chan *client.SolveStatus)
	display, err := progressui.NewDisplay(os.Stderr, progressui.AutoMode)
	if err != nil {
		return err
	}

	eg, ctx := errgroup.WithContext(ctx)
	eg.Go(func() error {
		_, err := c.Build(ctx, opt, "", func(ctx context.Context, gw gateway.Client) (*gateway.Result, error) {
			def, err := pipeline(gw).Marshal(ctx)
			if err != nil {
				return nil, err
			}
			return gw.Solve(ctx, gateway.SolveRequest{Definition: def.ToPB(), Evaluate: true})
		}, ch)
		return err
	})
	eg.Go(func() error {
		_, err := display.UpdateFrom(ctx, ch)
		return err
	})
	return eg.Wait()
}

// connect talks to the BuildKit that is built into the Docker daemon,
// the same way `docker build` does.
func connect(ctx context.Context) (*client.Client, error) {
	d, err := docker.New(docker.FromEnv)
	if err != nil {
		return nil, err
	}
	return client.New(ctx, "",
		client.WithContextDialer(func(ctx context.Context, _ string) (net.Conn, error) {
			return d.DialHijack(ctx, "/grpc", "h2c", nil)
		}),
		client.WithSessionDialer(func(ctx context.Context, proto string, meta map[string][]string) (net.Conn, error) {
			return d.DialHijack(ctx, "/session", proto, meta)
		}),
	)
}

// pipeline describes the whole CI run as one build graph.
func pipeline(gw gateway.Client) llb.State {
	// Run the toolchain on BuildKit's own platform, whatever machine we're on.
	worker := gw.BuildOpts().Workers[0].Platforms[0]
	golang := llb.Image("docker.io/library/golang:1.26-alpine",
		llb.Platform(worker),
		llb.WithMetaResolver(gw),
	)
	src := llb.Local("src", llb.ExcludePatterns([]string{"dist", "ci"}))

	// Every Go step gets the source and shared build and module caches.
	goStep := func(name string, script string, env ...string) llb.ExecState {
		opts := []llb.RunOption{
			llb.Args([]string{"sh", "-c", script}),
			llb.Dir("/src"),
			llb.AddEnv("CGO_ENABLED", "0"),
			llb.AddMount("/src", src, llb.Readonly),
			llb.AddMount("/root/.cache/go-build", llb.Scratch(), llb.AsPersistentCacheDir("ci-go-build", llb.CacheMountShared)),
			llb.AddMount("/go/pkg/mod", llb.Scratch(), llb.AsPersistentCacheDir("ci-go-mod", llb.CacheMountShared)),
			llb.WithCustomName(name),
		}
		for i := 0; i+1 < len(env); i += 2 {
			opts = append(opts, llb.AddEnv(env[i], env[i+1]))
		}
		return golang.Run(opts...)
	}

	// Checks write a small report into /out, so the release depends on them.
	vet := goStep("go vet", "go vet ./... && echo ok > /out/vet.txt").AddMount("/out", llb.Scratch())
	test := goStep("go test", "set -o pipefail; go test -v ./... 2>&1 | tee /out/test.txt").AddMount("/out", llb.Scratch())

	outputs := []llb.State{vet, test}
	for _, p := range targets {
		name := fmt.Sprintf("greet-%s-%s", p.OS, p.Architecture)
		bin := goStep("go build "+name, "go build -trimpath -o /out/"+name+" .",
			"GOOS", p.OS, "GOARCH", p.Architecture).AddMount("/out", llb.Scratch())
		outputs = append(outputs, bin)
	}

	// dist/ is everything above, layered together.
	return llb.Merge(outputs, llb.WithCustomName("collect dist"))
}

Connecting To Docker’s BuildKit #

BuildKit has been Docker’s default builder since Docker 23, and the daemon exposes it over the same socket as everything else. connect borrows the trick docker build itself uses: DialHijack turns a request to the Docker API into a raw connection, one for BuildKit’s gRPC API and one for the session. The session is how BuildKit reaches back into our process to read local files and to write exported files. No extra daemon, no buildkitd to install.

Describing The Graph #

pipeline doesn’t run anything. Like in the frontend post, llb.Image, Run and Merge only describe steps. In plain words:

  1. Start from golang:1.26-alpine

    Pinned to the platform of BuildKit’s own worker. On a Mac, Docker runs Linux in a VM, and that VM’s platform is what we want, not darwin.

  2. Mount the source and two caches into every step

    llb.Local("src") is our project directory. The Go build and module caches are persistent cache mounts, so they survive between runs.

  3. Vet, test and build each write into an empty /out

    Each step’s /out mount becomes its result: a report, or a binary for one platform.

  4. Merge all the /out mounts into one tree

    That merged tree is dist/.

Running It #

c.Build hands us the same gateway.Client a frontend gets, so llb.WithMetaResolver(gw) can look up the Go image’s config through Docker (that’s how go ends up on the PATH). We solve the graph, and the local exporter copies the result into dist/. Meanwhile progressui draws the same progress output you’re used to from docker build.

Run It #

shell 
➜ cd ci
➜ go run . ..

The first run took 28 seconds on my machine. Here’s the interesting part: the five Go steps took 118 seconds when you add them up. BuildKit saw that vet, test and the three builds don’t depend on each other and ran them all at once.

shell 
➜ ls ../dist
greet-darwin-arm64  greet-linux-amd64  greet-linux-arm64  test.txt  vet.txt

➜ file ../dist/greet-darwin-arm64
../dist/greet-darwin-arm64: Mach-O 64-bit arm64 executable, flags:<|DYLDLINK|PIE>

Run it again without changing anything and it’s done in about a second:

text 
#4 go test
#4 CACHED
#5 go build greet-linux-amd64
#5 CACHED
#6 go build greet-linux-arm64
#6 CACHED
#7 go vet
#7 CACHED
#8 go build greet-darwin-arm64
#8 CACHED

Change a line of code and only the steps that care run again, with warm Go caches. For me that was 2 seconds. Notice dist and ci are excluded from llb.Local, so writing the output, or editing the pipeline itself, never counts as a source change.

When A Test Fails #

Break the test and the pipeline stops with the test output right there in the log:

text 
#8 0.400     main_test.go:7: greeting("") = "Hello, World!"
#8 0.400 --- FAIL: TestGreeting (0.00s)
#8 0.400 FAIL

go run exits non-zero, so it works in any CI system, and nothing is written to dist/. You can’t ship a binary from a run where the tests failed.

The Weird Part: Nothing Runs Unless It’s Needed #

Why do vet and test write those little report files at all? Because a build graph is lazy. BuildKit only runs steps that the final result depends on. If go vet produced nothing that ends up in dist/, BuildKit would skip it entirely. I tried it: take vet out of the outputs slice (Go makes you keep the variable around with _ = vet) and it never runs. No error, no warning, it’s just gone.

So the reports aren’t decoration. They’re how the release depends on the checks. It’s a different way to think about a pipeline. You don’t list steps in order, you describe what the result is made of, and the order and parallelism fall out of that.

What Else You Can Build With These Libraries #

This pipeline and the last post’s frontend barely scratch the surface. A few ideas I’d like to try next:

  • A Dockerfile linter with no daemon. BuildKit’s Dockerfile parser is a plain Go package. Parse every Dockerfile in a repo and flag unpinned base images, latest tags or a missing USER, in milliseconds.
  • A policy frontend that wraps the Dockerfile frontend. A frontend can call the real Dockerfile frontend itself, then check or change the result: add labels, enforce approved base images, attach an SBOM. Teams opt in with one # syntax line.
  • Debugging a failed step. The gateway client can start a container on any state in the graph, so you can drop into a shell inside the exact filesystem where a step failed. It’s what docker buildx debug does.
  • Shared caches for CI. SolveOpt takes cache exports and imports, so this exact pipeline can push its cache to a registry or the GitHub Actions cache and start warm on every runner.
  • Drawing the graph. A marshaled definition is just a list of operations and their inputs. Turn it into Graphviz and you can see why something rebuilt.

Conclusion #

That’s it! Under 150 lines of Go, a Docker daemon you already have, and you get a pipeline that runs steps in parallel, caches everything, works the same on a laptop and in CI, and can’t ship untested code. No YAML and no Dockerfile, just a program you can read, test and refactor like anything else. Happy hacking!

Aug 2026 · #go Running A Go Service With systemd Write a systemd unit for a Go service, tell systemd when it's ready, keep the watchdog fed, reload on SIGHUP and lock it down with sandboxing options. Aug 2026 · #go Graceful Shutdown: Respecting SIGTERM With Go Catch SIGTERM with signal.NotifyContext and let http.Server.Shutdown finish in-flight requests before your Go service exits. Mar 2021 · #go Hot Reload: Respecting SIGHUP With Go Reload a Go web server's configuration when it receives SIGHUP, without restarting the process or dropping traffic.