# BuildKit As A Library: A CI Pipeline In Go > Drive Docker's built-in BuildKit from a Go program to vet, test and cross-compile a project in parallel, with caching, and export the binaries to your machine. - Author: Ross Edman - Published: 2026-09-10 - Tags: docker, go - URL: https://rossedman.io/blog/computers/buildkit-as-a-library/ **TL;DR:** BuildKit isn't just what runs `docker build`. It's a Go library. Describe your CI steps as a build graph, connect to the BuildKit inside Docker, and you get parallel steps, caching and cross-compiled artifacts in ./dist from a plain `go run`. In the [last post](https://rossedman.io/blog/computers/custom-docker-syntax-with-buildkit/) we wrote a BuildKit frontend: a plugin that runs *inside* `docker build` and decides what a build file means. This time we'll flip it around. BuildKit is a set of Go libraries, and nothing says you need a Dockerfile, a frontend or even `docker build` to use them. We'll write a small Go program that is our whole CI pipeline. It vets and tests a project, cross-compiles it for three platforms, and drops the binaries in `./dist` on your machine. Everything runs in containers, in parallel, with caching, using the BuildKit that already ships inside Docker. This is the same idea tools like Dagger and Earthly are built on. Let's dive into it. --- ## The Project The thing we're building is about as small as a Go project gets: a `greeting` function, a test for it, and a `main`. The pipeline lives in its own module under `ci/`, so BuildKit's dependencies never touch the app's `go.mod`. - greet/ - go.mod - main.go - main_test.go - ci/ - go.mod - **main.go** ```shell ➜ cd ci ➜ go mod init example.com/ci ➜ go get github.com/moby/buildkit@v0.34.0 github.com/moby/moby/client ``` ## The Pipeline Here's all of it. We'll walk through it after. ```go package main import ( "context" "fmt" "log" "net" "os" "path/filepath" "github.com/moby/buildkit/client" "github.com/moby/buildkit/client/llb" gateway "github.com/moby/buildkit/frontend/gateway/client" "github.com/moby/buildkit/util/progress/progressui" docker "github.com/moby/moby/client" ocispecs "github.com/opencontainers/image-spec/specs-go/v1" "github.com/tonistiigi/fsutil" "golang.org/x/sync/errgroup" ) // The platforms we ship binaries for. var targets = []ocispecs.Platform{ {OS: "linux", Architecture: "amd64"}, {OS: "linux", Architecture: "arm64"}, {OS: "darwin", Architecture: "arm64"}, } func main() { src := "." if len(os.Args) > 1 { src = os.Args[1] } if err := run(context.Background(), src); err != nil { log.Fatal(err) } } func run(ctx context.Context, src string) error { c, err := connect(ctx) if err != nil { return err } defer c.Close() srcFS, err := fsutil.NewFS(src) if err != nil { return err } opt := client.SolveOpt{ // Files from this machine, exposed to the build as llb.Local("src"). LocalMounts: map[string]fsutil.FS{"src": srcFS}, // Write the final state to ./dist instead of making an image. Exports: []client.ExportEntry{{ Type: client.ExporterLocal, OutputDir: filepath.Join(src, "dist"), }}, } ch := make(chan *client.SolveStatus) display, err := progressui.NewDisplay(os.Stderr, progressui.AutoMode) if err != nil { return err } eg, ctx := errgroup.WithContext(ctx) eg.Go(func() error { _, err := c.Build(ctx, opt, "", func(ctx context.Context, gw gateway.Client) (*gateway.Result, error) { def, err := pipeline(gw).Marshal(ctx) if err != nil { return nil, err } return gw.Solve(ctx, gateway.SolveRequest{Definition: def.ToPB(), Evaluate: true}) }, ch) return err }) eg.Go(func() error { _, err := display.UpdateFrom(ctx, ch) return err }) return eg.Wait() } // connect talks to the BuildKit that is built into the Docker daemon, // the same way `docker build` does. func connect(ctx context.Context) (*client.Client, error) { d, err := docker.New(docker.FromEnv) if err != nil { return nil, err } return client.New(ctx, "", client.WithContextDialer(func(ctx context.Context, _ string) (net.Conn, error) { return d.DialHijack(ctx, "/grpc", "h2c", nil) }), client.WithSessionDialer(func(ctx context.Context, proto string, meta map[string][]string) (net.Conn, error) { return d.DialHijack(ctx, "/session", proto, meta) }), ) } // pipeline describes the whole CI run as one build graph. func pipeline(gw gateway.Client) llb.State { // Run the toolchain on BuildKit's own platform, whatever machine we're on. worker := gw.BuildOpts().Workers[0].Platforms[0] golang := llb.Image("docker.io/library/golang:1.26-alpine", llb.Platform(worker), llb.WithMetaResolver(gw), ) src := llb.Local("src", llb.ExcludePatterns([]string{"dist", "ci"})) // Every Go step gets the source and shared build and module caches. goStep := func(name string, script string, env ...string) llb.ExecState { opts := []llb.RunOption{ llb.Args([]string{"sh", "-c", script}), llb.Dir("/src"), llb.AddEnv("CGO_ENABLED", "0"), llb.AddMount("/src", src, llb.Readonly), llb.AddMount("/root/.cache/go-build", llb.Scratch(), llb.AsPersistentCacheDir("ci-go-build", llb.CacheMountShared)), llb.AddMount("/go/pkg/mod", llb.Scratch(), llb.AsPersistentCacheDir("ci-go-mod", llb.CacheMountShared)), llb.WithCustomName(name), } for i := 0; i+1 < len(env); i += 2 { opts = append(opts, llb.AddEnv(env[i], env[i+1])) } return golang.Run(opts...) } // Checks write a small report into /out, so the release depends on them. vet := goStep("go vet", "go vet ./... && echo ok > /out/vet.txt").AddMount("/out", llb.Scratch()) test := goStep("go test", "set -o pipefail; go test -v ./... 2>&1 | tee /out/test.txt").AddMount("/out", llb.Scratch()) outputs := []llb.State{vet, test} for _, p := range targets { name := fmt.Sprintf("greet-%s-%s", p.OS, p.Architecture) bin := goStep("go build "+name, "go build -trimpath -o /out/"+name+" .", "GOOS", p.OS, "GOARCH", p.Architecture).AddMount("/out", llb.Scratch()) outputs = append(outputs, bin) } // dist/ is everything above, layered together. return llb.Merge(outputs, llb.WithCustomName("collect dist")) } ``` ### Connecting To Docker's BuildKit BuildKit has been Docker's default builder since Docker 23, and the daemon exposes it over the same socket as everything else. `connect` borrows the trick `docker build` itself uses: `DialHijack` turns a request to the Docker API into a raw connection, one for BuildKit's gRPC API and one for the **session**. The session is how BuildKit reaches back into our process to read local files and to write exported files. No extra daemon, no `buildkitd` to install. ### Describing The Graph `pipeline` doesn't run anything. Like in the frontend post, `llb.Image`, `Run` and `Merge` only describe steps. In plain words: **Start from golang:1.26-alpine**: Pinned to the platform of BuildKit's own worker. On a Mac, Docker runs Linux in a VM, and that VM's platform is what we want, not `darwin`.**Mount the source and two caches into every step**: `llb.Local("src")` is our project directory. The Go build and module caches are persistent cache mounts, so they survive between runs.**Vet, test and build each write into an empty /out**: Each step's `/out` mount becomes its result: a report, or a binary for one platform.**Merge all the /out mounts into one tree**: That merged tree is `dist/`. ### Running It `c.Build` hands us the same `gateway.Client` a frontend gets, so `llb.WithMetaResolver(gw)` can look up the Go image's config through Docker (that's how `go` ends up on the `PATH`). We solve the graph, and the `local` exporter copies the result into `dist/`. Meanwhile `progressui` draws the same progress output you're used to from `docker build`. ## Run It ```shell ➜ cd ci ➜ go run . .. ``` The first run took 28 seconds on my machine. Here's the interesting part: the five Go steps took 118 seconds when you add them up. BuildKit saw that vet, test and the three builds don't depend on each other and ran them all at once. ```shell ➜ ls ../dist greet-darwin-arm64 greet-linux-amd64 greet-linux-arm64 test.txt vet.txt ➜ file ../dist/greet-darwin-arm64 ../dist/greet-darwin-arm64: Mach-O 64-bit arm64 executable, flags:<|DYLDLINK|PIE> ``` Run it again without changing anything and it's done in about a second: ```text #4 go test #4 CACHED #5 go build greet-linux-amd64 #5 CACHED #6 go build greet-linux-arm64 #6 CACHED #7 go vet #7 CACHED #8 go build greet-darwin-arm64 #8 CACHED ``` Change a line of code and only the steps that care run again, with warm Go caches. For me that was 2 seconds. Notice `dist` and `ci` are excluded from `llb.Local`, so writing the output, or editing the pipeline itself, never counts as a source change. ## When A Test Fails Break the test and the pipeline stops with the test output right there in the log: ```text #8 0.400 main_test.go:7: greeting("") = "Hello, World!" #8 0.400 --- FAIL: TestGreeting (0.00s) #8 0.400 FAIL ``` `go run` exits non-zero, so it works in any CI system, and **nothing** is written to `dist/`. You can't ship a binary from a run where the tests failed. > **Keep the output visible.** The test step pipes through `tee` so the output lands in the log and in `test.txt`. `set -o pipefail` keeps the pipe from swallowing `go test`'s exit code. ## The Weird Part: Nothing Runs Unless It's Needed Why do vet and test write those little report files at all? Because a build graph is **lazy**. BuildKit only runs steps that the final result depends on. If `go vet` produced nothing that ends up in `dist/`, BuildKit would skip it entirely. I tried it: take `vet` out of the `outputs` slice (Go makes you keep the variable around with `_ = vet`) and it never runs. No error, no warning, it's just gone. So the reports aren't decoration. They're how the release *depends* on the checks. It's a different way to think about a pipeline. You don't list steps in order, you describe what the result is made of, and the order and parallelism fall out of that. ## What Else You Can Build With These Libraries This pipeline and the last post's frontend barely scratch the surface. A few ideas I'd like to try next: - **A Dockerfile linter with no daemon.** BuildKit's Dockerfile parser is a plain Go package. Parse every Dockerfile in a repo and flag unpinned base images, `latest` tags or a missing `USER`, in milliseconds. - **A policy frontend that wraps the Dockerfile frontend.** A frontend can call the real Dockerfile frontend itself, then check or change the result: add labels, enforce approved base images, attach an SBOM. Teams opt in with one `# syntax` line. - **Debugging a failed step.** The gateway client can start a container on any state in the graph, so you can drop into a shell inside the exact filesystem where a step failed. It's what `docker buildx debug` does. - **Shared caches for CI.** `SolveOpt` takes cache exports and imports, so this exact pipeline can push its cache to a registry or the GitHub Actions cache and start warm on every runner. - **Drawing the graph.** A marshaled definition is just a list of operations and their inputs. Turn it into Graphviz and you can see why something rebuilt. ## Conclusion That's it! Under 150 lines of Go, a Docker daemon you already have, and you get a pipeline that runs steps in parallel, caches everything, works the same on a laptop and in CI, and can't ship untested code. No YAML and no Dockerfile, just a program you can read, test and refactor like anything else. Happy hacking!