UNIX domain vs TCP socket.

With increasing adoption of microservices, containers and design patterns like sidecar, communication efficiency starts to play an important role in system performance. Reliability, congestion control, packet ordering and many other useful features made TCP the most widely used network communication protocol, but while these properties are relevant in the context of unreliable networks, they only add unnecessary overhead when used for IPC. But what kind of overhead do we expect to observe in practice? To find out, we'll write Go benchmarks that establish connection between client and a server and play ping-pong using messages of size MsgSize. Since most of the logic is shared between UNIX domain and TCP socket benchmarks, it was factored out into a helper function benchmark

import (
    "net"
    "os"
    "testing"
)

const UnixAddress = "/tmp/benchmark.sock"
const MsgSize = 1
...
func benchmark(b *testing.B, domain string, address string) {
    read := func(conn net.Conn, buf []byte) {
        nread, err := conn.Read(buf)
        if err != nil {
            b.Fatal(err)
        }
        if nread != MsgSize {
            b.Fatalf("unexpected nread = %d", nread)
        }
    }

    write := func(conn net.Conn, buf []byte) {
        nwrite, err := conn.Write(buf)
        if err != nil {
            b.Fatal(err)
        }
        if nwrite != MsgSize {
            b.Fatalf("unexpected nwrite = %d", nwrite)
        }
    }

    l, err := net.Listen(domain, address)
    if err != nil {
        b.Fatal(err)
    }
    defer l.Close()

    go func() {
        conn, err := l.Accept()
        if err != nil {
            b.Fatal(err)
        }
        defer conn.Close()

        buf := make([]byte, MsgSize)
        for n := 0; n < b.N; n++ {
            read(conn, buf)
            write(conn, buf)
        }
    }()

    conn, err := net.Dial(domain, address)
    if err != nil {
        b.Fatal(err)
    }
    defer conn.Close()

    buf := make([]byte, MsgSize)

    b.ResetTimer()
    for n := 0; n < b.N; n++ {
        write(conn, buf)
        read(conn, buf)
    }
    b.StopTimer()
}

all that's left is to write a UNIX domain benchmark

func BenchmarkUnixDomain(b *testing.B) {
    if err := os.RemoveAll(UnixAddress); err != nil {
        panic(err)
    }
    benchmark(b, "unix", UnixAddress)
}

that's using and TCP benchmark

func BenchmarkTcpSocket(b *testing.B) {
    benchmark(b, "tcp", "127.0.0.1:6666")
}

And here are the results:

BenchmarkUnixDomain-16            164767              6859 ns/op
BenchmarkTcpSocket-16              33434             35807 ns/op

Which means that for a MsgSize of 1 UNIX domain socket communication is 5X faster than TCP-based one. Increasing MsgSize to 8096 has little effect on the relative ratio:

BenchmarkUnixDomain-16            140995              7981 ns/op
BenchmarkTcpSocket-16              30488             37790 ns/op

This serves as a good reminder of Occam’s Razor, that, put simply, states: “the simplest solution is almost always the best.” and a reason why many daemons, like Docker, use UNIX domain sockets.

UNIX domain vs TCP socket.

Or avoiding unnecessary overhead.