Tests

Leave testing to the `Test` function

Go distinguishes between “test helpers” and “assertion helpers”:

Test helpers are functions that do setup or cleanup tasks. All failures that occur in test helpers are expected to be failures of the environment (not from the code under test) — for example when a test database cannot be started because there are no more free ports on this machine. For functions like these, calling t.Helper is often appropriate to mark them as a test helper. See error handling in test helpers for more details.
Assertion helpers are functions that check the correctness of a system and fail the test if an expectation is not met. Assertion helpers are not considered idiomatic in Go.

The purpose of a test is to report pass/fail conditions of the code under test. The ideal place to fail a test is within the Test function itself, as that ensures that failure messages and the test logic are clear.

As your testing code grows, it may become necessary to factor out some functionality to separate functions. Standard software engineering considerations still apply, as test code is still code. If the functionality does not interact with the testing framework, then all of the usual rules apply. When the common code interacts with the framework, however, some care must be taken to avoid common pitfalls that can lead to uninformative failure messages and unmaintainable tests.

If many separate test cases require the same validation logic, arrange the test in one of the following ways instead of using assertion helpers or complex validation functions:

Inline the logic (both the validation and the failure) in the Test function, even if it is repetitive. This works best in simple cases.
If inputs are similar, consider unifying them into a table-driven test while keeping the logic inlined in the loop. This helps to avoid repetition while keeping the validation and failure in the Test.
If there are multiple callers who need the same validation function but table tests are not suitable (typically because the inputs are not simple enough or the validation is required as part of a sequence of operations), arrange the validation function so that it returns a value (typically an error) rather than taking a testing.T parameter and using it to fail the test. Use logic within the Test to decide whether to fail, and to provide useful test failures. You can also create test helpers to factor out common boilerplate setup code.

The design outlined in the last point maintains orthogonality. For example, package cmp is not designed to fail tests, but rather to compare (and to diff) values. It therefore does not need to know about the context in which the comparison was made, since the caller can supply that. If your common testing code provides a cmp.Transformer for your data type, that can often be the simplest design. For other validations, consider returning an error value.

// Good:
// polygonCmp returns a cmp.Option that equates s2 geometry objects up to
// some small floating-point error.
func polygonCmp() cmp.Option {
    return cmp.Options{
        cmp.Transformer("polygon", func(p *s2.Polygon) []*s2.Loop { return p.Loops() }),
        cmp.Transformer("loop", func(l *s2.Loop) []s2.Point { return l.Vertices() }),
        cmpopts.EquateApprox(0.00000001, 0),
        cmpopts.EquateEmpty(),
    }
}

func TestFenceposts(t *testing.T) {
    // This is a test for a fictional function, Fenceposts, which draws a fence
    // around some Place object. The details are not important, except that
    // the result is some object that has s2 geometry (github.com/golang/geo/s2)
    got := Fencepost(tomsDiner, 1*meter)
    if diff := cmp.Diff(want, got, polygonCmp()); diff != "" {
        t.Errorf("Fencepost(tomsDiner, 1m) returned unexpected diff (-want+got):\n%v", diff)
    }
}

func FuzzFencepost(f *testing.F) {
    // Fuzz test (https://go.dev/doc/fuzz) for the same.

    f.Add(tomsDiner, 1*meter)
    f.Add(school, 3*meter)

    f.Fuzz(func(t *testing.T, geo Place, padding Length) {
        got := Fencepost(geo, padding)
        // Simple reference implementation: not used in prod, but easy to
        // reason about and therefore useful to check against in random tests.
        reference := slowFencepost(geo, padding)

        // In the fuzz test, inputs and outputs can be large so don't
        // bother with printing a diff. cmp.Equal is enough.
        if !cmp.Equal(got, reference, polygonCmp()) {
            t.Errorf("Fencepost returned wrong placement")
        }
    })
}

The polygonCmp function is agnostic about how it’s called; it doesn’t take a concrete input type nor does it police what to do in case two objects don’t match. Therefore, more callers can make use of it.

Note: There is an analogy between test helpers and plain library code. Code in libraries should usually not panic except in rare circumstances; code called from a test should not stop the test unless there is no point in proceeding.

Designing extensible validation APIs

Most of the advice about testing in the style guide is about testing your own code. This section is about how to provide facilities for other people to test the code they write to ensure that it conforms to your library’s requirements.

Acceptance testing

Such testing is referred to as acceptance testing. The premise of this kind of testing is that the person using the test does not know every last detail of what goes on in the test; they just hand the inputs over to the testing facility to do the work. This can be thought of as a form of inversion of control.

In a typical Go test, the test function controls the program flow, and the no assert and test functions guidance encourages you to keep it that way. This section explains how to author support for these tests in a way that is consistent with Go style.

Before diving into how, consider an example from io/fs, excerpted below:

type FS interface {
    Open(name string) (File, error)
}

While there exist well-known implementations of fs.FS, a Go developer may be expected to author one. To help validate the user-implemented fs.FS is correct, a generic library has been provided in testing/fstest called fstest.TestFS. This API treats the implementation as a blackbox to make sure it upholds the most basic parts of the io/fs contract.

Writing an acceptance test

Now that we know what an acceptance test is and why you might use one, let’s explore building an acceptance test for package chess, a package used to simulate chess games. Users of chess are expected to implement the chess.Player interface. These implementations are the primary thing we will validate. Our acceptance test concerns itself with whether the player implementation makes legal moves, not whether the moves are smart.

Create a new package for the validation behavior, customarily named by appending the word test to the package name (for example, chesstest).
Create the function that performs the validation by accepting the implementation under test as an argument and exercises it:
```
// ExercisePlayer tests a Player implementation in a single turn on a board.
// The board itself is spot checked for sensibility and correctness.
//
// It returns a nil error if the player makes a correct move in the context
// of the provided board. Otherwise ExercisePlayer returns one of this
// package's errors to indicate how and why the player failed the
// validation.
func ExercisePlayer(b *chess.Board, p chess.Player) error
```
The test should note which invariants are broken and how. Your design can choose between two disciplines for failure reporting:
- Fail fast: return an error as soon as the implementation violates an invariant.
  
  This is the simplest approach, and it works well if the acceptance test is expected to execute quickly. Simple error sentinels and custom types can be used easily here, which conversely makes testing the acceptance test easy.
```
for color, army := range b.Armies {
    // The king should never leave the board, because the game ends at
    // checkmate.
    if army.King == nil {
        return &MissingPieceError{Color: color, Piece: chess.King}
    }
}
```
- Aggregate all failures: collect all failures, and report them all.
  
  This approach resembles the keep going guidance in feel and may be preferable if the acceptance test is expected to execute slowly.
  
  How you aggregate the failures should be dictated by whether you want to give users the ability or yourself the ability to interrogate individual failures (for example, for you to test your acceptance test). Below demonstrates using a custom error type that aggregates errors:
```
var badMoves []error

move := p.Move()
if putsOwnKingIntoCheck(b, move) {
    badMoves = append(badMoves, PutsSelfIntoCheckError{Move: move})
}

if len(badMoves) > 0 {
    return SimulationError{BadMoves: badMoves}
}
return nil
```

The acceptance test should honor the keep going guidance by not calling t.Fatal unless the test detects a broken invariant in the system being exercised.

For example, t.Fatal should be reserved for exceptional cases such as setup failure as usual:

func ExerciseGame(t *testing.T, cfg *Config, p chess.Player) error {
    t.Helper()

    if cfg.Simulation == Modem {
        conn, err := modempool.Allocate()
        if err != nil {
            t.Fatalf("No modem for the opponent could be provisioned: %v", err)
        }
        t.Cleanup(func() { modempool.Return(conn) })
    }
    // Run acceptance test (a whole game).
}

This technique can help you create concise, canonical validations. But do not attempt to use it to bypass the guidance on assertions.

The final product should be in a form similar to this for end users:

// Good:
package deepblue_test

import (
    "chesstest"
    "deepblue"
)

func TestAcceptance(t *testing.T) {
    player := deepblue.New()
    err := chesstest.ExerciseGame(t, chesstest.SimpleGame, player)
    if err != nil {
        t.Errorf("Deep Blue player failed acceptance test: %v", err)
    }
}

Use real transports

When testing component integrations, especially where HTTP or RPC are used as the underlying transport between the components, prefer using the real underlying transport to connect to the test version of the backend.

For example, suppose the code you want to test (sometimes referred to as “system under test” or SUT) interacts with a backend that implements the long running operations API. To test your SUT, use a real OperationsClient that is connected to a test double (e.g., a mock, stub, or fake) of the OperationsServer.

This is recommended over hand-implementing the client, due to the complexity of imitating client behavior correctly. By using the production client with a test-specific server, you ensure your test is using as much of the real code as possible.

Tip: Where possible, use a testing library provided by the authors of the service under test.

`t.Error` vs. `t.Fatal`

As discussed in decisions, tests should generally not abort at the first encountered problem.

However, some situations require that the test not proceed. Calling t.Fatal is appropriate when some piece of test setup fails, especially in test setup helpers, without which you cannot run the rest of the test. In a table-driven test, t.Fatal is appropriate for failures that set up the whole test function before the test loop. Failures that affect a single entry in the test table, which make it impossible to continue with that entry, should be reported as follows:

If you’re not using t.Run subtests, use t.Error followed by a continue statement to move on to the next table entry.
If you’re using subtests (and you’re inside a call to t.Run), use t.Fatal, which ends the current subtest and allows your test case to progress to the next subtest.

Warning: It is not always safe to call t.Fatal and similar functions. More details here.

Error handling in test helpers

Note: This section discusses test helpers in the sense Go uses the term: functions that perform test setup and cleanup, not common assertion facilities. See the test functions section for more discussion.

Operations performed by a test helper sometimes fail. For example, setting up a directory with files involves I/O, which can fail. When test helpers fail, their failure often signifies that the test cannot continue, since a setup precondition failed. When this happens, prefer calling one of the Fatal functions in the helper:

// Good:
func mustAddGameAssets(t *testing.T, dir string) {
    t.Helper()
    if err := os.WriteFile(path.Join(dir, "pak0.pak"), pak0, 0644); err != nil {
        t.Fatalf("Setup failed: could not write pak0 asset: %v", err)
    }
    if err := os.WriteFile(path.Join(dir, "pak1.pak"), pak1, 0644); err != nil {
        t.Fatalf("Setup failed: could not write pak1 asset: %v", err)
    }
}

This keeps the calling side cleaner than if the helper were to return the error to the test itself:

// Bad:
func addGameAssets(t *testing.T, dir string) error {
    t.Helper()
    if err := os.WriteFile(path.Join(d, "pak0.pak"), pak0, 0644); err != nil {
        return err
    }
    if err := os.WriteFile(path.Join(d, "pak1.pak"), pak1, 0644); err != nil {
        return err
    }
    return nil
}

Warning: It is not always safe to call t.Fatal and similar functions. More details here.

The failure message should include a description of what happened. This is important, as you may be providing a testing API to many users, especially as the number of error-producing steps in the helper increases. When the test fails, the user should know where, and why.

Tip: Go 1.14 introduced a t.Cleanup function that can be used to register cleanup functions that run when your test completes. The function also works with test helpers. See GoTip #4: Cleaning Up Your Tests for guidance on simplifying test helpers.

The snippet below in a fictional file called paint_test.go demonstrates how (*testing.T).Helper influences failure reporting in a Go test:

package paint_test

import (
    "fmt"
    "testing"
)

func paint(color string) error {
    return fmt.Errorf("no %q paint today", color)
}

func badSetup(t *testing.T) {
    // This should call t.Helper, but doesn't.
    if err := paint("taupe"); err != nil {
        t.Fatalf("Could not paint the house under test: %v", err) // line 15
    }
}

func goodSetup(t *testing.T) {
    t.Helper()
    if err := paint("lilac"); err != nil {
        t.Fatalf("Could not paint the house under test: %v", err)
    }
}

func TestBad(t *testing.T) {
    badSetup(t)
    // ...
}

func TestGood(t *testing.T) {
    goodSetup(t) // line 32
    // ...
}

Here is an example of this output when run. Note the highlighted text and how it differs:

=== RUN   TestBad
    paint_test.go:15: Could not paint the house under test: no "taupe" paint today
--- FAIL: TestBad (0.00s)
=== RUN   TestGood
    paint_test.go:32: Could not paint the house under test: no "lilac" paint today
--- FAIL: TestGood (0.00s)
FAIL

The error with paint_test.go:15 refers to the line of the setup function that failed in badSetup:

t.Fatalf("Could not paint the house under test: %v", err)

Whereas paint_test.go:32 refers to the line of the test that failed in TestGood:

goodSetup(t)

Correctly using (*testing.T).Helper attributes the location of the failure much better when:

the helper functions grow
the helper functions call other helpers
the amount of helper usage in the test functions grow

Tip: If a helper calls (*testing.T).Error or (*testing.T).Fatal, provide some context in the format string to help determine what went wrong and why.

Tip: If nothing a helper does can cause a test to fail, it doesn’t need to call t.Helper. Simplify its signature by removing t from the function parameter list.

Don’t call `t.Fatal` from separate goroutines

As documented in package testing, it is incorrect to call t.FailNow, t.Fatal, etc. from any goroutine but the one running the Test function (or the subtest). If your test starts new goroutines, they must not call these functions from inside these goroutines.

Test helpers usually don’t signal failure from new goroutines, and therefore it is all right for them to use t.Fatal. If in doubt, call t.Error and return instead.

// Good:
func TestRevEngine(t *testing.T) {
    engine, err := Start()
    if err != nil {
        t.Fatalf("Engine failed to start: %v", err)
    }

    num := 11
    var wg sync.WaitGroup
    wg.Add(num)
    for i := 0; i < num; i++ {
        go func() {
            defer wg.Done()
            if err := engine.Vroom(); err != nil {
                // This cannot be t.Fatalf.
                t.Errorf("No vroom left on engine: %v", err)
                return
            }
            if rpm := engine.Tachometer(); rpm > 1e6 {
                t.Errorf("Inconceivable engine rate: %d", rpm)
            }
        }()
    }
    wg.Wait()

    if seen := engine.NumVrooms(); seen != num {
        t.Errorf("engine.NumVrooms() = %d, want %d", seen, num)
    }
}

Adding t.Parallel to a test or subtest does not make it unsafe to call t.Fatal.

When all calls to the testing API are in the test function, it is usually easy to spot incorrect usage because the go keyword is plain to see. Passing testing.T arguments around makes tracking such usage harder. Typically, the reason for passing these arguments is to introduce a test helper, and those should not depend on the system under test. Therefore, if a test helper registers a fatal test failure, it can and should do so from the test’s goroutine.

Use field names in struct literals

In table-driven tests, prefer to specify field names when initializing test case struct literals. This is helpful when the test cases cover a large amount of vertical space (e.g. more than 20-30 lines), when there are adjacent fields with the same type, and also when you wish to omit fields which have the zero value. For example:

// Good:
func TestStrJoin(t *testing.T) {
    tests := []struct {
        slice     []string
        separator string
        skipEmpty bool
        want      string
    }{
        {
            slice:     []string{"a", "b", ""},
            separator: ",",
            want:      "a,b,",
        },
        {
            slice:     []string{"a", "b", ""},
            separator: ",",
            skipEmpty: true,
            want:      "a,b",
        },
        // ...
    }
    // ...
}

Keep setup code scoped to specific tests

Where possible, setup of resources and dependencies should be as closely scoped to specific test cases as possible. For example, given a setup function:

// mustLoadDataSet loads a data set for the tests.
//
// This example is very simple and easy to read. Often realistic setup is more
// complex, error-prone, and potentially slow.
func mustLoadDataset(t *testing.T) []byte {
    t.Helper()
    data, err := os.ReadFile("path/to/your/project/testdata/dataset")

    if err != nil {
        t.Fatalf("Could not load dataset: %v", err)
    }
    return data
}

Call mustLoadDataset explicitly in test functions that need it:

// Good:
func TestParseData(t *testing.T) {
    data := mustLoadDataset(t)
    parsed, err := ParseData(data)
    if err != nil {
        t.Fatalf("Unexpected error parsing data: %v", err)
    }
    want := &DataTable{ /* ... */ }
    if got := parsed; !cmp.Equal(got, want) {
        t.Errorf("ParseData(data) = %v, want %v", got, want)
    }
}

func TestListContents(t *testing.T) {
    data := mustLoadDataset(t)
    contents, err := ListContents(data)
    if err != nil {
        t.Fatalf("Unexpected error listing contents: %v", err)
    }
    want := []string{ /* ... */ }
    if got := contents; !cmp.Equal(got, want) {
        t.Errorf("ListContents(data) = %v, want %v", got, want)
    }
}

func TestRegression682831(t *testing.T) {
    if got, want := guessOS("zpc79.example.com"), "grhat"; got != want {
        t.Errorf(`guessOS("zpc79.example.com") = %q, want %q`, got, want)
    }
}

The test function TestRegression682831 does not use the data set and therefore does not call mustLoadDataset, which could be slow and failure-prone:

// Bad:
var dataset []byte

func TestParseData(t *testing.T) {
    // As documented above without calling mustLoadDataset directly.
}

func TestListContents(t *testing.T) {
    // As documented above without calling mustLoadDataset directly.
}

func TestRegression682831(t *testing.T) {
    if got, want := guessOS("zpc79.example.com"), "grhat"; got != want {
        t.Errorf(`guessOS("zpc79.example.com") = %q, want %q`, got, want)
    }
}

func init() {
    dataset = mustLoadDataset()
}

A user may wish to run a function in isolation of the others and should not be penalized by these factors:

# No reason for this to perform the expensive initialization.
$ go test -run TestRegression682831

When to use a custom `TestMain` entrypoint

If all tests in the package require common setup and the setup requires teardown, you can use a custom testmain entrypoint. This can happen if the resource the test cases require is especially expensive to setup, and the cost should be amortized. Typically you have extracted any unrelated tests from the test suite at that point. It is typically only used for functional tests.

Using a custom TestMain should not be your first choice due the amount of care that should be taken for correct use. Consider first whether the solution in the amortizing common test setup section or an ordinary test helper is sufficient for your needs.

// Good:
var db *sql.DB

func TestInsert(t *testing.T) { /* omitted */ }

func TestSelect(t *testing.T) { /* omitted */ }

func TestUpdate(t *testing.T) { /* omitted */ }

func TestDelete(t *testing.T) { /* omitted */ }

// runMain sets up the test dependencies and eventually executes the tests.
// It is defined as a separate function to enable the setup stages to clearly
// defer their teardown steps.
func runMain(ctx context.Context, m *testing.M) (code int, err error) {
    ctx, cancel := context.WithCancel(ctx)
    defer cancel()

    d, err := setupDatabase(ctx)
    if err != nil {
        return 0, err
    }
    defer d.Close() // Expressly clean up database.
    db = d          // db is defined as a package-level variable.

    // m.Run() executes the regular, user-defined test functions.
    // Any defer statements that have been made will be run after m.Run()
    // completes.
    return m.Run(), nil
}

func TestMain(m *testing.M) {
    code, err := runMain(context.Background(), m)
    if err != nil {
        // Failure messages should be written to STDERR, which log.Fatal uses.
        log.Fatal(err)
    }
    // NOTE: defer statements do not run past here due to os.Exit
    //       terminating the process.
    os.Exit(code)
}

Ideally a test case is hermetic between invocations of itself and between other test cases.

At the very least, ensure that individual test cases reset any global state they have modified if they have done so (for instance, if the tests are working with an external database).

Amortizing common test setup

Using a sync.Once may be appropriate, though not required, if all of the following are true about the common setup:

It is expensive.
It only applies to some tests.
It does not require teardown.

// Good:
var dataset struct {
    once sync.Once
    data []byte
    err  error
}

func mustLoadDataset(t *testing.T) []byte {
    t.Helper()
    dataset.once.Do(func() {
        data, err := os.ReadFile("path/to/your/project/testdata/dataset")
        // dataset is defined as a package-level variable.
        dataset.data = data
        dataset.err = err
    })
    if err := dataset.err; err != nil {
        t.Fatalf("Could not load dataset: %v", err)
    }
    return dataset.data
}

When mustLoadDataset is used in multiple test functions, its cost is amortized:

// Good:
func TestParseData(t *testing.T) {
    data := mustLoadDataset(t)

    // As documented above.
}

func TestListContents(t *testing.T) {
    data := mustLoadDataset(t)

    // As documented above.
}

func TestRegression682831(t *testing.T) {
    if got, want := guessOS("zpc79.example.com"), "grhat"; got != want {
        t.Errorf(`guessOS("zpc79.example.com") = %q, want %q`, got, want)
    }
}

The reason that common teardown is tricky is there is no uniform place to register cleanup routines. If the setup function (in this case mustLoadDataset) relies on a context, sync.Once may be problematic. This is because the second of two racing calls to the setup function would need to wait for the first call to finish before returning. This period of waiting cannot be easily made to respect the context’s cancellation.

Tests

Tests

Leave testing to the Test function

Designing extensible validation APIs

Acceptance testing

Writing an acceptance test

Use real transports

t.Error vs. t.Fatal

Error handling in test helpers

Don’t call t.Fatal from separate goroutines

Use field names in struct literals

Keep setup code scoped to specific tests

When to use a custom TestMain entrypoint

Amortizing common test setup

Leave testing to the `Test` function

`t.Error` vs. `t.Fatal`

Don’t call `t.Fatal` from separate goroutines

When to use a custom `TestMain` entrypoint