TheLazy.Dev
Portfolio GitHub

Blog

go |afero |testing |mocking

Mocking the filesystem in Go using Afero

drpaneas

Panagiotis Georgiadis

Posted on December 16, 2020

Mocking the filesystem in Go using Afero

Let's start the project

cd $GOPATH/src/github.com/drpaneas
mkdir checkfile; cd checkfile
git init -q
go mod init
touch main.go
code . # for vscode
Enter fullscreen mode Exit fullscreen mode

Coding Task

Write a function that checks if a specific directory exists on disk. For example:

if folderExists(directory) {
  // do something
}
Enter fullscreen mode Exit fullscreen mode

Bonus: Feel free to add fileExists(filename) as homework.

Acceptance Criteria

Hints:

  • Writing a unit test for filesystem checking is not trivial. You should NOT create a real file on the system, because then your test will depend on the I/O of the file system itself. The last resort is mocking the filesystem. There are quite a few powerful libraries like spf13/afero for this purpose (mocking of a filesystem). These packages will create temporary files in the background and clean up afterward.
  • As an exercise, try to write the test first. Let it fail. Then try to fix it, by writing the actual implementation of the function.

Extra Help

NOTICE:

If you are unable to mock this test, in worst case scenario, create a folder pkg/utils/testdata and put actual files you can use for your test.

Let's code 

// main.go
package main

import (
    "os"

    log "github.com/sirupsen/logrus"
)

// exists reports whether the named file or directory exists
func exists(path string, isDir bool) bool {
    if path == "" {
        log.Debug("Path is empty")
        return false
    }

    info, err := os.Stat(path)
    if err != nil {
        if os.IsNotExist(err) { // look for the specific error type
            return false
        }
    }

    return isDir == info.IsDir()
}

// FolderExists reports whether the provided directory exists.
func FolderExists(path string) bool {
    return exists(path, true)
}

// FileExists reports whether the provided directory exists.
func FileExists(path string) bool {
    return exists(path, false)
}


func main() {
    // Just for demo purposes
    if FolderExists("logs") {
        log.Info("Folder exists")
    } else {
        log.Error("Folder does not exist")
    }

    if FileExists("helloworld.txt") {
        log.Info("File exists")
    } else {
        log.Error("File does not exist")
    }
}
Enter fullscreen mode Exit fullscreen mode

One way to test this is by actually using the program itself.
Running it as it is, it's expected to fail:

ERRO[0000] Folder does not exist                        
ERRO[0000] Folder does not exist
Enter fullscreen mode Exit fullscreen mode

If we create the two necessary things we look for, then it's expected to work:

# Create the file and directory
touch helloworld.txt
mkdir logs

# Run it again
go run main.go
INFO[0000] Folder exists                                
INFO[0000] File exists
Enter fullscreen mode Exit fullscreen mode

Normally you would create separate package and not use main.
So let's do it!

mkdir -p pkg/utils/
touch pkg/utils/filesystem.go
Enter fullscreen mode Exit fullscreen mode

The code will look like this:

// main.go
// -------

package main

import (
    "github.com/drpaneas/checkfile/pkg/utils"
    log "github.com/sirupsen/logrus"
)

func main() {
    if utils.FolderExists("logs") {
        log.Info("Folder exists")
    } else {
        log.Error("Folder does not exist")
    }

    if utils.FileExists("helloworld.txt") {
        log.Info("File exists")
    } else {
        log.Error("File does not exist")
    }
}
Enter fullscreen mode Exit fullscreen mode 
// pkg/utils/filesystem.go
// -----------------------

package utils

import (
    "os"

    log "github.com/sirupsen/logrus"
)

// exists reports whether the named file or directory exists
func exists(path string, isDir bool) bool {
    if path == "" {
        log.Debug("Path is empty")
        return false
    }

    info, err := os.Stat(path)
    if err != nil {
        if os.IsNotExist(err) { // look for the specific error type
            return false
        }
    }

    return isDir == info.IsDir()
}

// FolderExists reports whether the provided directory exists.
func FolderExists(path string) bool {
    return exists(path, true)
}

// FileExists reports whether the provided directory exists.
func FileExists(path string) bool {
    return exists(path, false)
}
Enter fullscreen mode Exit fullscreen mode

Just to be on the safe side we didn't forget anything major, let's run it again to make sure everything is alright.

# Run it again
go run main.go
INFO[0000] Folder exists                                
INFO[0000] File exists
Enter fullscreen mode Exit fullscreen mode

Test with testdata

Ok,we need to write tests for pkg/utils/filesystem.go.
One way of doing this is by accessing a real file system to do testing, then we're talking about what you'd call system or integration testing. That's fine in itself: for instance, you could run such tests in a throw-away container as part of a CI pipeline.
But with this approach you can't sensibly do what is called unit-testing.

To proceed with that, we are going to use the generated boilerplate code that vscode happily offers (give you have the Go extensions installed).

Place the mouse cursor over the FolderExists() function and press the keys: CTRL+SHIFT+P.
From the drop-down menu, type: Go: test and select the one for the function.
This will create the filesystem_test.go file and populate it with a table test.
Add your test logic there:

// pkg/utils/filesystem_test.go
// ----------------------------

package utils

import "testing"

// Add those variables as well
var (
    existingFolder    = "./testdata/a-folder-that-exists"
    nonExistingFolder = "./testdata/a-folder-that-does-not-exists"
)

func TestFolderExists(t *testing.T) {
    type args struct {
        path string
    }
    tests := []struct {
        name string
        args args
        want bool
    }{
        // ----- test logic start ----- //
        {
            "Returns true when given folder exists",
            args{path: existingFolder},
            true,
        },
        {
            "Returns false when given folder does not exist",
            args{path: nonExistingFolder},
            false,
        },
        // ----- test logic end ----- //
    }
    for _, tt := range tests {
        t.Run(tt.name, func(t *testing.T) {
            if got := FolderExists(tt.args.path); got != tt.want {
                t.Errorf("FolderExists() = %v, want %v", got, tt.want)
            }
        })
    }
}
Enter fullscreen mode Exit fullscreen mode

The tests is relying upon real data on the actual hard drive.
Run tests and you will see they will fail:

go test ./... -v
?       github.com/drpaneas/checkfile   [no test files]
=== RUN   TestFolderExists
=== RUN   TestFolderExists/Returns_true_when_given_folder_exists
    filesystem_test.go:33: FolderExists() = false, want true
=== RUN   TestFolderExists/Returns_false_when_given_folder_does_not_exist
--- FAIL: TestFolderExists (0.00s)
    --- FAIL: TestFolderExists/Returns_true_when_given_folder_exists (0.00s)
    --- PASS: TestFolderExists/Returns_false_when_given_folder_does_not_exist (0.00s)
FAIL
FAIL    github.com/drpaneas/checkfile/pkg/utils 0.098s
FAIL
Enter fullscreen mode Exit fullscreen mode

Create the necessary testdata and try again:

mkdir -p pkg/utils/testdata/a-folder-that-exists
Enter fullscreen mode Exit fullscreen mode 
$ go test ./... -v?       github.com/drpaneas/checkfile   [no test files]
=== RUN   TestFolderExists
=== RUN   TestFolderExists/Returns_true_when_given_folder_exists
=== RUN   TestFolderExists/Returns_false_when_given_folder_does_not_exist
--- PASS: TestFolderExists (0.00s)
    --- PASS: TestFolderExists/Returns_true_when_given_folder_exists (0.00s)
    --- PASS: TestFolderExists/Returns_false_when_given_folder_does_not_exist (0.00s)
PASS
ok      github.com/drpaneas/checkfile/pkg/utils (cached)
Enter fullscreen mode Exit fullscreen mode

Now, see the coverage. Go into the package and run cover.

 cover
PASS
coverage: 77.8% of statements
ok      github.com/drpaneas/checkfile/pkg/utils 0.293s
Enter fullscreen mode Exit fullscreen mode

PS: You need to have this snippet into your ~/.bashrc:

# Go coverage trick
cover () {
    t="/tmp/go-cover.$$.tmp"
    go test -coverprofile=$t $@ && go tool cover -html=$t && unlink $t
}
Enter fullscreen mode Exit fullscreen mode

Let's enhance the testing coverage, by adding:

        {
            "Returns false when provided path is empty",
            args{path: ""},
            false,
        },
Enter fullscreen mode Exit fullscreen mode

Run the tests again and the coverage:

go test ./...

$ cover ./pkg/utils
ok      github.com/drpaneas/checkfile/pkg/utils 0.232s  coverage: 88.9% of statements
Enter fullscreen mode Exit fullscreen mode

Add the tests for the other function as well:

The variables first:

var (
    existingFile      = "./testdata/a-file-that-exists"
    nonExistingFile   = "./testdata/a-file-that-doesn-not-exist"
)
Enter fullscreen mode Exit fullscreen mode

The test code:

        {
            "Returns true when given file exists",
            args{path: existingFile},
            true,
        },
        {
            "Returns false when given file does not exist",
            args{path: nonExistingFile},
            false,
        },
        {
            "Returns false when provided path is empty",
            args{path: ""},
            false,
        },
Enter fullscreen mode Exit fullscreen mode

And create the appropriate file for testing:

touch pkg/utils/testdata/a-file-that-exists
Enter fullscreen mode Exit fullscreen mode

Run the tests (they should all pass):

$ go test -v ./...?       github.com/drpaneas/checkfile   [no test files]
=== RUN   TestFolderExists
=== RUN   TestFolderExists/Returns_true_when_given_folder_exists
=== RUN   TestFolderExists/Returns_false_when_given_folder_does_not_exist
=== RUN   TestFolderExists/Returns_false_when_provided_path_is_empty
--- PASS: TestFolderExists (0.00s)
    --- PASS: TestFolderExists/Returns_true_when_given_folder_exists (0.00s)
    --- PASS: TestFolderExists/Returns_false_when_given_folder_does_not_exist (0.00s)
    --- PASS: TestFolderExists/Returns_false_when_provided_path_is_empty (0.00s)
=== RUN   TestFileExists
=== RUN   TestFileExists/Returns_true_when_given_file_exists
=== RUN   TestFileExists/Returns_false_when_given_file_does_not_exist
=== RUN   TestFileExists/Returns_false_when_provided_path_is_empty
--- PASS: TestFileExists (0.00s)
    --- PASS: TestFileExists/Returns_true_when_given_file_exists (0.00s)
    --- PASS: TestFileExists/Returns_false_when_given_file_does_not_exist (0.00s)
    --- PASS: TestFileExists/Returns_false_when_provided_path_is_empty (0.00s)
PASS
ok      github.com/drpaneas/checkfile/pkg/utils 0.301s
Enter fullscreen mode Exit fullscreen mode

Check the testing coverage:

cover ./pkg/utils
ok      github.com/drpaneas/checkfile/pkg/utils 0.268s  coverage: 100.0% of statements
Enter fullscreen mode Exit fullscreen mode

Although we already hit the 100% testing coverage, I would like to add another test-case.

Add these two code block respectively:

// TestFolderExists()
// ...
        {
            "Returns false when provided path is not a directory",
            args{path: existingFile},
            false,
        },
Enter fullscreen mode Exit fullscreen mode 
// TestFileExists
// ...
        {
            "Returns false when provided path is not a file",
            args{path: existingFolder},
            false,
        },
Enter fullscreen mode Exit fullscreen mode

... and that's it!
Run again the tests to verify everything works and we are good to go.

go test ./...
?       github.com/drpaneas/checkfile   [no test files]
ok      github.com/drpaneas/checkfile/pkg/utils 0.332s
Enter fullscreen mode Exit fullscreen mode

Test without testdata

Ok now let's do the same without the actual testdata.
To unit-test your function you need to replace something it uses with something "virtualised".
Exactly how to do that is an open question.
Right now, a work is being done on bringing filesystem virtualisation right into the Go standard library, but while it's not there yet, there's a 3rd-party package which has been doing that for ages, — for instance, github.com/spf13/afero.

First let's remove the actual real data, and see the tests fail:

rm -r ./pkg/utils/testdata
Enter fullscreen mode Exit fullscreen mode

So now the tests are failing obviously:

go test ./...
?       github.com/drpaneas/checkfile   [no test files]
--- FAIL: TestFolderExists (0.00s)
    --- FAIL: TestFolderExists/Returns_true_when_given_folder_exists (0.00s)
        filesystem_test.go:47: FolderExists() = false, want true
--- FAIL: TestFileExists (0.00s)
    --- FAIL: TestFileExists/Returns_true_when_given_file_exists (0.00s)
        filesystem_test.go:86: FileExists() = false, want true
FAIL
FAIL    github.com/drpaneas/checkfile/pkg/utils 0.404s
FAIL
Enter fullscreen mode Exit fullscreen mode

To do that we will use Afero.
Writing a unit test for filesystem checking is not trivial. You should NOT create a real file on the system, because then your test will depend on the I/O of the file system itself.

Let's modify our code to use Afero for both implementation and test.
So for the filesystem.go just add:

// Making the global, sharing the memory filesystem
var (
    FS  afero.Fs     = afero.NewMemMapFs()
    AFS *afero.Afero = &afero.Afero{Fs: FS}
)
Enter fullscreen mode Exit fullscreen mode

and replace the os.Stat to AFS.Stat. That's all.
For the filesystem_test.go just add:

func init() {
    // create test files and directories
    AFS.MkdirAll(existingFolder, 0755)
    afero.WriteFile(AFS, existingFile, []byte("test file"), 0755)
}
Enter fullscreen mode Exit fullscreen mode

Run the tests and they should pass:

go test ./... -v
?       github.com/drpaneas/checkfile   [no test files]
=== RUN   TestFolderExists
=== RUN   TestFolderExists/Returns_true_when_given_folder_exists
=== RUN   TestFolderExists/Returns_false_when_given_folder_does_not_exist
=== RUN   TestFolderExists/Returns_false_when_provided_path_is_empty
=== RUN   TestFolderExists/Returns_false_when_provided_path_is_not_a_directory
--- PASS: TestFolderExists (0.00s)
    --- PASS: TestFolderExists/Returns_true_when_given_folder_exists (0.00s)
    --- PASS: TestFolderExists/Returns_false_when_given_folder_does_not_exist (0.00s)
    --- PASS: TestFolderExists/Returns_false_when_provided_path_is_empty (0.00s)
    --- PASS: TestFolderExists/Returns_false_when_provided_path_is_not_a_directory (0.00s)
=== RUN   TestFileExists
=== RUN   TestFileExists/Returns_true_when_given_file_exists
=== RUN   TestFileExists/Returns_false_when_given_file_does_not_exist
=== RUN   TestFileExists/Returns_false_when_provided_path_is_empty
=== RUN   TestFileExists/Returns_false_when_provided_path_is_not_a_file
--- PASS: TestFileExists (0.00s)
    --- PASS: TestFileExists/Returns_true_when_given_file_exists (0.00s)
    --- PASS: TestFileExists/Returns_false_when_given_file_does_not_exist (0.00s)
    --- PASS: TestFileExists/Returns_false_when_provided_path_is_empty (0.00s)
    --- PASS: TestFileExists/Returns_false_when_provided_path_is_not_a_file (0.00s)
PASS
ok      github.com/drpaneas/checkfile/pkg/utils 0.349s
Enter fullscreen mode Exit fullscreen mode

Notice we have no testdata any more.
Basically, Afero allows you to not use os directly but write all your code in a way so that instead of the os package it calls methods on an instance of a type implementing particular interface: in the production code that object is a thin shim for the os package, and in the testing code it's replaced by whatever you wish; afero has a readily-available in-memory FS backend to do this.

To sum up, although it's a good idea to do this, there might be cases where Afero fails to offer you this isolation.
For example if you try to do anything on top of the basic filesystem usage, then Afero might not be enough.
For example, if you try to extract, zip or compress a fail, you will see that those functions are not compatible with Afero, thus you will have either to send a PR to afero or suffer by creating lot's of extra code for achieving that.

💖 💪 🙅 🚩
drpaneas
Panagiotis Georgiadis

Posted on December 16, 2020

Join Our Newsletter. No Spam, Only the good stuff.

Sign up to receive the latest update from our blog.

Related

Mocking the filesystem in Go using Afero
go Mocking the filesystem in Go using Afero

December 16, 2020