I developed a version control software for graphic designers and 2D/3D artists called Snowtrack in Angular and Electron. In this blog post, I will cover some technical challenges about file locks which I faced during the development of Snowtrack.

What is Snowtrack?

Snowtrack is an intuitive, easy-to-use, and super-fast version control software for graphic projects. Its purpose is to make version control accessible to graphic designers and 2D/3D artists with a non-technical workflow.

To get a better understanding of Snowtracks user interface check out the following screenshot:

What I used to build it

For the UI application I used a combination of Angular and Electron. The underlying version control engine is called SnowFS, an open-source project I developed as a fast and simple alternative to Git and Git LFS. Feel free to check it out on GitHub. A few months ago I wrote a blog post about it here on dev.to.

Technical challenge no. 1

Graphic projects can differ in size tremendously. From a single Photoshop file up to a 50 GB file set of 3D scenes, textures, and assets. These project types come with their own set of problems. In the following I want to clear up some misconceptions about the topic around file locking.

File Locking

Take a look at the code snippet below.

// Process 1
fd = fs.openSync("~/foo", "w");

// Process 2
fd = fs.openSync("~/foo", "w");

Imagine more than one process wants to open the same file at the same time. What do you think will happen?

Answer: It depends on the OS and if you're the maintainer of all processes.

When you call fs.openSync NodeJS will forward the call behind the scenes to an OS function as you can see from this C code

static ssize_t uv__fs_open(uv_fs_t* req) {
  return open(req->path, req->flags | O_CLOEXEC, req->mode);
}

The function open(..) is an OS function and available in all operating systems. But the internals of this function differ between Windows, Linux and macOS so I will cover them separately.

macOS/Linux

Technically, neither macOS nor Linux have true file-locking mechanisms. Although you can read or write-lock a file using a function called fcntl, only programs which use this function regard and respect the file lock. This means, any other process which doesn't use fcntl and directly wants to open a file can acquire a file handle and manipulate the content as long as the file permissions allow it. What a bummer.

That's why file locking on macOS and Linux is also called "advisory file locking".

Windows

Windows is more complicated in that matter. Windows offers two functions to open a file. Either through the Windows API function called CreateFile (yes, that's really the name to open files),...

...or through open(..). But the open(..) function on Windows is a POSIX extension and uses CreateFile internally as well.

As we've seen above NodeJS uses open(..), but since we know that this is just a wrapper for CreateFile, let's check out that function:

// The low-level open function of Windows.
HANDLE CreateFile(
  LPCSTR                lpFileName,
  DWORD                 dwDesiredAccess,
  DWORD                 dwShareMode,
  LPSECURITY_ATTRIBUTES lpSecurityAttributes,
  DWORD                 dwCreationDisposition,
  DWORD                 dwFlagsAndAttributes,
  HANDLE                hTemplateFile
);

CreateFile has a parameter called dwShareMode. A file that is opened with dwShareMode=0 cannot be opened again until its handle has been closed.

So if you use open(..) on a file that was already open by another process with CreateFile(…, dwShareMode=0) you receive this error message:

The process cannot access the file because it is being used by another process

On the other hand, if you use fs.openSync in NodeJS, or open(..) in C/C++, to open a file that hasn't been opened yet, you cannot prevent another application from modifying it*.

* Unless you you use file permissions as a workaround, but that’s not really a file lock.

To prove this, you will see that our fs.openSync call executes CreateFile with the read/write shared flags to comply with the POSIX standard.

This means on Windows you cannot prevent another application from opening and modifying your file if you don't use CreateFile.

What does this have to do with Snowtrack?

Imagine a user saving a big file in a graphic application and while the file is still being written to disk, the user attempts to commit the file change. How does Snowtrack deal with this?

As we learned, open(..) has no file locking and most applications don't even follow the file protocol and Snowtrack cannot control how Photoshop, Blender, and co. open and write their files.

This means the only reliable chance of detecting if a file is still being written by another process is to check prior to a commit if any process on the system has a write handle on that file.

• On Windows, I solved this with a custom helper process and and the Windows API of Restart Manager which is mainly used for installers to ensure the files it is about to replace are not open anymore.

• On MacOS I invoke the system process /usr/sbin/lsof (list open files) with an inclusion of the working-directory to speed up the execution of this command.

What else?

The development of Snowtrack came with countless technical challenges and I would be happy to share more insights.

File locking, Electron/Angular race conditions, I/O saturation, build server, update mechanisms, edge cases, .. with this project I touched many subjects and I would be happy to write a follow-up blog post if you are interested. Let me know in the comments below.

If you want to support SnowFS, Snowtrack or me then feel free to join me on Twitter.

Thanks for reading :-)

TLDR

Don't get me started on file-locking.

Addendum: What about the "File In Use" dialog in Windows?

If you are a Windows user you might have seen this error message before:

Windows, or rather NTFS, behaves very different compared to other file systems like HFS+, APFS, ext3, ...

There is no equivalent to inodes in NTFS and therefore no garbage collection deletes the file if the last file handle to an already deleted file is closed. The File in Use dialog only indicates, that if any process has a file handle to a given file (no matter how it got opened), it cannot be renamed, moved, or deleted. That does not imply a file lock on the file content.