The Basics of Dependency Maintenance in NPM/yarn
Thor Galle
Posted on June 7, 2021
Think back: you're about to start contributing to a web project. You clone it, run npm install, and then... you get one error after the other. You can probably name at least one of these experiences. In the worst case, this can lead us to abandon our plans of contribution. The bad news is that this bad experience is ubiquitous. The good news is that it is entirely avoidable, and it's not even that hard to avoid!
This post is a collection of best practices regarding dependency maintenance in NPM, they help you keep your project in a healthy state so that if you ever have to hand it over, invite collaborators, or revisit it, it won't be an absolute pain in the ***.
The tips
- Make sure you understand Semantic Versioning
- Use & commit the lockfile
- Update dependencies regularly
- Take on less dependencies
(clicking will lead you to the specific section)
I wrote this as a synthesis of my experience maintaining a Nuxt.js website project for two years at Columbia Road, and working on several other projects in a sub-optimal state. It is intended for readers with some general JS-stack web development experience, but with little experience maintaining a web project.
Note that the focus here lies on the consumption side: the managing and updating of dependencies in a web project you're working on. This is not discussing how to maintain an NPM package that you have published, although there should be some overlap in best practices. Basic familiarity with git is assumed.
1. Make sure you understand Semantic Versioning
First things first: whether you're installing dependencies, updating dependencies, or you're publishing your own package and making changes to it, understanding semantic versioning (semver) is essential.
Most software projects today use semver to tag versions of their program releases (eg. v1.2.15). The versions have three numeric parts: major.minor.patch. The major version should be increased by one ('bumped') when the software interface of the dependency has breaking changes (which means: your code will break or behave differently if you update the dependency without changing your code). Even when the breaking change is seemingly small and simple, like a changed function name, the major version should have been bumped. If package publishers do not respect this, it can easily lead to problems when people consuming those packages update their dependencies: they end up installing incompatible dependency code!
Another important realization is that semver defines several range types of versions, that is, saying that any version that is included in a certain range of versions is OK to install as a dependency. The caret range (~) in particular is the default version notation used when you run npm install some-package to add a new package to your project (thus, in your package.json). It mostly allows variations in the minor and patch versions, which is usually safe. However, its exact definition is important to check out, as there is an exception that allows more versions than you might expect!
2. Use & commit the lockfile
Both NPM and Yarn have had a system a lockfile for some time now: package-lock.jsonin NPM or yarn.lock in Yarn. But what does this file do?
This file keeps track of the exact versions of all your dependencies and their sub-dependencies. Concretely, it lists which dependencies are stored in your node_modules folders at this moment.
This is very useful, because another developer with the same lockfile can install exactly the same dependency tree on a fresh npm install. Without a lockfile in place, different dependency versions could be installed at different times despite being installed from the same package.json. The reason for this is that "desired versions" in package.json are often specified as a relatively loose range, such as the caret range discussed before.
The problem with having a different (newer) version of a dependency than another team member, for example 3.7.24 instead of 3.5.2, is that it always carries the risk some changed behavior that breaks your code in one way or another.
Commit your lockfile so that everyone shares access to it, and changes to the dependency tree are tracked in git. This will even enable you to time-travel (roll back) to the exact state of an older version of your project via git.
Note also that in general, you should avoid mixing the usage of npm and yarn in your team, because using two different lockfiles may lead to problems where you end up installing different package versions despite having a lockfile. The risks of having such issues have decreased over the last years (npm now also reads yarn.lock, which it didn't before), but even if it's just for consistency & simplicity, picking one is better.
Read more on lockfiles in the NPM docs.
3. Update dependencies regularly
Check for new dependency updates regularly, for example, every 2-3 months, depending on how frequently you work on your project.
Why not just stick with what works now?
Not updating your dependencies means incurring technical debt. This debt will make it progressively harder and costlier to update later, when you really need to.
For the example's sake: let's say you didn't update dependencies for two years, this can can cause major headaches. Here's why:
- The JS ecosystem moves fast. Some of your dependencies might already be deprecated in two years time!
- It may become harder to update the dependencies, because dependencies typically make assumptions about their environment or other ('peer') dependencies (for example,
sass-loaderis built for a specific Node.js and webpack version). The latest version of one dependency might not fit in your old, potentially deprecated, environment anymore, causing issues. - Changelogs and migration guides become progressively harder to find as time advances. Guides to update to a new major version are sometimes stored on a project's website, which might lose history quicker than git. It then requires detective-level work (e.g., the Internet Archive) to find back these migration guides. And when finding them, they may be outdated. I had this issue when updating from Nuxt 1.4 to Nuxt 2, which had been released a year before I did the update.
- Community support is better for fresh issues. When developers notice an issue in a fresh update, they usually file a GitHub issue on a project. This makes it possible to find quick workarounds and solutions that you can apply as a community. But if you research issues two years later; A) chances are no one cares anymore to help with a two-year old issue. B) It may be harder to find old solutions amongst a mess of other search results: search engines seem to prioritize more recent search hits.
Now imagine that your product has a critical security flaw caused by one of its dependencies. To fix it you might encounter any of the above headaches. In the worst case, you have to do heavy refactoring or simply restart your project.
Hence, spending maintenance time to keep things up-to-date is probably cheaper in the long run. Here's a simple routine of how to do it:
Dependency updating routine
Use this routine to update your dependencies:
- Run
npm outdatedor similar to know which dependencies are outdated.- "Wanted" describes the maximum version of a package that is allowed by the range in your
package.json(which is usually the caret (~) range, which excludes major breaking versions). - "Latest" is the last version available (includes major breaking versions when available!).
- "Wanted" describes the maximum version of a package that is allowed by the range in your
- For each dependency, check the changelog or version notes of all the versions ("releases") of the dependency between your target version, and the one you have currently installed. This step is crucial when you're doing a major-version update. It could be useful when doing a minor-version update; and can often be skipped when it is a patch-level version (see the section about semver.
If you use exact version definitions and are on an old major version, the Wanted column will only show the version you have installed (for example, In this case, also run 👆 Caveat: watch out when you use exact packages and you see a major upgrade
vue-router@3.5.3) When Latest shows a major version upgrade that you don't want (vue-router@4.0.14), this can obscure minor/patch updates that you do want.npm view vue-router@3 version to get the list of available v3.x.x package, and manually check if you can bump the minor/patch version.
To find the GitHub Releases of a package: Always read the release notes/changelog of breaking changes. Breaking changes might require you to change your code in some places when doing the update. Watch out especially for notes that relate to other dependencies: maybe a breaking change in one package requires another dependency to be updated or installed alongside it? Maybe it requires a new Node.js version?
👆 How to find the changelog of a dependency?
There are various approaches that open-source package maintainers use to keep a change log. But mostly, they use GitHub's versions and releases pages to write notes on what changed (and updating instructions).
npm repo <package-name> to immediately open the GitHub repo of a pack
👆 What if there is a major version (breaking) change?
3. If updating seems safe, perform the updates. Updating one dependency per commit, or at least, updating outdated packages in batches, may be useful when doing many updates after not updating for a long time. I usually do it like this: Why? This way you can more easily find back the dependency update that broke something in your project. If you do Ideally you have a fully-automated & reliable test suite that can test whether things broke or not on every update. This is often not the case however, so you most likely should: If you do 1 update with 1 commit at a time, and an issue surfaces at the end, you can efficiently detect which dependency update introduced it with the git tool Do you have functioning project installed locally without lockfile that has (very) old dependencies, e.g. it was running on node 4? Don't start updating immediately! You risk losing control over the project beyond repair. Make a duplicate of the entire project folder to start the update, there's a high chance that an update will break something. Without lockfile, you can't travel back to the original functioning state. By duplicating you can preserve your original
👆 Choose the applicable update method depending on the version you want to update to
Choose the applicable update method depending on the version you want to update to
- npm install <package_name>@<version_number>, always installs the given version, ignoring whatever version ranges are specified in your package.json. In fact, it will by default modify your package.json to hold the version you're installing. You might need to use this method to bump the major version, or when you don't use caret version ranges but exact versions in your package.json.
- npm update <package_name> installs the latest version permitted by your package.json, for example: new minor and patch versions when you were using a caret version range.
👆 Update strategy: include limited update in a single commit
npm update with a list of 30 dependencies to be updated, there is a high likelihood that something will go wrong. And it might be hard to pinpoint which (combination) of those 30 dependencies was the culprit.
Test after each update
Detect a problematic commit (= dep update) with
git bisect
git bisect (docs, included in git). Instead of rolling back the commits one-by-one to see where the issue was introduced, this tool guides you through a "binary search" through a commit range, asking you to mark a commit as "good" or "bad" depending on their behavior, quickly narrowing down the range of potential commits that might be the culprit, ending at a single one! 👆 ⚠️ Don't have a lockfile? Duplicate the project before updating!
node_modules which enables you inspect a functioning version of the project if needed, or to roll back if you're really stuck.
4. After updating, run npm audit to figure out which dependencies have a security vulnerability. Is a fix announced? If not, maybe you can switch to a different dependency.
👆 What if a sub-dependency is causing a security issue?
If a sub-dependency (dependency of a dependency, or even further) is causing the issue, but your direct dependency does not offer a new version that updates the sub-dependency, you can try to "override" the sub-dependency in the direct dependency using in Yarn using selective version resolutions (since yarn v1, released on Sep 16, 2020, we're at 1.22.5 at the moment of writing).
4. Take on less dependencies
It's obvious but worth mentioning: the easiest way to avoid dependency issues is to avoid having dependencies. Taking on a dependency is always a trade-off between the value or time-savings it offers, and the maintenance costs of managing code that you didn't build yourself.
Recap: action points
Here are action points to follow-up on these best practices.
- Learn about semver
- Make sure your lockfile is committed to git
- Set up a scheduled recurring task in your calendar/task manager to update your dependencies
I hope this helped! In a possible next post on this topic, I may look at some more advanced ways of future-proofing & debugging, such as:
- Preventing issues with a
ciinstall - Using exact semver versions for all your dependencies
- Best practices specifically for Node.js projects
Credits
Apart from my own experiences, these notes were partially based on the talk "Future-proof dependency management in TS projects talk" by Olavi Haapala
Tech Weeklies - Future-proof dependency management in TS projects - 05/20 - Olavi Haapala
Posted on June 7, 2021
Join Our Newsletter. No Spam, Only the good stuff.
Sign up to receive the latest update from our blog.