Depp - How it was built
Rahul Tarak
Posted on November 3, 2021
Just briefly wanted to talk about how this built and how it works, of course the entire codebase is open-source so feel free to poke around yourself
So this package works by wrapping around the esbuild go api to add this extra functionality, so yes technically we built your whole code base to parse what dependencies you are using. This might sound incredibly stupid, but considering esbuild
speed and flexibility, it is still significantly quicker than most solutions written in javascript
I realize this is not a great benchmark but considering most other tools don't have good monorepo support it was the best I can do. I compared depp
(this tool) vs depcheck
(a popular javascript tool with almost 3k stars) on a previous project https://github.com/wei/socialify
depp 0.42s user 0.36s system 135% cpu 0.576 total
depcheck 1.69s user 0.13s system 116% cpu 1.567 total
This tool was about 2.7x
faster than depcheck
while practically building the entire codebase
Using esbuild
like this also allows me to build this tool, without writing an entire javascript/typescript
parser
Setup
First we need to get the paths of all the package.json
files and then all the source files,
I did this using something I built for an older project which is a modification of go standard library's [Glob](https://github.com/CryogenicPlanet/depp/blob/master/utils.go#L79-L87)
This modification allow me to easily find all nested package.json
and source files
Now we can read the package.json
files and get all the packages in the repository, we can store these as a hash map or dictionary to compare against in the future.
Finally, we can pass the source files to esbuild
as entry points and have it do its magic
Callbacks
First we use Resolve
and Load
callbacks from the esbuild
api to check for packages as they are being imported.
There are some caveats here tho, we don't want to go down a rabbit hole of every import. Any import from node_module
we don't care about, these are generally nested imports of modules importing other modules. So we can mark these files as external
Another optimization we can add is, any non .ts, .tsx, .js,.jsx
file can be completely ignored. This is a bit painful in go because it does not support regex lookaheads (?
)
Now with any actual import, we can check the kind of resolve it is, whether it is a import-statement
, require-call
or something else we don't care about. Finally we can check if the imported path contains any of our packages
Metafile
Additionally we use the esbuild
meta file to do a second pass on any import we might have missed or that slip through or resolve callbacks. The esbuild
meta file contains a bunch of information about inputs
and their imports
which we can use to see if a given package has been used or not
Duplicates
While the process of finding unused packages is quite involved, finding duplicates is quite easy on the other hand. It is just about correctly inputting the parsed package.json
s and checking for duplicates and storing their versions. There is a bit more work for check unused @types
packages but it mostly similar to duplicates
Not going to go into it much here, that said it probably constitutes its own post of how to bundle go
binaries with an npm package
and how to release them
All that said this was a fairly fun project to make in the 8 or so hours it took, and it has already been fairly useful internally. Hope you find some use in the tool itself or maybe how I built it
If you want to try out the tool checkout
CryogenicPlanet / depp
A fast unused and duplicate dependency checker
Posted on November 3, 2021
Join Our Newsletter. No Spam, Only the good stuff.
Sign up to receive the latest update from our blog.