Script up your projects

At bitcrowd, we love conventions. Being an agency, we jump projects, codebases, frameworks and languages quite a lot - to a point where it's impossible to keep all the details about every project one's head. It's not necessary to actually have this knowledge in practise though. Some people knowing the details of some projects is enough. They can onboard others who then slowly forget the details of their previous projects… A well-established, semi-stable cycle of knowledge transfer with some silos here and some information lost there - but overall working.

Still, there is this nice and fluffy feeling of coming to a new project, seamlessly setting it up and immediately finding your way around. Taking developers by their hands upon entry, these hygge codebases appear nice, friendly and motivating. And developer happiness aside, even on the dark and cold business side of things, people coming to these "welcoming" codebases are probably likely to be more productive with the head start they get there.

How can we get to this shiny place?

Conventions ⚖️

Notorious self-optimization and reaching for ultimate "productivity" are neoliberal derailments. But decluttering daily tasks from unnecessary cognitive overhead, can actually have a calming, decelerating effect on our personal work life: it gives us a cleaner, virtual "desk", less stress and easier time to set details aside and focus on the important things.

In software development, conventions can be a useful tool to reduce the cognitive effort involved when switching between projects. Backed by a known outline of what to expect where, developers can, for the moment, put aside the nitty-gritty details of which version X of framework Y a project uses and instead focus on things like domain or datamodel to easier wrap their head around the new problem space. Utilizing agreed upon patterns, we can take load off peoples' shoulders for the ultimate "hygge" developer experience.

Situation 🚣

In agency work, with its fast and frequent project switches, the "onboarding" and "getting started" phase on a codebase are especially crucial. But product teams shouldn't overwhelm new members with days of fiddling with different package managers either.

First Contact 🐣

Coming to a new project. What are the first steps you always take?

You most likely have a look at the README first (hopefully it's a good one) 🔦
You search the README for installation and setup instructions (the classic "getting started") 🔎
You spend a few ~minutes~ hours copy-pasting things from the README into your shell to get to a state where you can finally run the project 🚜
You nag your colleagues for the steps which are missing, outdated or simply not documented 🕵️

All in all you pretty much spend around half a day setting up the project and already lost the joy of contributing to it. Time, effort and motivation you could have spent on getting to know the domain better and that will be missing in later phases of the project.

A sad situation for a group of people aiming to utilize computer to "solve problems". Projects with a bad setup and onboarding experience are likely to distract and discourage people. Coming out of a rough setup safari, one may feel insecure and as if one knew "nothing" about the project.

Day-in day-out 🌚🌝

Once actually working on a project, one has to run certain development tasks now and then: database migrations, the test suite, managing translations, etc. One may want utilize all possible brain capabilities on memorizing those for each possible framework or alternatively just search up and down the shell history for the one command one at some point managed to successfully copy-and-paste. But what if you don't even know what to search for?

A classic example: You come back to the project after a few days on a different one. You fetch the latest changes on the main branch and then… Run the migrations? Update your JavaScript version? Update packages and then run migrations? Is it npm install or yarn install? A steam of questions and decisions to take before you can even start your editor or run the first test. And there is also the extra cognitive overload of context switches within the project: switching between feature branches, pairing with a colleague on their ticket, etc.

Agreeing on conventions, on a common way to do and approach things across projects may reduce this overload and give people an easier time thought the day 🐖.

Scripts to the Rescue 🚑

People at Github made an attempt to fix this situation: scripts to rule them all. The idea is to have common set of executable scripts for common developer tasks in a script/ directory in the root of every project:

A consistent bootstrapping experience across all our projects reduces friction and encourages contribution.¹

While every project may use different tools or languages, the script directory consistently following the same pattern everywhere gives developers something to hold on to. Scripts as an anchor in the sea of cognitive overload after checking out a codebase:

script/bootstrap to install/update dependencies
script/setup to set up a project for the first time
script/update to update a project to run at its current version
script/test to run tests
script/console to opens a console
…

Technical setup instructions can of course live as 100+ copy-pasteable steps in the README. Putting them into a script has roughly the same effort but a huge benefit: it can do the work for future developers 🤖. Scripts can work well as a runnable documentation for a project's tooling setup. Imagine switching between Python and JavaScript projects. How to run migrations here, what are the default CLI flags there? Instead of infinitely searching the shell history, simple scripts in the project can wrap and document common developer tasks.

The selection of files in your script directory of course depends on every team's individual situation and workflow. Just go with Marie Kondo, get rid of scripts which don't "spark joy" and add others where you feel pain.

Recommendations 🛒

We adapted this pattern for our own workflows at bitcrowd. Here are some loose recommendations on what we find useful:

script/test-e2e

On a lot of projects, high level end-to-end tests are slower. So it's nice to be able to run them with a separate command. With that, script/test can focus on running unit tests only, finish faster and provide important feedback earlier.

script/lint

We are obsessed with linters. Running them as part of the test suite does not fit our way of working. For us it makes sense to run tests and linters separately as they hint at different problems in your code. Maybe one wants to not care about linting until one has a working implementation or tests and prefers to do linting and code cosmetics later?

script/format

Some ecosystems support formatters. If machines can do the formatting for us, why not accept their help?

Your own 🎨

Add your own scripts, like script/deploy, script/psql, etc., there are no limits. Come up with whatever suits your organization and team. The only thing: stay consistent and have conventions. That's where the real benefits lie.

For instance script/manage to facilitate the manage.py utility in Django projects:

#!/usr/bin/env bash

# Run manage.py tasks
# Usage: scripts/manage [...args]

set -o errexit
set -o pipefail
set -o nounset

SCRIPTDIR=$(cd "$(dirname "$0")"; pwd)

cd "$SCRIPTDIR/.."

exec poetry run python manage.py "$@"

Or on a project which requires a specific version of a PostGIS database, using a utility script to to conveniently run the database in a Docker container. Developers may run script/db start to start the database and script/db stop to stop it:

#!/usr/bin/env bash

set -o errexit
set -o nounset
set -o pipefail

CONTAINER_NAME=project_x_db

find_container() {
  docker ps \
    --all \
    --quiet \
    --filter name="$CONTAINER_NAME"
}

start_db() {
  if [ "$(find_container)" = "" ]; then
    exec docker run \
      --interactive \
      --tty \
      --name "$CONTAINER_NAME" \
      --env LC_ALL=C.UTF-8 \
      --env POSTGRES_USER="$POSTGRES_USER" \
      --env POSTGRES_PASSWORD="$POSTGRES_PASSWORD" \
      --env POSTGRES_DB="$POSTGRES_DB" \
      --publish 5432:5432 \
      postgis/postgis:9.6-2.5-alpine \
      postgres
  else
    exec docker start \
      --interactive \
      "$CONTAINER_NAME"
  fi
}

stop_db() {
  exec docker stop "$CONTAINER_NAME"
}

print_usage() {
  echo "Usage: db [<option>]"
  echo ""
  echo "Options:"
  echo "  start   Start database."
  echo "  stop    Stop database."
  echo "  --help  Show this message and exit."
}

if [ $# -lt 1 ]; then
  print_usage
  exit 1
fi

case $1 in
  start)
    start_db
    ;;
  stop)
    stop_db
    ;;
  *)
    print_usage
    exit
    ;;
esac

How to do scripts? 🚌

Who would ever blindly run a script on their machine? Better take a look at the contents first. If written, formatted and documented with care, scripts work as a great way to consistently document development workflows. And since we never get things right the first time: continue to update your scripts (e.g. when new people are onboarded and run into issues). Integrating your scripts into your daily workflows and running them regularly also helps to iron out rough edges. We for instance usually run script/test, script/test-e2e and script/lint as part of our CI pipeline.

When it comes to writing scripts, pick whatever suits your usecase and what you feel comfortable with. For the sake of compatibility, Bash can be a good choice. But be aware, it can be tricky at times… ⚠️:

The weird thing about shell scripts is that even strong advocates of good practices gladly forget all they know when it comes to shell scripting.²

Linting your shell scripts with Shellcheck can improve the situation a lot though. It is also a great resource and opportunity to learn about best practices.

Here are some resources we found useful for leveling up our scripting:

Conclusion 🍼

Scripting is not a silver bullet, but it can make developer lives a bit easier. And it's a great rabbit hole to get lost in 🕳

Blog