Bisecting vendors

greg0ire

Grégoire Paris

Posted on January 5, 2024

Bisecting vendors

Hey there! If you're landing on this page, it's probably because somebody (most likely me), asked you to "bisect this regression down to a single commit". If that's not the case but you are still interested in helpfully reporting a regression, read on.

I promise we'll see what this technical mumbo jumbo could possibly mean, but first, let us understand what the issue is.

Regression! 💥

You have an application you are working on, and you update your dependencies regularly, because you know hygiene matters. 🚿 The only way you will feel clean is when seeing composer outdated outputs nothing.

And then, one day, after an upgrade, one of your automated tests breaks. That's a regression, and since the only thing you did was upgrading your packages, it must be caused by a vendor. Since you're upgrading regularly, the list of packages you upgraded is quite small, and after reverting the upgrade, and upgrading the packages one by one, you narrow it down to a single package, that we will call doctrine/orm to protect the innocent and the guilty.

You know the right thing to do is to report the issue, and you proceed to do so, making sure to let people know that the issue didn't occur in "version 2.16.x" and then everything broke after upgrading to "version 2.17.x".

And then… crickets… 🦗 Days elapse, and nobody answers. Why is that?

Open source development

If you compare 2.17.x with 2.16.x, you can see quite a few contributions, over a long period of time, and by many different people. Some of them are part of the Doctrine core team, some others are returning contributors, and a few made their very first contribution to that repository. They all do that on their spare time, sometimes after a long day of paid work, take vacations, have a family to take care of. The core team is subscribed to issues, but it's unlikely that contributors are.

At least you known what to do now: ping them all in a new message, of course! Hold on, hold on, I was just kidding, step away from that keyboard please. 🙏

There is no need to ping anyone… it's possible that a maintainer already took a glance at the releases and tried to find a commit title that looks like it could have to do with your issue. And if they did not respond, then it's probably not that obvious.

What would really help would be finding the right person to ping. This means you have to find the pull request, or even better, the commit that introduced the issue. The first thing to do is to update the issue and mention the exact version before and after the upgrade. It does count as narrowing things down.

After checking, you find that you were on 2.16.3, and then upgraded to 2.17.0.

Git bisect to the rescue

If you don't know what git bisect is, please hit pause on this blog post, and take a moment to go watch a part of this talk by Pauline Vos at Forum PHP 2021, and to practice git bisect a bit yourself. This blog post can wait.

You're back? Great! Now… how can you apply that to this problem? Does Composer maybe have a magical subcommand to do this? Not quite it seems. There's a plan for something, but it won't help finding a single commit:

Composer bisect command #11119

Discussed in https://github.com/composer/composer/discussions/11072

Originally posted by mad-briller September 21, 2022 This is just a random idea i've had while upgrading projects and having a hard time narrowing down issues, not sure if issues or discussions is the right place to put this.

Even with all the will in the world to respect semver, packages inevitability break backwards compatibility in ways they did not forsee, everyone is human afterall.

Chasing the version that introduced the issue can be quite hard currently, as you have to load up packagist and look at released versions and install them one by one to see if they are the issue. This reduces the likelihood that a developer will report the issue to the package maintainers, and also makes the developer less likely to actually update the package, as its safer to stay on the "last working version" and put an explicit version number in composer.json.

When a incorrect change is introduced in git, git bisect makes it super easy to chase the commit that introduced the issue. It would be great if composer had a composer bisect <package> that worked similarly. This would make narrowing down which version introduced a bc break much easier.

Thanks for your time.

Let's do it manually then. The first issue you are facing is that doctrine/orm is probably not installed as a git repository, but was probably unpacked from an archived instead, so… there is no git history, this is just a snapshot. This means you cannot use git bisect yet, you first have to reinstall doctrine/orm as a git repository first.

To be fair, it's not going to be completely manual, Composer can still help us a lot here. There are ways you can make it prefer an install from source rather than from a tarball, and the simplest way I know to do this is to run composer reinstall doctrine/orm --prefer-source. You should see some cloning happening:

$ composer reinstall doctrine/orm --prefer-source
  - Removing doctrine/orm (2.17.2)
  - Syncing doctrine/orm (2.17.2) into cache
  - Installing doctrine/orm (2.17.2): Cloning 393679a479 from cache
Enter fullscreen mode Exit fullscreen mode

Now the fun begins. Here is how to proceed:

$ cd vendor/doctrine/orm
$ git switch --detach 2.16.3
Enter fullscreen mode Exit fullscreen mode

At this point you should maybe clear some caches, and check that things work. If they do work, you can start the bisection.

$ git bisect start
status: waiting for both good and bad commits
$ git bisect good # because it works 👍
status: waiting for bad commit, 1 good commit known
$ git switch --detach 2.17.0
warning: you are switching branch while bisecting
Enter fullscreen mode Exit fullscreen mode

At this point, you should clear the same caches, and check that the regression still happens.

Does it still happen? Great. Mark the current commit as bad.

$ git bisect bad # because it broke 👎
Bisecting: 33 revisions left to test after this (roughly 5 steps)
[some-hash] Merge pull request #42 from well-known-maintainer/tedious-maintenance-work
Enter fullscreen mode Exit fullscreen mode

Boom! Bisection started. Clear the caches… does it work? If yes, run git bisect good, if not, run git bisect bad, and repeat until you see something like the following:

somehashffe340934df9 is the first bad commit
commit somehashffe340934df9
Author: Well Meaning Contributor <well-meaning@contribut.or>
Date:   Tue Feb 31 25:34:19 2026 +0200

    Cool change that hopefully breaks nothing (#12345)

 path/to/mission/critical/file.php | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)
Enter fullscreen mode Exit fullscreen mode

Congrats, now you have a name, a commit hash, and a pull request number (if you are lucky). Mention all three in your issue and you will get far better help. It should go without saying, and sadly often does not, but be nice when doing so.

After that:

  • The author is more likely to be aware of the issue and fix it.
  • You and maintainers can examine the commit to see if they are able to spot the issue, which just became way more likely to get fixed.

If you do not see a pull request number, you should try finding the commit on Github first (at https://github.com/doctrine/orm/commit/somehashffe340934df9), and on that page, the pull request should appear right above the name of the author.

A screenshot of Github's UI. The pull request number is visible

Bisecting a monorepository split

Ok, but what if the issue is not with doctrine/orm, but with symfony/dependency-injection? The hash will be for symfony/dependency-injection, and the corresponding hash for symfony/symfony will be different. I often forget to mention this when asked to explain why I am not a huge fan of monorepositories, but this is definitely one of the reasons. You are not stuck though, it's just a bit more involving. What you can do in this kind of case is bypass Composer completely and symlink to a local clone:

  1. Clones the monorepository somewhere on your disk (for instance in /tmp/symfony)
  2. Run rm -fr vendor/symfony/dependency-injection.
  3. Create a symlink from vendor/symfony/dependency-injection to /tmp/symfony/src/Symfony/Component/DependencyInjection.
  4. Finally proceed with the bisection.

If you know of more elegant ways to do this, please let us know in the comments, and I will edit this post.

Happy troubleshooting!

💖 💪 🙅 🚩
greg0ire
Grégoire Paris

Posted on January 5, 2024

Join Our Newsletter. No Spam, Only the good stuff.

Sign up to receive the latest update from our blog.

Related