At some point or the other, we all read about importance of decoupling components while writing code. The coupling doesn't just exists in low level code, but one can also observer it at architecture levels. With the exploding popularity of micro services, and the lack of knowledge around building them (or rather the lack of curiosity of actually reading about them), has lead to creeping the same coupling problems between improperly designed micro services.

But let's not go into complex issues, and look at something which is much more simple and obvious ones you notice it. Its about coupling that is created from using a certain version of library dependency across services, and poor API design that doesn't abstract its effect from leaking out.

We have around 30-40 different micro services that just our team owns. Recently we started doing some dependency upgrades for our serde libraries that is used to pass object states across different services, and realised that a supposedly simple upgrade is proving to be non trivial.

The issue arose from the fact that we have plethora of micro services which all share states between each other, and used a serde to achieve the same. The serde however releases changes which are backward incompatible even across minor version upgrades (doesn't follows semver) and so there exists the possibility of breaking api calls between services since the API leaks the serialised objects as API's request / response arguments.

The way the services communicated also makes upgrading dependency iteratively difficult as they follow somewhat mesh structure, and thus we are stuck in a situation where:

1. one has to upgrade it for all required services in one go. 2. pause the traffic.
2. verify that we have covered all services by testing over a traffic slice.
3. Activate the traffic flow. In case something goes wrong, find the missing upgrade, and start with step 1 in loop.

I hope that I could say these are legacy services, but sadly they are services that were recently developed.

The important thing here is the learning that we get from the incident, majorly making micro services truly decoupled, and to achieve that, I guess at least one important thing is designing APIs in a way that doesn't leak effects from their dependencies.

Oh and also choosing 3p libraries that follow semver in case they are available.