We provided 5 of our developers with access to GitHub Copilot for 2 months to find out if we need to buy it for everyone. Here are the unexpected results of the experiment

Introduction

As someone who constantly searches for ways to improve personal coding performance and performance of my development teams, I have been experimenting a lot with AI driven utilities that might assist developers in writing code. TabNine, ChatGPT 3 & 4 and, finally, GitHub Copilot that wraps a special version of OpenAI's CodeX in a tool that is easy-to-integrate with modern IDEs.

All these tools have proven useful in many situations and allowed me to create utilities for automating daily routines with ease I had never experienced before. My initial intention was to provide all the developers in my teams with access to this tool. But to better justify this regular spending to my company, I decided to conduct a little experiment: a pilot group had been given paid access to GitHub Copilot with a duty to share feedback regularly.

I have to say that it was a very right decision because the results weren't what I expected to see. And, sorry for the spoiler, we changed our plans - we're not giving it to everyone.

Before we started: Security

Like any company that considers information security a paramount subject, we reviewed the terms of usage of GitHub Copilot with our legal team. We ensured that running GitHub Copilot in a client machine doesn't involve long-term data collection by Microsoft. The prompts are discarded immediately after providing (see details).

We didn't include Copilot Chat in the scope of the pilot project (that is in beta at the moment of writing). Using the Chat might require accepting more conditions for data collection and processing.

Setup

For the pilot group, we have selected 5 software developers of different experience (from 3 years in IT to 10+ years) and different skillset (React.js in the frontend, Ruby on Rails and Node.js in the backend, GitLab CI for CI/CD pipelines). These 5 people were given GitHub Copilot licenses for 10 weeks and were asked to provide feedback every 2 weeks.

The common themes in the feedback have been collected by me (I also was a part of this program. I wanted to stop paying for it from my own wallet) and are presented in this article.

Observations

It's easy to start using it

What everyone has noticed and mentioned is how easy it is to install in VS Code. Basically, after 2 minutes of installing and configuring GitHub Copilot, the tool is ready for use and provides suggestions in all the files you work in. Convenience of installing is extreme.

But I have to mention that some people found it easier to migrate from Sublime to VS Code to start using it. Technically, it should be possible with Sublime though (this one looks interesting but we didn't spend time on it).

When it's wrong, you are (rightfully) not happy

One of the things noticed by more than one person was the effect of a wrong suggestion. When the suggestion is incorrect, you lose time. At best, you gave a second to Copilot to provide you with something you don't want to see. At worst, you have your code broken and your page turns red. And absolutely always, you have a half-a-second pause to see what it has got for you. You wait. For a fraction of a second but you wait. These micro-pauses are not what we liked.

It might be helpful (and dangerous) for beginners

A couple of people have mentioned that they think GitHub Copilot might be more useful to beginners in technologies. While I partially agree with that, I also have concerns. Any complicated tool requires a skilful operator. The suggestions from AI-powered tools must be validated by a person who knows what the code does and what it should do. Without this validation, it might be like an untamed id without ego. Who might want more chaos in their project?

Tricky things (sometimes) don't get less tricky with Copilot

One of more experienced developers tried to use Copilot for tricky things he is not extremely familiar with. The example is writing PostgreSQL query that filters records based on nested parameters stored in JSONB value. The suggestions given by Copilot were always somewhat imperfect. A solution was found with help of good old ChatGPT after specifying context explicitly (schema, version of PostgreSQL, version of ORM and so on).

It might document your function better than you

We spotted that at times GitHub Copilot helps a lot with documenting a method or a class. It's not lazy, it will write what's needed to full extent. It considers the context of the project, all the implications of underlying logic and it produces a good short summary. Probably, you don't want all of your methods to be documented in plain English. Or do you?

Short takes

• Sometimes it surprises you with how much you're guessable.
• If you have a typo in your code, it inherits it and can suggest code with this typo again.
• Try start typing Database tables walk into a bar for a bit of fun.

Summary & Decision

The results are quite surprising and that's why it's in this blog. Every developer in our experiment came to a simple conclusion: if the company doesn't provide the license, they wouldn't purchase it themselves. But that statement might be a bit too generalised - I personally know a few people who pay for this tool for personal use. Including myself. The drawbacks of having wrong suggestions are usually overweighed by ability to create an Apple Script or getting a suggestion on Jira API protocol without leaving IDE.

Copilot Chat seems to be a more promising feature. A conversation-style tool (like ChatGPT) allows adding context that you realise is lacking. Also, it provides a much more natural environment of chatting with an extremely experienced developer who knows programming inside out but probably unaware of what exactly you're working on right now.

We decided to communicate to the developers that they can request a license for themselves. They will need to write a form, to justify the purchase. In simpler terms, we let developers request a tool from the company, just like a specific IDE or GUI for a database. This approach will help us optimise our software expenses while ensuring that those who genuinely want and need the tool have access to it.

GitHub Copilot is not a one-size-fits-all solution, and it was never intended to be.

Searching for the next one. Stay tuned.