Here at Stout we screen a lot of technical candidates. Our first step is to filter the resumes. (We have little need for glass window installers despite the fact that they match the keywords "windows" and "installation!") Then we conduct a technical interview to assess a candidate's skill level. For candidates who will be contracting with us directly we like the extra step of doing a coding exercise.

Why not just request a code sample?

We do like to get code samples from candidates, but we rarely get useful ones—and frequently we get nothing at all. Candidates are understandably leery about sharing code that is owned by the companies they've worked for, and sometimes they aren't able or willing to give an anonymized sample.

Sometimes we get code from hobby projects, but that usually isn't deep or up to production quality. Other samples are just too small to glean anything useful from. Very occasionally we get code stolen from the Internet, which is helpful in identifying candidates who are stupid!

Thus, the coding exercise.

A good coding exercise is about more than just coding. It should also be about process, communication and personality / working styles. Our coding exercise follows a process that is common to many of our contracting projects:

• It's geared towards remote work, so it's conducted via email.
• The programmer is given a generic task with some requirements.
• The programmer needs to do some analysis and come up with an implementation plan, along with time estimates.
• After we've reviewed the plan, the programmer is directed to implement some or all of it—or is given slimmed down requirements as needed.
• The programmer delivers the finished product and reports how long it took.
• It's important to note that the tasks we give them are generic. They are something any programmer can understand and don't require any specific domain knowledge. While we are asking them to spend time on this exercise (typically less than an hour for the analysis and somewhere around an hour for the implementation), we aren't asking them to do free work. The task they are implementing is for the exercise only. We don't use the results.

The exercise must be small enough that it can be comprehended easily and implemented quickly, while still being complex enough to produce meaningful results. Coming up with a good exercise is quite a challenge!

Having programmers implement the popular "FizzBuzz" test would tell us whether they could code, but it would tell us neither how well they deal with real problems, nor whether they can analyze or estimate.

I've administered our coding exercise to a number of programmers over the past few years, and I've seen some interesting results (focusing on the failures here because they are more interesting!):

• Some developers aren't a good personality fit; they can be difficult or unpleasant to deal with even for the length of the exercise, or even fail to respond to the exercise all.
• Some fail to comprehend requirements or follow directions. Some people take a lot of back and forth iteration to understand what others grok in one go. Even when told specifically to "do X, don't do Y" some developers will ignore you and do Y anyway.
• The quality of the implementation plans vary wildly. Some come back with excellent detail and a complete plan, while others come back with plans that are sketchy or actually infeasible.
• Estimates for implementation have varied from 0.5 hours to over 8 hours, though some of that is due to differences in plans or requirements.
• Strangely some programmers will deliver code that doesn't actually work or run. I've had implementations with a single click crash bug or implementations with a lot of code that just doesn't work. Makes you wonder if the programmer actually ran it!
• Our exercise requests "production quality" code, but apparently that's a very broad definition.

No coding exercise is perfect and I'm sure we've passed up great candidates who just misunderstood something or were having a bad day. But our exercise gives us invaluable insight into how a programmer might operate on real world projects within our specific process and requirements.

Tips for Companies

• Recognizing that candidates are in short supply, avoid some of the common coding exercise turn-offs. Don’t expect a candidate to enthusiastically accept a coding exercise before they’ve had an opportunity to meet with the hiring manager. Most candidates won’t undertake a coding exercise unless they are jazzed up about the opportunity. Integrate the coding exercise into, say, a half day interview that includes short sessions with the hiring team. Candidates are willing to make a finite investment with a company and its interview process. They balk at interview after interview, exercise after exercise, spread out over several weeks.
• If you want to use a coding exercise for your company, first figure out what process points you want to cover. For example, if you do a lot of requirements gathering then perhaps you want to give them an underspecified task and have a dialog where you can see how they elicit requirements.
• Focus on finding a task that is simple enough that it doesn't require discovery and isn't too specific to your problem domain (unless you require programmers to have pre-existing domain knowledge). Any coding should be specific to your language / platform. If your company uses a practice like Test Driven Design or pair programming, then your coding exercise should too. It may be helpful to provide a starting project or skeleton code that has TODOs to be finished by the programmer.

Tips for Candidates

• Read the instructions. Then read them again. If there is something that isn’t clear, ask for more information. Many coding exercises deliberately leave out vital information to see how candidates address that.
• Focus on completing features or sub features instead of bouncing around and completing nothing. I'd rather see a working backend with no UI, or conversely, a working UI with stubbed or mocked backend then have everything be a work in progress. Maybe you display data, but can't yet sort it or edit it.
• As you go, leave TODOs about things you still need to do. That will help you not to forget things, but more importantly if you run out of time it shows that you are aware of something that needs to be addressed, not just that you forgot about it entirely.
• Aim for production quality, but a little under polished is okay. You should absolutely do things like closing database connections, preventing SQL Injection attacks, or releasing/disposing of resources (as any production quality code should), but not everything needs to be as clean and refactored as full production code. This only applies to a time-limited coding exercise though! If you are turning in a pre-written code sample, I assume you think it's polished and production quality!
• Please test your code, or at least run it! If I can't run your sample or it breaks with a single click you didn't anticipate, it doesn't reflect well! Expect your code to be reviewed by developers who know how to break things (like putting text in number fields).
• Comments and error handling matter, but too much of either can be as bad as too little. Again, aim for what you consider to be production quality, with TODOs if you are constrained by time.

Summary

Whether you give the programmer time by him- or herself to implement the code or you pair program, you will come away with a far better understanding of how the programmer approaches tasks. If, as a candidate, you are lucky enough to be given a paired programming coding exercise, realize that this is a great way to get to know how your team interacts and approaches problems. For both, coding exercises give you better insight into whether or not you've got a good fit!

This is a technical/business article catered to developers, hiring/project managers, and other technical staff looking to improve their skills. Sign up to receive The Informatizer in your email inbox.

If you're looking for a job in the tech industry, visit our job board to see if you qualify for some of our positions. If you're looking to hire technical talent for your company, please contact us.

Stout Systems is the software consulting and staffing company Fueled by the Most Powerful Technology Available: Human Intelligence®. We were founded in 1993 and are based in Ann Arbor, Michigan. We have clients across the U.S. in domains including engineering, scientific, manufacturing, education, marketing, entertainment, small business and robotics. We provide expert level software, Web and embedded systems development consulting and staffing services along with direct-hire technical recruiting and placements.