So over the years of writing software I’ve been subjected to Codility tests a couple times and generally they tend to leave me feeling very conflicted. I think part of it might be due to the impersonal nature of the tests. I understand that a busy organization doesn’t really have time to reach out to everyone and automated testing can be an effective part of the screening process. But with that said, if I am spending an hour or two doing a test for the organization I should get something more back than just dead air.
This is something that has really bothered me because in general the tests I’ve gotten from Codility have not been “basic coding skills” like what Jeff Atwood complains most programmers can’t do. Rather I tend to get harder problems that are not Fizz Buzz, but are more like something from their lesson on dynamic programming. Part of this is due to the nature of the organizations I tend to apply for, but the general complaint of dead air after taking one of the tests seems common on Google. Now, I’m not expecting someone to call me and explain what I did wrong, but Codility does support automatic grading and report generation and seeing that report would be useful.
One of the common themes in the developer community is that you should always be working to improve your craft. If you can’t see how you did and the results of the tests, how can you work to improve your craft? Granted part of the reason why the results might not be shared is to prevent people from posting answers; however, it is naive to think that the answers will not eventually get out there in the public. There are a number ways to catch someone cheating the test and most people aren’t creative enough to write distinctly unique code given a reference example. Thus, not providing some sort of automated feedback might be one of the things that bothers me the most.
Something else that really bothers me is the timer. While someone might argue that you are always under the clock when you are at work, realistically, if you are trying to crank out code of any complexity and there is a clock counting down in the background, something went horribly wrong or you are doing it wrong. At this point I’m fairly confident that watching the clock tick down in the corner does nothing to help my thought processes. For a basic string manipulation problem it is likely not such a big deal, but a dynamic programming challenge? When I encounter a hard problem in the work place one of the steps I might take to solve it might be to write some notes down paper and sketch out expected solutions… which leads us to the next problem I have.
Codility claims the following:
Codility gives you objective insights into your candidates programming skills, showing you their thinking process before you interview them.
however, if part of my thinking process is to whiteboard a problem then that insight is going to be lost. Also, since Codility generates a numeric score, realistically, how many hiring managers are going to look at the code for insight if the score is 26 out of 100 points? Again, my thoughts towards Codility are largely skewed because I tend to see a lot of higher level dynamic programming problems as opposed to more basic ones but the point stands.
The focus on dynamic programming problems might be the thing that bothers me the most. In general, even when working of very complex projects, the vast majority of the code is not going to be complex algorithms but your basic CRUD operations. This might be where some sour grapes come into play: I know I’m not the best coder in the world (although I like to think I’m pretty decent) but I also know that at the end of the day I’m able to solve the problem and the amount of time isn’t necessarily that dramatic.
I’d be very curious to see what would happen if the timer on Codility counted up as opposed to down. I doubt that the actual results for hiring managers would be that different given the scoring, but it would add a very interesting dimension to things. Without the pressure of the clock would more people finish the challenge in the expected time frame? Would a number of people who didn’t finish before finish within the five to ten minutes of the currently allotted time frame? For that matter, would you have some people that finished quickly and got low scores? I know that if I was given an hour to solve a problem that normally takes five I might look to get creative with my code in the remaining time to show off. Pragmatically though, having to do maintenance work on code where someone was showing off is not fun and can make it take longer to find the bug or add the feature.
At the end of the day, I’m still very conflicted about Codility. The lack of any feedback about the results is likely one of the things that bothers me the most but the clock must be a close second. While I understand why they are doing what they are doing and to a certain extent I agree it is a good idea, it strikes me as something that could be done much better than it actually is.