Improving Wetware

Because technology is never the issue

(In)Validating the 10X Productivity Difference Claim

Posted by Pete McBreen Tue, 11 Jan 2011 17:04:00 GMT

Recently Laurent Bossavit wrote about Fact And Folklore In Software Engineering - (link is to the english translation). He pointed out that there are few hard numbers available about the productivity of software developers. Yes, there are plenty of anecdotes, but what studies that were done were a long time ago and weak methodologically compared to what we expect of current studies on human performance.

Bossavit then goes on to tackle the claims of 10X productivity

We can now circle back to this widely circulated “fact” of the software profession, according to which “programmer productivity varies by a factor of 10 (or 5, or 20) between the best and worst individuals”. This is a remarkable statement, not least because of its implications: for instance, programmer compensation does not vary accordingly.

Steve McConnell then gets dragged in for his review of the 10X literature. Bossavit claims that the research that has been done is not sufficient to validate the claim for 10X differences.

But “work” is the right term here: tracking down articles and sometimes entire books, some of them out of print, just to scan them for conclusions which turn out not to appear anywhere. It is a sad fact that critical examination of ill-supported assertions takes a lot more time than making the assertions in the first place; this asymmetry accounts for a lot of what is wrong with public perceptions of science in general, and possibly for the current state of our profession.

Although Bossavit’s article can read a bit like a personal attack, the problem of unsupported claims is something I have seen a lot in the “debate” over Global Warming. Indeed there is now a term for the generation of unsupported claims the Gish Gallop that was first used in attacks on Evolution and now against Climate Change.

As Bossavit rightly points out, it is much easier to make a claim than it is to refute the claim. Especially when the claim feels right and has had a long history of being accepted as common knowledge.

Steve McConnell replied to the article by Bossavit with another blog entry - Origins of 10X — How Valid is the Underlying Research? in which he revisits the same papers and summarizes with the conclusion that there is Strong Research Support for the 10x Conclusion.

McConnell acknowledges that there could be some methodological weaknesses with the original studies, but states that the body of research that supports the 10x claim is as solid as any research that’s been done in software engineering. Personally I think that falls into the category of damning with faint praise.

But is the difference as large as we think?

Bossavit did raise one point in his article that McConnell did not address - programmer compensation does not vary accordingly.

This is a telling point - if the difference is productivity can be 10X, why is it that salaries rarely fall outside the 2X range for experienced developers. Ignoring the lower starting pay issues, once a person had 3-5 years in the industry, salaries in North America are of the order of $50,000/year, Apart form a few outlier major cities with crazy cost of living expenses, it is hard to find anyone actively involved in software development (not a manager) who is earning more than $100,000/year.

It could be that the research is old - which was a criticism made for my use of the early studies in my Software Craftsmanship book, after all it is suspect when a book written in 2000 is referring to studies done back in 1972 or even 1988.

Unfortunately there have been no real studies of programmer productivity in the recent era. Yes there have been lots of claims made for the Agile approaches, but there are no real methodologically sound studies that are easily found. True there may have been studies that I cannot find, but I would guess that if any such study had been done then the authors would be making money off it by now and it would become known.

Overall it would seem that the software engineering community does not have any solid evidence to back up the 10X claim using current development tools and techniques. The anecdotal evidence we have would suggest that maybe there is a 3X difference between currently practicing software developers, and there may be some outliers who are much better than that but that those individuals are few and far between.

But that is just another unsupported claim. Yes, there is an obvious difference is capability between experienced developers, but there is no easy way to measure it, and what studies were published on the topic were from research done a long time ago, practically in the prehistory of software development.

All of the above is why I promote the idea of Software Craftsmanship over Software Engineering.