Jim paints a 5 ft by 3 ft watercolor landscape in three days and sells it for $1,000.

They both get their job done in three days. One covers far more area with paint than the other. One gets paid more than the other. Both of their customers are happy with the result because they paid for it. Who is the most productive?

Is it even meaningful to ask that question?

Doesn't it depend upon WHAT it is that is produced? Doesn't the WHAT have to be the same in both cases to be able to say which is more or less?

Perhaps before we talk about being more or less productive we need to identify what it is that we are producing. We should make sure its real, meaningful, and relevant to our purpose and that we are all talking about the same thing. Then finally, we must measure something that can't be easily inflated or faked. Only then can we talk about more or less comparisons and be saying something other than words without real meaning.

As I see it we have two questions to answer:

What is it that we are actually producing when we write software?

How can we reliably measure it?

We don't have good answers for either. Actually, I don't think we even have moderately good bad answers.

Isn't it interesting that a large and increasing fraction of the world's economy is based upon something we don't really know what is and can't actually measure?

    Doesn't it depend upon WHAT it is that is produced? 

1) how much does joe's paintjob increase the value of the home for the HGTV-obsessed house-flipper?

2) how much will the value of jim's watercolor appreciate after he kills himself following a week-long binge of heroin?

put another way: how efficiently is value being created? the important answers lie more often in the decision process, and less often in the implementation schedule.

for example, a web developer working on a brochureware site: the right question may be along the lines of 'how will the work you're doing affect the conversion rate?'. in reality, those decisions aren't up to him -- the might lie in the internet strategy department or some such. they push the order, the developer implements. a real measure of results is closer to 'did the designer make the right decisions to yield results?' rather than 'did the developer get the code done more or less quickly than last time?' in my experience, when someone who is a non-developer is trying to quantify developer 'productivity', what they're really doing is dodging the question of whether or not their decision-making was good or bad, usually when they know it's bad and are looking for something to roll downhill.

more interesting to me, as a developer and sometimes manager, is whether or not a developer can deliver when they say they can? if clay says it'll be done in a week, and then it is, he's been sufficiently productive. 'sufficiently productive' is maybe the best thing we can do. the race analogy is an apt one -- if developer 1 can produce a set of functionality with the same defect level faster than developer 2, we can say that d1 is 'more productive', but it's a difficult thing to quantify absolutely. take the number of hours a developer spends working on a given project point, multiply it by their hourly rate, and you get a development cost for that project point. then consider the value that project will add to the product, over time. if you have 30,000 customers who haven't bought the product yet because it doesn't have Feature X, and adding feature X costs dollars Y, then your value-add is something line Z= customers * product price. your efficiency is something like Z/Y.

clearly you want to minimize cost, and therefore keep your efficiency as high as possible, but the ways to manipulate that, and a lot of points to consider:

a simple example -- if you can hire a developer that will take twice as long, but work at 1/3 of the rate, would the revenue lost during the extended development period negatively affect the profit efficiency?

the important questions are somewhat higher up.

I like the notion that productivity should be considered as rate of delivery of value. Unfortunately, this gets into the sticky questions of value to whom and for what purpose. An even more sticky question would be how to measure it if you can answer the first two questions.

On the "somewhat higher up" questions. Are we to hold the programmer responsible for such things as "conversion rates" which may be more impacted by advertising, quality of web site, and momentary media buzz than by anything the programmer did? If so, how is this measuring the productivity of the programmer. Its difficult to make the connection. That is unless you are the whole team from start to finish.

Perhaps the best we can do is compare present and past "productivity" to see what has improved and what has not. If there is a net improvement, then "productivity" has increased else not. Trying to put a number on it beyond plus or minus may be a hopeless dream.

Still, for something that seems to be driving the world's economy, a hopeless dream is not very satisfactory.

comparing two programmers is possible. comparing a single programmer against a theoretical ideal is improbable. finding a remotely accurate formula for measuring developer productivity and yielding data that's actionable in the business space is really, really unlikely.

One aspect of TDD that isn't often quoted is that it helps get a concrete measure of productivity (and it's one of the themes of Beck's book). If you can see that today you've knocked out 100 tests, that's some measure of progress. It's a running theme through agile-processes. Small iterations / stories / code changes with measurable outcomes.

We can't quantify it or inspect it. We can't corroborate it. We can simply ask a programmer whether they are productive or not and trust them to answer for themselves.

And if Alice meets Bob on the street and they disagree over what makes them productive, there is really no way to settle the question. If Alice says she is more productive using Ruby than Python, but Bob says the opposite, we must trust that they are both right, and furthermore we can't really draw any conclusion.

Because Charles may report that he is more productive with Ruby, or Python, or perhaps Arc, and again he is the only one who knows.

Programming isn't a natural mode of thought, and programming languages aren't a natural mode of expression. I think it's unrealistic to expect that we'll ever find a universal best practice.

Imagine that productivity was as simple as bug-free function points per iteration. In this hypothetical case it's entirely possible that Alice is more productive in Ruby and Bob is more productive in Python, and we can measure the productivity.

However, the problem as given is that we don't even know that Alice produces more in Ruby or Bob in Python. All we have is their subjective assertions. That is the question I have: What are Alice and Bob talking about, and do we even know whether there is a correlation between their self-diagnosed productivity and anything useful we can measure like project success?

I don't think so. Consider Charles the Arc programmer and Debbie the PHP programmer. They work on identical Ruby projects for the same amount of time.

Charles reports that he was horribly unproductive using Ruby's brain-damaged implementation of half of Lisp. Debbie reports she was way more productive in Ruby than she ever was in PHP.

But in reality, Charles accomplished far more using his knowledge of Lisp, while Debbie spent most of her time learning how a truly dynamic language worked. Nevertheless, she felt "flow" and "freedom" while learning Ruby.

Can we trust their self-diagnosis? According to them, Debbie was more productive than Charles.

Really, the only way to measure programmer productivity is how fast they can implement a perfectly detailed spec. If you have some unknowns in there, your measurements will tell you nothing.

Therefore, I might suggest a test whereby two programmers are told to implement a program with a spec that is completely known, in the same language, and with the same set of libraries, in their perfered (or an independently chosen ) working environment. What this comes down to is just a race, and some people have good days, and some have bad days.

If you run this test several times with different specs, you might be able to objectively tell whom is better given those environmental characteristics. Of course, these measurements will be nearly useless in a real project environment for the reasons cited in the first paragraph.

And it's pretty hard to quantify any unknowns. Somebody said that programmers do know when they're productive. I know when I am and that's basically whenever I write anything at all that I _know_ will contribute to the functional or algorithmic completion. Merely that constitutes productivity!

That means I know that "if I keep doing this I will finish the program at some point". Writing boilerplate doesn't count. If I keep slamming getters, setters, and a like into my source file twelve hours a day it won't make the actual program ever ready. All I'm typing in are effectively prerequisites.

Boilerplate doesn't count even if it's mandatory in the language I'm using: I'm not sure if I could ever feel productive with Java eventhough I know I would really, really have to write a hundred setters/getters in order to finish. The logical step that would make me productive on my own scale would be to scoop power from a better language.

You're correct, as taking that to it's end conclusion would imply it's impossible to improve (which I'm sure most would disagree with).

Perhaps the question "How do you measure improvement?" is relevant. This might be a roundabout way to yield the former.

However, when we use the word "productivity" in every other aspect of business, we are describing the production of output. We don't measure how the output is produced, or the subjective feelings of the producers.

So what output are we discussing when we talk about productivity? Or are we using the word in a special way that has nothing to do with productiviy as it is used elsewhere?

As to the rest, I dunno. Let's consider the following scenario: every time you wrote something down on paper, there was a chance the paper would spontaneously burst into flame. There are strict, reproducible rules governing when the paper would and would not catch on fire. People go to school to learn these rules, which are varied and subtle.

In this world I've described, is the best writer a person who can write the most essays that don't spontaneously combust? I feel like as programmers we think that because we can easily quantify whether something works, we can also quantify whether it's good. Or worse, that everything which doesn't burn away is essentially the same: just words on paper.

It's simply not true. The fact that writing a program feels more practical than writing a novel doesn't change that.

The first programmer has spent a lot more time in doing it (like 5x the second programmer) and carefully verified each line to make sure the code and algorithm are correct both conceptually and in practice, and tested that the code won't fail in any of the imaginable corner cases.

The second programmer wrote it over the weekend, didn't test much else but merely cross-checked the code with a few working test cases similar to what the program will be used with for starters. He maybe followed his intuition to guide how to write or let his creative flow dictate the details of the program but he really hasn't had time to verify why.

There seems to be no clear notion of which programmer was more productive. It all depends on what perspective the project is viewed. On the other hand, I believe there's always some absolute value underneath and productivity isn't entirely an subjective property.

(Mandatory car analogy: If you want a Trabant and I sell you a Mercedes-Benz for the price of a Trabant and you still treat the car like a Trabant, the Mercedes is still better built regardless of how you extract value out of it.)

When the project was deployed, both programs worked correctly as they are similar. The cost per line/token of the first programmer is five times the cost of the second programmer. On the other hand, had the second programmer undergone the same rigorous verification it would have taken 2x the time of the first programmer.

Apparently the first programmer's code has a lot more value simply in being well-thought, and the programmer knows exactly why he has arrived in the solutions he wrote in the code. If the code broke the programmers would be on very different levels with regard to how to start debugging. On the other hand, the second programmer produced the same program in 1/5th of the time which may greatly please the management. However, the second programmer might easily become a grand failure in the next project as in this project he already reached the same solution if not by accident then at least by luck. Both programmers produced the same comments as they wrote it mainly for the future maintainers, as usual, instead of as a record of the whole mental process they experienced.

I like to think that most programmers enjoy being productive, and strive for increased productivity (rather than working just for a paycheck). I therefore also believe that programmers expend great effort over the course of their careers optimizing their work habits to maximize productivity.

All programmers are different, but none of us is completely different from all others. There is much to be gained from examining what the others have learned in the process of programming. The danger in this is thinking that there is some objective Truth in programming productivity.

So, Bob disagrees with Alice over what makes a programmer productive. Assuming both Bob and Alice are effective programmers, what we should really be examining are the ideas they have in common.

For me, reading programming blogs from around the Internet gives me a giant mental Venn diagram--the ideas in the center are what I focus on.

At the end (maybe of 12 months to allow for maintenance) you'll have a quantifiable result saying that Bob is more productive than alice.. (for that task - and probably tasks like it)

* I didn't say that it was easy or that we should spend that much time on a pissing match.

Controlled experiments with human judges seem like the best answer.

But you know it when you see it. (Even if somebody else might not agree.)

What are you gonna do? All the halfway interesting concepts in life work like that.

Who's really tried?

There's been a few research papers here and there, measuring the things that are easy to measure in a corporate environment.

But who has both the incentive and the skills to measure a very difficult to quantify subjective experience?

It is absolutely a very complex subjective experience, but it ain't magic.

It IS very had to measure but it's not impossible just because no one's really tried yet.

But then again, maybe if we drill down deep enough, we'd find common themes or aspects for why Alice feels more productive with Ruby and Bob with Python.

That's not to say that programmers have no clue, just that our internal radar isn't exactly infallible. People tend to feel "productive" when they feel like they're "getting a lot done," but real productivity depends on whether you're doing the right things and whether you're doing them well.

And, let's face it, most programmers (myself included) really enjoy that feeling of just cranking on something and becoming so engrossed that they don't notice time pass. So I think that people are biased towards thinking that always corresponds with maximum productivity simply because it's enjoyable, and not because it's truly always optimal.

So true.

http://www.geekherocomic.com/2009/01/19/efficiency-and-compr...

> That's not to say that programmers have no clue, just that our internal radar isn't exactly infallible. People tend to feel "productive" when they feel like they're "getting a lot done," but real productivity depends on whether you're doing the right things and whether you're doing them well.

So, is it safe to say that if you 'feel productive', you're most likely not?

In Peopleware DeMarco and Lister explain the Gilb Measurability Principle by what they call Gilbs law: "Anything you need to quantify can be measured in some way that is superior to not measuring it at all."

That does not mean that it will be easy, cheap or perfect, only that it will be better than not measuring.

Tom Gilb: "Sometimes, it seems difficult to identify satisfactory scales of measure. Often, the only ones that can be found are indirect, imprecise, and have other problems associated with them. From my point of view, these problems can be tolerated. The specific scales of measure, and meters for measuring, can always be worked on, and improved over time. In all cases, an attempt at quantified specification is better than vague words."

So I disagree that productivity can be measured, but agree that there are things you can measure and agree that measuring those things is valuable.

In the quoted example, Joel's eigth test measures the environment for interruptions. This is a good thing to measure. However, this is not the same thing as measuring productivity, because then we jump to the conclusion that "Working alone in an office with a door is more productive than pair programming because pairs interrupt each other constantly."

So, my thesis is that you can and should measure lots of things, but if you are not directly measuring productivity it is misleading to pretend that you are.

The trouble starts once we get the idea that our personal quirks are somehow universal.

However, waking up always catches me by surprise.

The nice thing about programming (or writing, or tons of other creative endeavors) is that feeling you are more productive will likely actually make you more productive. If you're frustrated because your tools are holding you back, that's going to spill over into your work and make you unproductive. If you feel like you're getting things done, you're more likely to get things done later. So you just use the ones you like best, and assume you'll be more productive because they're the ones you like best.

The only time when you're certainly not productive is when you're arguing on the internet.

Then again, other times I think that success is all about luck, and so the most productive team is the one that manages to avoid blowing it when they do happen to get lucky. Think of Digital Research, Yahoo, or Netscape. Maybe productivity here is measured in disasters avoided.

I know for sure that productivity isn't something as mundane as effort expended or lines of code written. Because really, the vast majority of features you implement won't matter to anyone.

I'd argue that the biggest battle is preventing people from being unproductive. Measure how often people have a productive state of "true", but don't measure them against each other.

How you do it isn't the important thing. The important thing is that you can look back later and review your work.

None of these metrics are perfect, but they're a heck of a lot better than nothing, or "I don't know". You can iterate, and refine the estimates as you go along. Just make a conscious effort to measure something. Getting actual data trumps all these endless arguments.

Want a number, rather than a subjective judgement? Too bad. Want a manager to make the determination, rather than a programmer? Too bad. Want to compare two coders without showing both their code to the same judge? Too bad. Experienced programmers know what works for them, but don't expect any good studies.

Most businesses seem to use software as a set of automated business rules or as a service to clients and what the owner of such companies are looking at is profit. How much money did it cost for you to write software that made $X million? Honestly I think that's the only metric people really care about.

What about maintainability and the SOLID principles? That's a matter of by spending $N million on maintenance how many millions have we saved on future cost to implement profit driving features.

If what you are doing at your company doesn't somehow lead back to profit (even something as simple as "letting the devs do this keeps the good ones here") I guarantee no one will want you to do it. If you do it and it does drive some revenue, the question will be was the cost worth the reward.

The problem most companies have is it is very difficult to relate what each task a programmer works on to each dollar of revenue earned.

Pointy-haired bosses notwithstanding, your employer has a stake in your productivity, and in my experience, they typically have a pretty good idea of who is productive and who is not.

It's hard to get an exact measurement of productivity, unless you have multiple people do the same sample problems. There is no standard unit of programmer production, because in the real world you should be solving an unsolved problem every time (or using a library).

Granted I am an "engineer" in a small company, not just a programmer, which I guess means I do things like write requirements, evaluate new platforms etc. IE lots of things not directly tied to implementing new features or improving existing ones. Not sure how that factors in.