« halifax: the future of transit | Main | book is ready! events being planned! »

TrackBack

TrackBack URL for this entry:
http://www.typepad.com/services/trackback/6a00d83454714d69e2015437e06792970c

Listed below are links to weblogs that reference basic questions about "cost/benefit analysis":

Comments

Feed You can follow this conversation by subscribing to the comment feed for this post.

EngineerScotty

The fly in the ointment, of course, is that there is widespread disagreement on the weights of these things among the communities that a (potential) project may serve.

If you are someone who will be paying taxes to support a transit line, but who won't be using it (such as someone living in a place not directly served by the line), your view of the weighting will likely differ from someone who will be directly impacted by the project.

Attempts to force a particular costing model as technically correct are more often than not attempts to place the thumb on the scale to achieve a predetermined outcome.

Joshua

I think this post raises a number of excellent points. There are fundamental flaws with the cost-benefit analysis process, because it has to turn everything into a dollar value for it to count. This is easy to do with some things (a few minutes faster for a lot of people adds up quick) but really difficult for other things (quality of urban environment, severance, pollution etc). What generally happens is the "too hard" stuff gets ignored.

A classic example of this is the post you did a few months back on the Parisian tramway that apparently had a negative BCR, even though clearly it had vastly improved the area it was located in. The cost-benefit analysis valued a few seconds of delay for motorists on a motorway above a massive improvement in urban quality.

I tend to think that transport decisions will always be political decisions - and that's fine, as long as we're transparent about it and as long as there's some accountability. We have a huge problem in New Zealand that central government holds most of the transport purse-strings, yet it's in local government elections that people think about transport when voting. A cost-benefit analysis is a useful part of assessing whether to proceed with something, but absolutely shouldn't be the be-all and end-all.

Generally, when forming an opinion on a transport project I ask some qualitative questions:

1) Could the benefits (or some of them) of this project be achieved through something else that would cost less?
2) What will the land-use outcome of this project be and is that consistent with our land-use plans?
3) What are the environmental impacts of the project and are they acceptable?
4) Does the project promote or reduce (and to what extent) car dependency?
5) Who benefits from the project, who doesn't benefit, how significant are the different benefits/disbenefits and how do they align with our 'big picture' goals for the area?

The answers to these questions typically tell us more than a single cost-benefit ratio number.

Morten Skou Nicolaisen

Excellent post. In addition to the points raised by Jarred, I would like to add three crucial arguments against heavy reliance on CBA:

1 - Distributional effects are ignored. Very mobile groups are typical recipients of benefits, while social and environmental costs are usually carried by local residents that are not necessarily very mobile.

2 - Discounting makes important environmental impacts disappear, as the most dramatic effects are often far in the future. Some of these are potentially irreversible or carry an enormous path dependency, but CBA methodology naively assumes that we can simply buy things like extinct species or depleted forests back. On top of this comes the pricing issues Jarret raises, where an example could be CO2 quotas hitting a ludicrous price of zero during 2007.

3 - In the transport sector CBA results are practically impossible to verify even after project completion, due to the intangible indicators that one must assign a monetary value. Travel time savings are usually the biggest benefit factor, but if we consider the Marchetti constant there seem to be little empirical evidence of any travel time savings as a result of new infrastructure. People simply travel longer as a result of reduced journey times. This means that on top of dubious unit pricing we are also faced with dubious unit measures. The 'objectivity' of CBA results are rarely objective at all, and is just one of many socially constructed yardsticks by which to measure performance.

Morten Skou Nicolaisen

... and I hope Jarrett can forgive me for misspelling his name twice in my previous post :)

Alan Howes

In the UK CBA (or COBA) has been dead for quite a while, its successors being NATA (New Approach to Appraisal) in England and STAG (Scottish Transport Appraisal Guidelines) in Scotland. No doubt something similar in Wales and NI too. All follow the same principles as MAE, and the abve discussion is history for us. I foolishly assumed that most of the civilised world was on the same track - is the US at the back of the pack?

I'd be interested to hear of the situation in other countries.

(Actually, England is now onto son of NATA, and STAG has been revised too - mainly in the interests of simplicity. And Stakeholder (including community) involvement is integral to both the systems.)

Jason Kramer

I would dispute your characterization that CBA "hides" value judgements in the factors to convert costs and benefits into dollars.

In fact, CBA explicitly uses dollars to place value judgments on various factors. Of course, those amounts are uncertain and subject to debate, but that doesn't mean they're hidden.

BruceMcF

One of the judgements intrinsic to CBA is the judgement that different types of costs and different types of benefits all have the same priority.

And that judgement is indeed hidden behind the CBA question of "what cost shall we assign to this minute and what benefit do we assign to that new opportunity available to property developers?" ... unless, of course, it is explicitly argued through and decided before performing the CBA what specific set of costs and benefits ARE of equuivalent priority, in which case the CBA is only one of several accounts in a multiple-account analysis.

Jarrett at HumanTransit.org

Jason.  I mean "hidden" in the sense that their nature as subjective value judgments is hidden by the ostensibly quantified, single-score conclusion. 

Murray

I'd be keen to read more about the MAE sometime. I am always reminded of something I read in a debate about Route 20 in Brisbane in the 1980s:

You can compensate an old woman for the loss of part of her front yard, but how do you compensate for the fact that she no longer feels safe or confident crossing the road to visit her friends?

Steve Munro

One central flaw in the MAEs used within Metrolinx studies here in Toronto is the "benefit" assigned to the cost of building a line. The more expensive the option, the more spin-off effects you create in secondary and tertiary spending on labour and materials. Building a $1-billion line generates more "benefit" than a $500-million one, even though that money could have been used for a completely unrelated project. The benefit of building two or more smaller projects rather than one big one is never examined because each proposal is evaluated in isolation. The final decision comes down to the politics of which project has the most supporters, and the expensive ones tend to win.

Zoltán

There are three particular problems with the workings of CBA:

- It assumes a property right to the status quo, and thus tends to reinforce it. For example, if you're someone that benefits from using a highway that's subsidised by the government at an average cost of $5 per user, and a $5 toll is proposed, a CBA would assume a disbenefit to society of $5 per user, rather than questioning whether you really ought to have been paying $5 all along.

- It usually establishes its values in an abstract way, most notably values of time. These are normally based upon stated preference surveys establishing how much people would be willing to pay for certain scenarios that cost or save them time. Having derived a value of time, it can multiply savings of say, 30 seconds over thousands of people, and come up with huge benefits. But no one has asked "How much would you be willing to pay to get to work 30 seconds quicker?". If one did ask, the answer would probably be nothing.

- Those abstract values are established by people like me, when I worked one summer some years ago doing surveys for an EU-funded study to establish willingness to pay for improved quality of water in rivers and lakes. People often didn't understand the questions, and I couldn't be bothered to explain, given the soul-crushing work and hours. Sometimes I filled things in for them so I could finish and go home. And often they double-checked that the cost scenarios I'd just put to them were hypothetical, and none of this would actually take place - because they wouldn't want to pay more for water in real life. ...defeating the object entirely.

Rob

As usual I'm late to this discussion - so I may get the last word but nobody else will see it!

This is a good post, thanks. I'd propose a more expansive principle still: value decisions should be made by people empowered to make them, not by technicians creating models. If you believe that, then the purpose of technical measures and evaluations is to inform the decision maker - and the black box that is benefit cost analysis fails to do that most of the time. Technical measures should be thought of as disclosure (like NEPA), not as a decision tool in my opinion.

There are some deep problems with translating benefits and impacts into dollar-denominated values. High among them is the heavier weight given to benefits and impacts on the wealthy because their "value of time" (or value of whatever is being measured) is based on their income. Is it really defensible to systematically devalue impacts to the poor? But that's built into most benefit-cost analyses.

One more concern: many transportation agencies apply a benefit-cost assessment to specific projects, but sometimes those projects need to be in place for the benefits of a larger program to be realized. In that case, the program benefits and costs need to be evaluated, but the projects essential for program success can't be treated the same way. There needs to be a way to account for interdependencies among projects, and to apply the analysis to the interrelated actions that need to occur to achieve the desired benefits. You wouldn't leave out a mile of track in the middle of a transit line for example - but sometimes project-level assessments can similarly miss the point of what a larger program is trying to accomplish.

Personally I prefer sometime more direct as a technical measure, like cost-effectiveness, to get a sense of how well money is being used to achieve some common measure of utility, like a reduction in congestion or increase in ridership. The more complicated the measure, the less likely the decision-maker will understand or use it. I agree with you that other important effects ought to be evaluated separately. The point is to figure out what's useful to the conscientious decision-maker, not what makes the technician happy.

Eric Doherty

Jason wrote: "I would dispute your characterization that CBA "hides" value judgements . . . those amounts are uncertain and subject to debate, but that doesn't mean they're hidden."

I have only ever come across one CBA tool where the amounts and value judgements are revealed and the spread sheets made available to the public - the VTPI's http://www.vtpi.org/tca/ . Because everything is transparent, it becomes rather like multiple accounts evaluation; you can apply your own values and change the quantitative values for each category. (Disclosure - I worked on the second edition).

In contrast, most CBA reports I have come across are basically black boxes. Sometimes there is not even a list of the factors included and excluded, never mind sources and rationales for the values.

Is it only in Canada that CBA is usually a black box procedure? Do other jurisdictions make the software and data available to the public?

Morten Skou Nicolaisen

@ Eric Doherty

Her in Denmark we now have a standard set of transport economic unit prices, which helps but doesn't eliminate the problem.

First, it is a daunting task for experts, let alone laymen (i.e. most decision makers/policy makers) to go through the analyses and data that went into constructing these numbers. At best you can look at them at evaluate whether they seem reasonable. This is a unit pricing black-box.

Second, the decision support documentation prepared by consultants often do not explicitly state the models or assumptions used to produce forecasts, making it almost impossible to evaluate their validity unless you start requesting all background reports and internal notes. Even then you might be left in the blank on some issues, such as elasticity values for induced demand (which is usually zero anyway, since it is often ignored completely). This is a unit measure black-box.

Waterloons.blogspot.com

Interestingly, today the Financial Post published a rebuttal opinion to Peter Shawn Taylor's piece here: http://opinion.financialpost.com/2011/12/08/lrt-on-track-to-benefit-waterloo/

Living in Waterloo Region and having been following our own winding road towards transit improvement, I've read a fair bit of what PST has to say about the LRT project... none of it is good, and little of it is fair. At one point he quipped that the region should use the money to buy everyone a new car. It's safe to say that PST has an axe to grind with this most recent article.

I definitely took issue with his description of MAE and his approach of CBA-uber-alles. But your description really helps separate what is analysis and what is value judgement: really frames the subject nicely. Thanks!

Nike High Heels 7

One easy-to-understand example is the ability to search, browse and quickly visualize data sets. One data scientist told us they spent significant time simply rooting around in their collection of data sets to understand what was out there, and then writing code to visualize what was in each data set. With Greenplum Chorus, that's now an extended search function with built-in visualization -- a huge time-saver.

The comments to this entry are closed.

the firm

Jarrett is now in ...

Related Posts Plugin for WordPress, Blogger...
Related Posts Plugin for WordPress, Blogger...