"Our greatest responsibility is to be good ancestors."

-Jonas Salk

Sunday, August 1, 2010

The Honestly Broken

In our last installment, I pointed out that there's a clear gap in Roger Pielke Jr.'s taxonomy in The Honest Broker: there is no role for anyone trying to remove bad science from the conversation. If one's alternatives are a) to advocate for a particular policy or b) to maximize available choices among policies there is no expertise-informed winnowing; bad policies which sound good cannot be sifted out.

Here I will look more closely at the central idea of The Honest Broker, and will argue that it, itself, constitutes invalid scholarship, and it, itself, should therefore not be given great weight in the public discourse.

Obtaining the Taxonomy:

Roger begins by proposing two models of science (linear vs stakeholder model), and two of democracy (Madisonian vs Schattsschneiderian). I won't make an effort to repeat his definitions; I find them hard to understand and so I would mess it up. I will simply note that each possible quadrant of "model of science" + "model of democracy" yields a role:

linear/Madisonian: Pure Scientist (does not care about policy)
stakeholder/Madisonian: Issue Advocate (evaluates science to support an agenda)
linear/Schattschneiderian: Science Arbiter (responds to questions from policy sector)
stakeholder/Schattschneiderian: Honest Broker (clarifies and expands choices)

Re-Obtaining the Taxonomy

Roger obtains a separate taxonomy by asking a question contingent on the answer to another question:

Q1: Is the decision context characterized by values consensus and low uncertainty?

If yes to Q1:

Q2a: Is the decision connected to policy?

If yes: Science arbiter

If no: Pure science

If no to Q1:

Q2b: Reduce scope of choice?

If yes: Issue advocate

If no: Honest Broker

Roger's Unproven Theorem

The two taxonomic systems yield the same results via very different means! The first taxonomy is based on your model of science and your model of democracy; it produces exactly one role for each possibility. The second taxonomy is based on your model of the specific question at hand, and then conditionally (in the Q2b case) on the motivation of the scientist or (in the Q2a case) on a further elaboration of the model.

To make matters even more confusing, one of the combinations seems so trivial as to be unnecessary: if a question has values consensus and low uncertainty, and only then, this taxonomy yields a "Science Arbiter" (*), which apparently is a very easy role. A Science Arbiter is apparently someone who tells people to do what they have already decided what to do. Is that a paid gig? Where do I apply?

(*) Fixed - see comments

An exasperating part of all is the "reduce scope of choice" question. This is a useless question to an informed person with an interest in policy. The informed person is interested in finding the best policy consistent with his or her information (increasing scope) and in avoiding policy that is not consistent with his or her information. The idea that one must choose one role or the other is not helpful. It is exactly this lack of a meaningful role for someone who is neither advocate nor neutral but actually engaged and informed that is a core problem I often complain about; and yet Roger eliminates such a role by construction.

But consider what has just happened: the same four-way taxonomy was achieved by pursing two very different paths. For instance, a person who works on a problem with either high uncertainty or high values contention who wants to increase the available scope of responses is someone who has a stakeholder view of science and a Schattschneider view of democracy. Your model of democracy and your choice of scientific pursuit are tightly coupled with a perfect constraint!

Now, you would think that getting to the same place by two different means would itself be material for a book. What an extraordinary coincidence! A deep and unintuitive theorem must be involved!

But no, the equivalence is merely stated without proof or even any expression of surprise or delight. This is presented in the second chapter and never defended. It was clear to me at that point that anything a scientist might take for clarity of thought was lacking in this book. After that I read more quickly but with less energy and enthusiasm. This kind of handwaving does not reward a careful reading.

House Built on a Weak Foundation

The rest of the book amounts to working out the implications of this taxonomy. But the taxonomy is confused. In particular, no realistic example of an "honest broker" is provided, and no really compelling definition is offerred either. We are only told that honest brokers are needed, that their role is to maximize the number of options available, and that in addition to advocates they are all that is needed.

The many glowing reviews Roger has received for this book, in my opinion, do not speak well for the state of discourse among the community in which he participates. Of course, it's well known that the sociology of science has a history of entertaining the most absurd relativism; compared to Bruno Latour, Roger is another Bertrand Russell. But, alas, compared to Bertrand Russell, he is another Bruno Latour, too.

Again, this is not to deny that Roger has identified a genuine and crucial issue, which I believe he has. Also, I think he has made some interesting observations along the way. But unfortunately, he has not adopted a scientific attitude of self-doubt. His ideas appear to be merely stated, and nowhere tested. They are also stated rather vaguely, so the the implausibility of reaching the same taxonomy from two completely different sets of dichotomies is not especially striking, unless you actually pause to think about it.

22 comments:

bigcitylib said...

Don't blame sociology of science for Roger. I doubt most of them have heard of him. Their bete noir these days is Steve Fuller. At least Roger spares us the post modernism.

But the problem with normative taxonomies is that scientists typically gnore them. This was often pointed out gleefully with respect to Popper: nobody was the boyscout he required them to be, and in the final analysis (argued Feyerabend and Kuhn and others), being that boyscout would have been scientifically counter-productive.

Actually, there is an interesting comparison between Fuller and Pielke Jr. Fuller has often argued that we have to re-rig status quo science and the University to better support dissenting science (hence his interest in Creationism--which he has argued might potentially become a real science if encouraged). But the problem is, Fuller needs somebody to ENFORCE his program, someone within the university system to redistribute the wealth as it were to rebel scientists. Pielke Jr. similarly lacks any means of imposing his taxonomy,and its highly unlikely anyone will regulate their behavior to be in accordance with it otherwise.

In any case, sociology of science these days is very careful of proposing normative systems.

David B. Benson said...

RPJr is misusing the word "broker", IMHO.

richardtol said...

It's a science arbiter, rather than a science advisor. There is nothing circular in the definition. A good example is someone who suffers a routine illness, has decided she wants to live, and calls in a doctor.

The options bit is not hard either. You have a problem and you're not sure whether to do A or B. You call in an expert. An issue advocate will tell you to do A. An honest broker will tell you if A then W, if B then Y, but you could also do C and then Z will happen.

In other words, an issue advocate will tell you what to do; an honest broker will inform you and let you make up your own mind.

guthrie said...

I'm afraid I seem to have lost the point here along the way.
What was the point about coming up with the concept of the honset broker etc?
Is it supposed to be a model of how the world works or of how it ideally should work?

And in the case of AGW, can Roger point to definite examples where the roles/ concepts are confused?

David B. Benson said...

Richard Tol --- Fine, but that is what an adviser does, not what a broker does.

A broker carries out some actions for you, usually for a commision; stock, yacht and real estate come to mind as does pawn.

richardtol said...

@David
I'd be happy to drop the word "broker". I guess an honest broker would ask whether you really want to buy a yacht and would show you small yachts and big ones. But please replace the word with something less confusing if that helps.

Deech56 said...

So is "honest broker" the 21st version of the 18th century "disinterested gentleman"?

EliRabett said...

Richard, Eli has been making the point about what brokers really do for years. Another think you might consider is that brokers in regulated markets have to qualify their buyers, e.g. find out how much risk they can stand, what their financial status is, etc. and vet their sellers

To repeat, Roger's naive and incorrect injection (let's be nice) of the "honest broker" into science policy has pushed discussion into a fruitless direction as you now admit. As with many such things, reality shows how hollow this is. IEHO looking at what brokers do in the real world better illuminates the issue.

Brokers do not expand the scope of choices available to clients, they narrow them. Brokers make markets. Brokers make a living by matching buyers to sellers and taking a commission (You thought they do it for free? What carrot wagon you fall off of bunny?). Ethical brokers will go out on the market seeking product suited to clients and will seek clients suited to products available to them. Ethical brokers have mutual obligations to sellers and buyers, to qualify the buyers and vet the sellers, not to sell every piece of nuclear waste to every rube with a cell phone.

Good brokers know what is available for purchase and what their buyer's needs are. They select the best matches (with allowance for the front and back end fees they are going to collect). The broker you want often tells the client NO, don't do that. Where the client insists on committing financial suicide the ethical broker is obligated to tell the buyer to take the business elsewhere.

EliRabett said...

Richard, what you describe is exactly what any kind of adviser does, including a science adviser, but that is NOT what Roger is calling an "honest broker".

You would find that everyone here agrees with your definition of a science advisor and thinks that that is a correct description of what should happen, with the arguments coming at what point the adviser resigns if the principle takes up stupidity or cupidity as a hobby.

richardtol said...

@Eli
If any adviser acts this way, we can all sleep soundly.

willard said...

And so an honest broker is an advisor, an advisor able to propose and argue for and against every possible alternative. His role does not rely on the fact-value dichotomy, but sure relies on a completely disinterested posture. On the face of it, insisting on the continuum of facts and values is only useful in this discussion to obfuscate the fact that the honest broker is not very different from an idealized Dutch (i.e. intuitionist) mathematician. The dichotomy between interest and disinterest is still apparent.

The best one can do about this honest broker idea is to learn to market multiple choices:

http://www.ted.com/talks/malcolm_gladwell_on_spaghetti_sauce.html

The TED talk above shows how there was a revolution in the way we sell spaghetti sauce when a psychophysicist realized that there is not one spaghetti sauce that is better, at least on the grocery store shells, as my mom still don't sell hers. There are best spaghetti sauces for the palates of the customers, basically: regular, spicy and chunky.

These targetted choices should not take into account what people say, but what their palate tell us.

The best an Honest Broker could do is to do exactly that: examine the market "climatic tastebuds" and come up with a limited offer of products, each of them targetted to a market segment.

And only then we see that the main merit in the broker is not that he's honest. Why insist on being honest, anyway: does that mean that other models of conselling are dishonest? The main merit is that he's a broker.

Portraying him as an advisor is just marketing fluff for a scientific audience, still hooked on the disinterested image of scientists.

PS: I dealt with the analogy with a doctor consultationin the thread An in the End.

Nick Barnes said...

compared to Bruno Latour, Roger is another Bertrand Russell. But, alas, compared to Bertrand Russell, he is another Bruno Latour, too.
This is great. Is it original?

Michael Tobis said...

Nick, thanks. Yeah, I think so.

Michael Tobis said...

To Big City's comment, it's clear that science as a community is not going to be directly convinced by Roger's arguments, or even much take note of them. That's not the point, though.

Roger has attained to celebrity among journalists and thus gets a hearing inside the beltway. And as we can see, the consequence of his ideas is that the channels of communication between experts and government remain as constricted and obscure as ever.

Remarkably, Roger has staked out a normative position and is defending it from a posture of authority. Rather than claiming special status for science, he is simply claiming special status for his taxonomy!

No serious person can take this seriously, but politicians, by nature, are not entirely serious. They thrive on being friendly, not on being rigorous. When times are good, this is adaptive; otherwise (as Churchill showed Chamberlain) it is cowardice.

Much as we might wish otherwise, the century we are entering will likely not be forgiving of this sort of cowardice. What Roger is doing is offering the cowards cover.

Hank Roberts said...

> The best an Honest Broker
> could do is ... examine
> the market "climatic
> tastebuds" and come up
> with a limited offer of
> products, each of them
> targetted to a market
> segment.

Yeah, that produces a wide range of choices, all fast food, and all bad for you. They know what you want; they don't give you what you need.

Look at the result. http://www.msnbc.msn.com/id/36664612/

Look up "supernormal stimulus" to understand the problem with giving people what they want.
http://www.psywww.com/intropsych/ch08_animals/supernormal_stimuli.html

We don't limit the sugar, salt, and fat we eat -- we evolved in scarcity, to live within the natural limit, getting all of those we could whenever we could.

Same as we don't have a limit on our circadian system, and the body clock falls apart without a day/night cycle. Nature always provided an limit. We evolved to live within the limit.

willard said...

Hank,

We sure can count on you to get the point ;-)

What the honest broker for spaghetti sauce forgets to tell was that the choice was about manufactured and bottled spaghetti sauce, something that is made with tomato paste, so mainly bad tasting junk.

What the honest broker for spaghetti sauce will never tell you, is that you can make a very decent tomato sauce by:

- taking a can of unsalted italian tomatoes;

- adding 3 tbs of a good, virgin olive oil, a pinch of salt, crushed garlic, spices and spicy stuff to taste;

- blend everything together;

- put it in the fridge for two-three days for best taste.

That's it! One does not even have to cook it! In fact, one could take fresh tomatoes too! It's lovely as a pizza sauce; in fact, this **is** pizza sauce.

What exactly are you gaining for going for that fatty, salty, and atrociously tasting stuff? The satisfaction to have wasted money. We should never forget how good it feels to waste money, here a between 300% and 500% waste, or more if you can your own tomatoes.

Or is tom-â-toes, I never recall.

Scruffy Dan said...

Maybe I am missing something, but I don't understand the point of expanding policy options.

I thought the role of expert advisors was to reduce the policy options being considered. An expert analysis of policy options ought to give policymakers some idea as to which policies are better, and which are flawed.

Isn't that the goal of expertise? To remove the bad policy from the table (or at least recommend against it?)

Of course experts can propose new policy options, but at some point shouldn't they work to disqualify the inadequate policy?

Note, I purposely didn't define who has the expertise. It could be a scientist, or a political operative. The goal of the policymaker could be to resolve an issue, or to do something entirely for political reasons (or any mix of the two). Going back to the Sand berm example, Jindal's goal may have been to protect the shoreline, or could have been to do something/anything that makes him look good to voters, or as I think likely a little of both.

Paul Daniel Ash said...

Willard, I will argue from the authority of multiple generations of ancestors from Calabria and Abruzzo that your smoothie is not tomato sauce.

However, it sounds wonderful and I'll try it!

willard said...

PDA,


I'm not here to limit policy decisions regarding defining the original essence of tomato sauce ;-) If you want to share the trade secret of your authorities, I'll be delighted to hear them!

My addendum regarding that it was a **pizza sauce** was made because it comes from a book to make bread and pizza. But it's simple, cheap and good. One could also crush the tomatoes with a pestle to have crunchy sauce.

That there is a more than 500 millions dollars market in commercial spaghetti sauce is beyond comprehension. Teaching this kind of trick in school would be very important. That could help hide the decline in lifestyle.

Buying a good cookbook will always be better than buying Junior's book. Giving to Doctors without Borders too.

Hank Roberts said...

Attempt to post some quotes failed; I'll try just the pointers:

http://pubs.amstat.org/doi/pdf/10.1198/jasa.2010.ap09811
... we as individual statisticians, and the American Statistical Association (ASA) have been called to action.... to take on the role of an honest broker of policy alternatives. .... A current illustration in the evidence-based medicine context is comparative effectiveness research, which received focus and funding from the American Recovery and Reinvestment Act of 2009. The time is now for us to take action as a discipline, as individuals, and as an association to ensure that policy decisions are driven by evidence.


The neutral/honest broker role in foreign-policy decision making: a reassessment.

Presidential Studies Quarterly
June 1, 2005
http://www.highbeam.com/doc/1G1-134170677.html

Hank Roberts said...

>> a limited offer of products,
>> each of them targetted to a
>> market segment

> They know what you want; they
> don't give you what you need.

Credit due:

"Now the preacher looked so baffled
When I asked him why he dressed
With twenty pounds of headlines
Stapled to his chest
But he cursed me when I proved it to him
Then I whispered, 'Not even you can hide
You see, you're just like me
I hope you're satisfied'
...
And she says, 'Your debutante just knows what you need
But I know what you want'
Oh, Mama ..." -- Dylan

André said...
This comment has been removed by a blog administrator.