"Our greatest responsibility is to be good ancestors."

-Jonas Salk

Tuesday, November 24, 2009

An Interesting Gripe

Never one to shy away from sticking his neck out, Ray Pierrehumbert (aka Raypierre) has actually submitted a top level article to Dot Earth. Ray's follow-up comment is especially interesting.
There is so much uninformed comment here I don't know where to begin. So many people are complaining about the "open-ness" of climate modelling, when in fact it is just about the most open area of science there is -- certainly more open than the economic modeling used to make trillion dollar decisions routinely. People complain about data sets not being released, when the data in question represents only a tiny slice of the total. You can get virtually all of the data from public sources. And do you want to see what's in a climate model? The algorithms used in the models are all documented in the peer reviewed literature. For most of the key models, you can get the source code and technical documentation and look at it for yourself -- many of the models even will run to some extent on a laptop.

So before you go on declaring a fatwa against climate scientists, why not take some time to get to know us? What we are like, and the way we do our work, bears no resemblance to the hateful cartoon you are pitching. And what do you think any of us would have to gain from a conspiracy to distort climate data anyway? If we were interested in making money, we could do it a lot more easily by just becoming investment bankers. Most of us are in this because we think the world is facing real peril, and we want to understand the nature of that peril better. A better world for my grandchildren to live in. Your grandchildren, too. Everybody's grandchildren, everywhere. Is that something to hate us for?
Now it's especially interesting to me because I do know Ray. I have confidence in his moral integrity and tremendous admiration for his intellectual capacity and domain knowledge. The trouble with this "getting to know Ray" (or someone like Ray) algorithm that Ray proposes is that, as any computational person worth his salt will immediately point out, it doesn't scale worth a damn. It is an effective technique at small scale, though, and I heartily recommend it to those few people who get the chance.

But it's also interesting what response this gets. "Biker Trash" replies
" There is so much uninformed comment here I don't know where to begin. "

Well, let's take a couple of specific examples that illustrate why this might be the case.

" So many people are complaining about the "open-ness" of climate modelling, when in fact it is just about the most open area of science there is -- certainly more open than the economic modeling used to make trillion dollar decisions routinely. "

Let's compare the open-ness of climate science with the degree of open-ness of the independent review and verification of, the Yucca Mountain Project, the certification of passenger aircraft by the FAA, construction of a nuclear power plant by the NRC, approval of a new drug by the FDA, construction of bridges, elevators, buildings, and many other cases that effect the health and safety of the public. This is the standard for the degree of open-ness required for all decisions that have the potential to effect the health and safety of the public.

Consider the Yucca Mountain Project. Every calculation, every piece of data, every detailed aspect of the independent review and verification of this project is online and accessible to anyone. Every nitty-gritty detailed part of every aspect without exception; http://ymp.gov/. Note that the basis of the Yucca Mountain Project is a natural process, directly analogous to that of climate science.

The same degree of open-ness is true for construction of nuclear power plants; check the NRC Web site; http://www.nrc.gov/reactors.html. The complete independent review and verification, not the extremely limited and incomplete information that appears in peer-reviewed literature, is publicly available.

Where is the documentation that the procedures and processes used to ensure that independent review and verification of climate science results have been properly applied. Where is the documentation of the procedures and processes that are required to be applied.

" And do you want to see what's in a climate model? The algorithms used in the models are all documented in the peer reviewed literature. "

The phrase, "the peer reviewed literature" is not very specific and the climate science literature is enormous. We have been told several times the same thing that you say here. Several people have tried to find the following information without success. Because climate science is your field of expertise and experience you could save us hours and hours of additional time if you could give at least a hint or two having more specificity.
BT goes on a bit of a tangent about computational fluid dynamics; an understandable and informed error. Many people think of climate physics as primarily a classical problem in fluid dynamics; they are misinformed in some fundamental ways that few people are qualified to explain in a way that will be acceptable to those within the field and convincing to those outside it. I have tried with very limited success and I won't try again just now. (In short, getting the fluid dynamics right to high order is not among the key problems. We aren't trying to solve a well-specified system. We are trying to specify the system.)

But the part of the complaint above, as far as I am concerned, is perfectly valid. Given the importance of climate modeling, one would expect a more rigorous and contemporary development process. Steve Easterbrook (along with his student Jon Pipitone) is the first person I know from the software engineering community to really take the development process of these models at face value and understand the some of the constraints that make the codes work the way they do. But Steve is just scratching the surface. There are efforts from within the community to improve matters, but I am among those who find them misguided.

As matters stand, it requires weeks to understand how to effectively run a climate model on a supported machine, and typically months to get it working on a new one. The success rate closely depends on how well integrated into the community one finds oneself. An outsider downloading a few blocks of code with no guidance will get nowhere.

So Ray's claim of openness reminds me of the part in the Hitchhiker's Guide where the bureaucrat defensively stated that Arthur's public condemnation had been duly publicized when the notice was in a locked cabinet in an unlit basement in a disused lavatory with a sign on the door saying "beware of the tiger!"

The thing is, climate scientists don't expect better, and never have seen much better. The fact is that despite the endlessly inflating claims of billions of dollars going to climate science, the software engineering staff on the leading American climate model has been reduced from eight to six, with the cuts coming from user support and documentation. This is why CCSM, though on release 3.1, is only documented to 3.0, and that only badly.

Yes, it's true, it's a trillion dollar problem, but the annual budget for studying it (once you subtract all the impact studies and the ecological observation programs and the satellite launches and satellite base stations, all very valuable but not what most people think of as climate science, and get right down to what is actually climate science) is about that of a high-profile Hollywood movie, and that for model development and maintenance on each individual model is far less. From 2003-2008 NCAR had to lay off approximately 55 people and lost another 77 positions due to attrition, totaling roughly 16% of NCAR positions, because of sub-inflationary NSF funding and decreases in other agency support. (Statement of program reductions at NCAR).

Here's a presentation of how NCAR is meeting a 9 million dollar shortfall this year. (PDF)

Yes, the support for the software used in climate science sucks. I promise you I hate it more than you do. Maybe if you all would stop treating climate science as a heinous enemy and supported improvements, or hell, even complete rewrites, things would get better.

What you see is what we get every bit as much as it is what you get. The climate community should stop pretending to any practical openness; in fact the only way to play is to sign up for a tour of duty. This is not because we are hiding anything. It is because we lack the resources (and to some extent the skills) to do any better. What is perceived as an intent to hide is mostly an incapacity to elucidate, exacerbated by some excesses of competitive zeal which come from a tightly competitive grants process.

8 comments:

Pangolin said...

So if I get you right the budgets of "Happy Feet" and "Wal-E" to mention two recent movies that had some climate relevant content got considerably more cash than the entire staff operating budgets of global climate scientists? Or would that be just the US effort? Why do I find this entirely believable.

I would suggest more penguins. LOTS more penguins.

Michael Tobis said...

I would say that the entire staff operating budgets for physical climatology in the US is comparable to Wall-E, somewhat more than Happy Feet, but it really depends how you count and where you draw the line.

People saying the climate change budget is in the low billions (which would be a couple of dozen big movies) are including lots of other stuff, dominated by the NASA earth observation satellite budget.

Anonymous said...

You're correct, Michael. The annual budget for the software engineers and others who work on a climate model wouldn't get Tom Hanks out of bed to work on the picture.

Marion Delgado said...

I saw someone ranting about Ray once, and I assumed it was Ray Ladbury, who's confrontational, as I am. But it was Ray Pierrehumbert. Ray has always gone the extra mile to explain anything to anyone anytime. You're right about it not scaling, though.

I remember during the later stages of work on ozone depletion it was all pretty personal. You emailed data or a comment and it was great, because that person in Antarctica didn't have to wait for his data or hook up to precious computer or net resources beyond mail coming in. If someone had a critique of something, they emailed the person who did the research, they knew what they were talking about, and it was CCed to anyone else who might have info.

That's the world that builds a strong science community, the one Ray Pierrehumbert lives in, but now climate science is making science part of public policy, just as scientists warned in the 1970s - when the Anthropogenic Era was just flexing its wings.

I share Chris Mooney's frustration: in many areas of science, we have many times more graduates than positions, and we have an unmet need for public policy intermediaries and presenters.

I'd love to do work like that, as a science-trained journalist, but paid work like that is not out there.

Aaron said...

And, how much do oil companies/NMA give to support the Newshour? (America's most trusted souce of news. Can we trust the commecials that surround the Newshour?)

Why can't the Newshour simply state that GHG cause global warming without throwing a bunch of lawyer weasel words in the sentence? Climate science is simply not getting through to the public. Even PBS will not just state the truth.

What the public hears every day is: Don't worry, the oil companies are aware of the problem and are working on it. The take away is, "do not worry about global warming." Thus, according to the WP poll, fewer people now consider GW a problem. All with a tax deductable contribution to PBS.

Why should the public put money into science, when the oil companies are already working on the problem?

Why can't we put out our message as well as the oil companies put out their message?

TokyoTom said...

Michael, thanks for these posts. Are any insurers involved in the modelling process, as far as you know?

Anonymous said...

There is a significant difference between "Biker Trash" examples and that in Climate Modelling. Yucca Mountain and commercial airline certification are a matter of public safety, wheras climate modelling is one of having the potential of influencing public policy. This potential to influence policy in part depends on whether policy makers have confidence in the results. For the public safety issues, the potential is that people die in an accident. Both are very important, but the different purposes logically lead to different standards.

Michael Tobis said...

Tom, not to my knowledge.

It's conceivable that some private interests would have a climate model (in the sense of a coupled GCM) that is substantially better than the public ones, I suppose. If they did, it is interesting to speculate what motives they would have for keeping it private vs publishing it.

I think that is not the case, however.

The only private climate modeling effort I have ever heard of, ironically, was funded by an oil company studying paleoclimate to determine where to look for petroleum deposits.