This documents some of my struggles trying to get some results out of climate models. The intellectual aspect is essentially nil - neither the scientist nor the programmer impulse is satisfied in any way by this nonsense. Basically I am dealing with a poorly documented code on a poorly documented platform with obscure error messages. The spawning of distributed memory jobs on a unique collection of processors with what amounts to a homebrew filesystem and a homebrew operating system which changes under your feet.
Once I had compiled code and tried to make a small change. It failed. I undid the change. IT STILL FAILED. I had a copy of the old executable and the old source. The old source and current source matched. The old executable and the current executable did not.
All of this is perfectly orthogonal to whether climate models are good models. But (if NCAR is any example) they are lousy software in terms of results per unit human effort.
My belief is that atmosphere models are excellent models for many purposes, ocean models are excellent models for some purposes, and coupled models are perhaps less fully tested than one might like but are absolutely necessary for our current state of knowledge.
But the experience of working with them is hellish, and being a single person trying to get them to do anything outside the environment in which they were developed is a lonely and demoralizing task. Frankly it (along with some other stresses outside the lab to be sure) almost killed me, and I am glad I am out of there and somewhat recovered.
Kate a.k.a. ClimateSight refers to this problem in a recent and interesting posting and has some comparable kvetching here; I promise you getting this stuff running at scale on an unsupported non-NCAR supercomputer facility is much harder than getting it to do a few timesteps on a commercial box.
Being a tenure-track scientist is tough, but settling for a staff scientist career (which I thought would be less stressful) really didn't turn out to be any picnic. There is much to say about this, some of which projects onto climateball obsessions and much of which doesn't. In general, climate folk don't complain much, partly for fear of giving aid and comfort to the enemy, and partly because there's a ludicrous geek cred in solving absurd and arbitrary problems - it doesn't do to whine too much.
But the problem is among those endemic to computational science. Brown. Knepley and Smith have a very cogent article on the subject "Run-Time Extensibility and Librarization of Simulation Software" whose abstract reads:
It is not only an important and excellent piece, but it also has quite an amusing opening section, though perhaps only to those who have suffered through the problem.Build-time configuration and environment assumptions are hampering progress and usability in scientific software. This situation, which would be utterly unacceptable in nonscientific software, somehow passes for the norm in scientific packages. The scientific software community needs reusable, easy-to-use software packages that are flexible enough to accommodate next-generation simulation and analysis demands.
And that, friends, is the story of how I pissed away several years of my life.