Even in the clearest cases of climate research, the main problems with climate science are almost all repeated across whole swaths of science.
There's an article at zdnet by Andrew Jones describing some of the problems in "HPC", i.e., "high performance computing" a.k.a. "big iron". It's fairly lightweight; it won't have anything new for people familiar with the field. But it may offer food for thought for outsiders.
The trick then must be to ensure the scientist code developer understands the methods of numerical software engineering, as well as its issues. Software engineers on the team must equally understand that the code is just part of the science, and not usually a goal in its own right.(Ooh, I just noticed. He said "trick"!)
Some unusual features of climate codes:
- there are no obvious whole system tests
- the constituent equations are not known at the spatial scale of interest
- the long-term ensemble average is of more interest than the particular solution
So while the problems are the same across the sciences at the broad-brush level, at the implementation level completely different thinking needs to go into the code; the architecture needs to reflect the testing strategy and the testing strategy needs to reflect the science.
In principle it's a beautiful, interdisciplinary problem. In practice, the relevant disciplines (computational science, software engineering, meteorology, oceanography and statistics) share little in the way of language or methods; even getting the closest pair (meteorology and oceanography) communicating was difficult.
Perhaps similar things can be said about other disciplines. I don't doubt that biological and biomedical applications have some uniqueness at the level of implementation detail such that a specialized branch of software engineering is required there too.
The existing pattern where software people are considered secondary has both the flaw that it is wrong in principle, and the flaw that it doesn't attract the best software people in practice. Throw in the truly dreadful legacy of hundreds of subtly incompatible versions of fortran and rapidly shifting platforms, and you have, well, a mess.
This doesn't mean that the work to date is anything short of a remarkable achievement. It may mean that we are reaching diminishing returns with existing practice.
Andrew Jones again:
However, we must also beware of the temptation to drive towards heavily engineered code throughout. Otherwise we run the risk that each piece of code gains a perceived value from historic investment that is hard to discard. And perhaps in some cases, what we need as much as renovation is to discard and restart.