Spending XMas day trying yet again to convince NSF to let me, not so much rewrite the climate models, as redesign the architecture of the models to match the workflow.
Given the nature of the call, the following is probably not going to strengthen my argument, but I think it's interesting and I welcome your input. (I'd especially welcome commentary from JM and JM).
The difficulties in constructing working high-performance codes color the scientific process and other decision support networks dependent on it.
To some extent the problem in climate modeling is based on the origins of the component models in operational prediction communities (such as weather prediction), wherein the goal of software design is the cost-efficient optimal projection of the state of the atmosphere into the near future. It's often noted that this is an initial value problem while running similar codes in climate mode is a boundary condition problem; the objectives are substantially different. Nevertheless, a weather code has a climate and a climate code has weather; these are structurally similar. Accordingly, the methods of the weather modeling community are injected into climate methodologies.
The problem is not in the different mathematical structure of the purposes at hand. It is in the different social structures. A weather model is write-once, run many times. Its purpose is efficiency and correctness. A climate model is an experimental platform. While it is efficiency constrained, flexibility and transparency are key to its utility, keys which are of trivial importance in operational settings.
I believe that despite the very slow progress of the past decade or so, climate modeling has the potential to be vastly more skillful. It seems at least that this should be put to the test. Flexibility, transparency, interoperability, testability and accessibility to automated reasoning are needed. Climate modeling needs to partake of modern agile development methodologies such as those at Google and similar very high productivity companies. Certainly the potential value add is there. It's time that some institutional structure existed to support this.