Sticking with a technical theme for a bit, and following onto my most recent previous article, I'd like to further address what I perceive as the limited productivity of climate modeling as an enterprise.
Getting software right is a very subtle skill. In general it is much easier to write barely useful software than to write powerful software, and broken software is easier still. On the other hand, observing that it is possible to do much better than has been done in the past has become trivially easy.
That is is possible to get software right is proven daily in the commercial sector. In my commercial career and especially more recently in the Chicago Python group I have been exposed to extremely talented programmers, some of whom are literally orders of magnitude more productive than ordinary programmers.
They are very particular regarding their choice of tool as any highly talented and productive person is, and many of them tend to choose Python or similar tools for most of their work. Of course, I cannot paint like Rembrandt even given the finest brushes and paints, but that doesn't mean brushes and paints don't matter. The tool isn't the point, though. The point is that a new approach has emerged.
You don't have to take my word for it anymore. The consequences of these capacities, interestingly developed almost entirely outside the academic sector, are increasingly visible through the productivity of Google, the largest collection of people with this sort of talent. Many smaller companies and organizations also partake of this cluster of techniques, but the remarkable productivity of Google is visible to anyone who cares to look.
So let me flip my question from the previous article, which was, whether computational technique can help to increase the rate of progress in climate modeling? Here is a recasting of that question:
Can we be sure that the greatly improved methodologies recently developed in the private sector can't be applied to climate modeling and related endeavors?
It seems to be obviously premature to say so.
The question, then, is whether climate modeling (and computational science, at least as applied to complex natural systems) matters. If it doesn't, we should all take up macrame or real estate. If it does matter, we should be paying very close attention to what works elsewhere. Not everything will apply, but what will apply will not be nothing, either.
Should Silicon Valley folks just build a climate model? It's a risky endeavor and I don't see a viable business proposition, but maybe, maybe somehow.
Should they just fund a climate model and not get involved? No. Foundation money directed toward software is probably even more likely to go astray into worthless boondoggles than government money.
Should they just shrug and forget the whole sorry business, accept that CO2 is a major forcing, and get on with other things? Maybe but I hope not. This abandons the adaptation side which is going to matter, trust me.
So I think the problem is institutional and motivational, not technical.