The only thing we can be sure of about the future is that it will be absolutely fantastic. So if what I say now seems to you to be very reasonable, then I have failed completely. Only if what I tell you appears absolutely unbelievable, have we any chance of visualizing the future as it really will happen.

- Arthur C. Clarke (h/t Brin)

Tuesday, November 13, 2007

Selling the Incomprehensible

Dear science funding agency;

I understand the desire for anonymous reviewers. I understand the reluctance to negotiate the terms of our engagement. I understand the reasons not to put proposers in touch with competitors who are also potential collaborators. I may not agree with these approaches but I understand them.

What I can't understand is why I have to act as if you were on a distant planet.

I can't understand why I don't get to put together a powerpoint presentation, and why I don't get to respond to reviewers' questions until it's too late.

It's clear that reviewers misunderstand proposals. Of course they do. That's not a criticism. It's an inevitable consequence of the complexity of what we do. These things are complicated, and your reviewers have a limited amount of time to get the idea.

The formalism of printed documents with tight page limits is just silly. It's proven every time we get our reviews back. Reviewers frequently miss the point. Didn't some of your reviewers miss the point of your last proposal? Most of them? Shouldn't we be making some efforts to be sure we understand each other?

Better ways have been developed to convey complex ideas. They are called "presentation" and "conversation".

I want to do what I would do in a business setting. I want to look you in the eye and explain to you why you would be foolish not to fund my proposal; i.e.;

1) that you have a problem,
2) that I know how to solve it
3) that my team has or can find the right people to solve it
4) that those objections which make any sense are already accounted for in the plan

If I can't look you in the eye, could we at least try instant messaging?

Why do you insist on presentation mechanisms that are practically guaranteed to fail to communicate the ideas and address the objections?

Why do you refuse to talk to me? What is the purpose of making multi-million dollar investments in an information vacuum?

Thanking you in advance for your prompt reply.

Update: Bryan Lawrence has substantial things to say about this, on his blog.

13 comments:

mz said...

Probably they think it would escalate out of hand, the presentations and wooing would get all more elaborate and the scientists' as well as the funders' time would be spent doing that discussion and presentation more than actual research or management.

Best way would be to allow some limited dialogue at least. And graphics, yes.

(I don't claim to know anything about this, just playing devil's advocate.)

James Annan said...

Oh dear. Another well-meaning gullible innocent to the slaughter...

These systems are not in place to identify the best proposals. They are there to protect and favour the powerful and famous. Much easier to trust the established names than take a punt on some new idea from an outsider, if they only have a couple of pages to outline that new idea. I think you can work out for yourself why there is no pressure for change...

One of the major benefits of getting out of the UK is we've managed to avoid this bullshit for the past 5 years. It is widely acknowledged that our research could never have been funded under that system (I'd already had plenty of knockbacks from people who had no idea what I was talking about, and who therefore assumed that I must have been the clueless one). I'm guessing that we have now passed the threshold at which our ideas may be taken seriously, but this is only a guess...

bryan lawrence said...

Well the official line is that we want to remove the possibility that if you look me in the eye I might cave in to your magnetic personality (and less magnetic science proposal) ...

... of course, this way we cave in to a minority of ill-informed reviewers who don't normally bother to read things properly.

(Did you hear about the recent palaver in the UK about junior doctor hiring in the NHS. It was anonomised (sp?) so well that the CVs had no job history, no publication record ... and in fact, no facts on which to make decisions ...)

Michael Tobis said...

Well it seems to me, middle-of-the-roader that I am that Bryan is too generous and James too cynical.

In my experience the more interesting the proposal less the likelihood of funding.

My experience is also that the fame of the PI has more impact than the value of the proposal, which somewhat validates James' point of view. I think the result is the same as he proposes, but the reason is just hidebound stubbornness rather than actual deliberate malice.

The fact is that as individuals we are overworked. The first thing to go is our unfunded obligation to do careful review. (Well, the first thing to go is genuine public outreach, but if that ever existed it was long gone by the time I showed up. Few hard scientists have the temerity to blog from a university account.)

I don't miss the singleminded pursuit of money in the private sector, but so many things we do in publicly funded science make so little managerial sense it is a wonder as much gets done.

Which is the main reason I'd like to see more private funding in the equation. Not because we need the money but because we need a little bit of adult supervision.

This is yet another example. Saying that it's a minority that doesn't understand the proposals at first blush seems to me tantamount to saying that the majority of proposals are pretty pedestrian. I wonder why that would be.

Bryan Lawrence said...

Well, I didn't mean to be generous :-)

There is no doubt that track record influences grant funding success (after all, why ask for the track record if you then ignore it).

There is also no doubt that getting a track record is getting harder and harder ... but I too dont believe it's malice, it's simply there are too many proposals and not enough fidelity in the refereeing, so panels are left with either a) too many proposals with the same high grade, or b) proposals which can be easily thrown out because one or more of the referees have said they are crap (in my experience not always correctly) ...

... but when you're sitting on a panel with proposal a with four high grades, and proposal b with two high grades and two low grades, what can you do?
Well, if you understand the science yourself (harder and harder given the breadth and depth of the material you see), then you can "moderate" the referees proposals ... but again, in my experience, that's when you get the "fund the devil you know" result.

And the same situation arises when you get two proposals with four high grades each. What are you left with to make a decision on? Track record or familiarity with the science. Same result ...

So what I meant to say was, mostly it's not enough quality reviews ...

(Which echoes Michael's point about time. I don't take enough time reviewing either, but then lately I've turned down doing them because I feel guilty about it).

Michael Tobis said...

OK, now Bryan is saying pretty much what I'm saying, except with the usual tone of resigned inevitability that accompanies just about everything in our misbegotten era.

Is it impossible to reconsider this?

Why is it that we have so much difficulty being sensible?

I'd like this article to get some attention, especially outside of science. If people in the private sector understood how casually their tax dollars were allocated they might be interested in addressing the situation.

Things have not yet reached a disastrous point, but the amount of frustration among the non-anointed is getting pretty substantial.

James Annan said...

As Bryan says, a fundamental problem is simply the pyramid scheme of oversupply that we are all part of. There are plenty of bloggers prepared to point this out, and sometimes even establishment players such as Nature write about it, even as the govt (all govts) talk about the "need" for more scientists. We are only "needed" in order to make sure that supply exceeds demand for current employment T&C.

Personally, I don't "need" the competition, and I think we should be honest enough to tell PhD students and the like exactly what their future prospects are. But so long as the system favours those who have sufficient authority to be heard, they will remain more concerned by their difficulties in finding post-docs and students rather than in ensuring their victims^Wproteges have reasonable career structure and opportunities beyond the next 3 year contract.

The standard (only) way to do interesting research is to steal time from your current project, and once you have got a couple of publications in this new area, you might be able to get a grant to do what you've already achieved. Of course this relies on you having a steady stream of funding for mundane research that doesn't really take as long as you pretend it will, which therefore has to be fronted by an established PI.

Bryan Lawrence said...

I'm torn between leaving this thread (because of overwork), or continuing it (because I agree that it's worth proposing change ... but then I did always have an intimate relationship with windmills). So here I am ...

At the moment the sequence goes: proposal, review, response, summary(*), decision. (*: for NERC, for whom I'm familiar, the summary step introduces an overall grade which may or may
not represent what the referees said ... and nearly always reflects the personal opinion of one or two folk on a large committee).

The problems amount to: 1) the reviewers may not be as expert as they think they are, and 2) may not give it the time it deserves, and 3) the summary step is nearly always done by folk who are even less
expert OR are opinionated in ways that ought to involve their exclusion from the process, 4) there is no way for a dialogue to at least remove misunderstanding from the sequence.

Sadly, 1 and 2 are probably beyond help, but 3 could be improved by allowing the response to referees reports to go back to the reviewer. At that point the reviewer either might gracefully admit defeat, or give it more time, and it ought to be clear if the latter hasn't happened (and thus the summary step should discount the first negative opinion). So this is more like a journal evaluation (and we all know that things go wrong with that, but at least it's better).

But, like Michael, I agree that (4) could make a big difference, but now I'm torn again. (Should we be influenced by personality or not?) Actually, I think by and large, personality does matter ... people who bullshit well are more likely to deliver something that someone else can use, because communication is obviously
something they do well ... the lonely genius might well do something great, but the rest of us might never know ... on average, I'd use communication skills to discriminate between two proposals that are otherwise similarly graded. In the UK there can often be a presentation step, but the level of discussion never really gets down to dealing with the grades and opinions of the reviewers (the proposer
never gets told the reviewers grades) ... and it probably should, but then you need the reviewers in the room (physically or virtually) to defend their opinions ... but that then makes reviews even more onerous (but those who are not expert are less likely to pretend they are).

What to do? A) I kind of like the IM concept, but really you'd need an whiteboard step too (it so often comes down to looking at a picture), and B) we need institutions to make headroom for proper time for reviewing, which will only happen if C) funding agencies accept the costs associated in doing so (they mostly like to think it's a free frictional cost, and it's not!)

Regrettably we have to start with C, probably at the government level I think ... and if I tried to do anything like that, my lance would be even blunter than it is now ...

inel said...

"I'd like this article to get some attention, especially outside of science. If people in the private sector understood how casually their tax dollars were allocated they might be interested in addressing the situation."

Well, I am glad you clarified that.

Mr. Gunn said...

As much as I like technology, I think the low-tech way this is done actually helps to level the playing field and put the focus on the content. The big fancy labs with extra money aren't able to get an advantage by hiring someone to make fancy presentations for them, a luxury junior faculty likely couldn't afford.

Of course, there are some kind of changes that need to be made to speed up the turnaround, but you can't make people answer their email.

EliRabett said...

Sorry I missed this when it started, but two points, one from the past and one from NIH.

In the "good old days" when there were no word processors and Adobe Acrobat, and copies cost well north of $0.20/page (yes Eli is that old) you didn't need page limits, because you had exhaustion limits for yourself, the typists (yes they existed) and the budgets.

With the coming of word processors and cheap laser printers and copying, two hundred page proposal packages with 40 pages of resumes, etc. became common. The Post Office rejoiced and the reviewers rebelled (think of the fun in a panel review when a bunch of these things landed on your doorstep and you had to read them and bring them to the panel). Thus page limits. As Eli said to an NSF guy this week, best thing you ever did was limit bio sketches to 1-2 pages.

NIH in the US has a useful procedure. Not much gets funded on the first go round. Resubmissions have to specifically address criticisms from the referees early on. You get tossed if you don't.

Michael Tobis said...
This comment has been removed by the author.
Michael Tobis said...

The NIH procedure is much better than what I've seen.

Both you and Bryan focus on the plight of the reviewer.

The reviewer's plight is not an issue in industrial procurement, because the reviewers are funded, i.e., stably employed. Also there is at least some idea that they should be rewarded for good decisions and penalized for bad. Also they control the volume and timing of information flow.

In contemporary science, where the reviewer is usually under pressure for his or her own viability, and the rewards for review are trivial and sometimes even perverse, the care put into the decisions is likely to be inadequate.

Furthermore the structure of the RFPs is bizarre...

Industrial RFP: please be the low bidder meeting the following specifications. NSF RFP: hmmm. Well, I don't know quite how to summarize it.

I'm sure this all emerged for good reasons. I'm not sure it works all that well.