tag:blogger.com,1999:blog-8524070301101240472.post5510596149504417882..comments2023-09-28T08:13:11.489-07:00Comments on Only In It For The Gold: Selling the IncomprehensibleMichael Tobishttp://www.blogger.com/profile/08229460438349093944noreply@blogger.comBlogger13125tag:blogger.com,1999:blog-8524070301101240472.post-15264560458998261162007-12-01T12:09:00.000-08:002007-12-01T12:09:00.000-08:00The NIH procedure is much better than what I've se...The NIH procedure is much better than what I've seen. <BR/><BR/>Both you and Bryan focus on the plight of the reviewer.<BR/><BR/>The reviewer's plight is not an issue in industrial procurement, because the reviewers are funded, i.e., stably employed. Also there is at least some idea that they should be rewarded for good decisions and penalized for bad. Also they control the volume and timing of information flow.<BR/><BR/>In contemporary science, where the reviewer is usually under pressure for his or her own viability, and the rewards for review are trivial and sometimes even perverse, the care put into the decisions is likely to be inadequate.<BR/><BR/>Furthermore the structure of the RFPs is bizarre... <BR/><BR/>Industrial RFP: please be the low bidder meeting the following specifications. NSF RFP: hmmm. Well, I don't know quite how to summarize it.<BR/><BR/>I'm sure this all emerged for good reasons. I'm not sure it works all that well.Michael Tobishttps://www.blogger.com/profile/08229460438349093944noreply@blogger.comtag:blogger.com,1999:blog-8524070301101240472.post-26098835786881388062007-12-01T12:07:00.000-08:002007-12-01T12:07:00.000-08:00This comment has been removed by the author.Michael Tobishttps://www.blogger.com/profile/08229460438349093944noreply@blogger.comtag:blogger.com,1999:blog-8524070301101240472.post-33968059737078344592007-12-01T11:47:00.000-08:002007-12-01T11:47:00.000-08:00Sorry I missed this when it started, but two point...Sorry I missed this when it started, but two points, one from the past and one from NIH.<BR/><BR/>In the "good old days" when there were no word processors and Adobe Acrobat, and copies cost well north of $0.20/page (yes Eli is that old) you didn't need page limits, because you had exhaustion limits for yourself, the typists (yes they existed) and the budgets.<BR/><BR/>With the coming of word processors and cheap laser printers and copying, two hundred page proposal packages with 40 pages of resumes, etc. became common. The Post Office rejoiced and the reviewers rebelled (think of the fun in a panel review when a bunch of these things landed on your doorstep and you had to read them and bring them to the panel). Thus page limits. As Eli said to an NSF guy this week, best thing you ever did was limit bio sketches to 1-2 pages.<BR/><BR/>NIH in the US has a useful procedure. Not much gets funded on the first go round. Resubmissions have to specifically address criticisms from the referees early on. You get tossed if you don't.EliRabetthttps://www.blogger.com/profile/07957002964638398767noreply@blogger.comtag:blogger.com,1999:blog-8524070301101240472.post-53732827718347634482007-11-16T20:23:00.000-08:002007-11-16T20:23:00.000-08:00As much as I like technology, I think the low-tech...As much as I like technology, I think the low-tech way this is done actually helps to level the playing field and put the focus on the content. The big fancy labs with extra money aren't able to get an advantage by hiring someone to make fancy presentations for them, a luxury junior faculty likely couldn't afford.<BR/><BR/>Of course, there are some kind of changes that need to be made to speed up the turnaround, but you can't <I>make</I> people answer their email.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-8524070301101240472.post-81124746086539497472007-11-16T16:17:00.000-08:002007-11-16T16:17:00.000-08:00"I'd like this article to get some attention, espe...<EM>"I'd like this article to get some attention, especially outside of science. If people in the private sector understood how casually their tax dollars were allocated they might be interested in addressing the situation."</EM><BR/><BR/>Well, I am glad you clarified that.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-8524070301101240472.post-43916956881183922562007-11-16T03:26:00.000-08:002007-11-16T03:26:00.000-08:00I'm torn between leaving this thread (because of o...I'm torn between leaving this thread (because of overwork), or continuing it (because I agree that it's worth proposing change ... but then I did always have an intimate relationship with windmills). So here I am ...<BR/><BR/>At the moment the sequence goes: proposal, review, response, summary(*), decision. (*: for NERC, for whom I'm familiar, the summary step introduces an overall grade which may or may<BR/>not represent what the referees said ... and nearly always reflects the personal opinion of one or two folk on a large committee).<BR/><BR/>The problems amount to: 1) the reviewers may not be as expert as they think they are, and 2) may not give it the time it deserves, and 3) the summary step is nearly always done by folk who are even less<BR/>expert OR are opinionated in ways that ought to involve their exclusion from the process, 4) there is no way for a dialogue to at least remove misunderstanding from the sequence.<BR/><BR/>Sadly, 1 and 2 are probably beyond help, but 3 could be improved by allowing the response to referees reports to go back to the reviewer. At that point the reviewer either might gracefully admit defeat, or give it more time, and it ought to be clear if the latter hasn't happened (and thus the summary step should discount the first negative opinion). So this is more like a journal evaluation (and we all know that things go wrong with that, but at least it's better).<BR/><BR/>But, like Michael, I agree that (4) could make a big difference, but now I'm torn again. (Should we be influenced by personality or not?) Actually, I think by and large, personality does matter ... people who bullshit well are more likely to deliver something that someone else can use, because communication is obviously<BR/>something they do well ... the lonely genius might well do something great, but the rest of us might never know ... on average, I'd use communication skills to discriminate between two proposals that are otherwise similarly graded. In the UK there can often be a presentation step, but the level of discussion never really gets down to dealing with the grades and opinions of the reviewers (the proposer<BR/>never gets told the reviewers grades) ... and it probably should, but then you need the reviewers in the room (physically or virtually) to defend their opinions ... but that then makes reviews even more onerous (but those who are not expert are less likely to pretend they are).<BR/><BR/>What to do? A) I kind of like the IM concept, but really you'd need an whiteboard step too (it so often comes down to looking at a picture), and B) we need institutions to make headroom for proper time for reviewing, which will only happen if C) funding agencies accept the costs associated in doing so (they mostly like to think it's a free frictional cost, and it's not!)<BR/><BR/>Regrettably we have to start with C, probably at the government level I think ... and if I tried to do anything like that, my lance would be even blunter than it is now ...Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-8524070301101240472.post-57482039041720078122007-11-15T18:55:00.000-08:002007-11-15T18:55:00.000-08:00As Bryan says, a fundamental problem is simply the...As Bryan says, a fundamental problem is simply the pyramid scheme of oversupply that we are all part of. There are plenty of bloggers prepared to point this out, and sometimes even establishment players such as Nature write about it, even as the govt (all govts) talk about the "need" for more scientists. We are only "needed" in order to make sure that supply exceeds demand for current employment T&C.<BR/><BR/>Personally, I don't "need" the competition, and I think we should be honest enough to tell PhD students and the like exactly what their future prospects are. But so long as the system favours those who have sufficient authority to be heard, they will remain more concerned by their difficulties in finding post-docs and students rather than in ensuring their victims^Wproteges have reasonable career structure and opportunities beyond the next 3 year contract.<BR/><BR/>The standard (only) way to do interesting research is to steal time from your current project, and once you have got a couple of publications in this new area, you might be able to get a grant to do what you've already achieved. Of course this relies on you having a steady stream of funding for mundane research that doesn't really take as long as you pretend it will, which therefore has to be fronted by an established PI.James Annanhttps://www.blogger.com/profile/04318741813895533700noreply@blogger.comtag:blogger.com,1999:blog-8524070301101240472.post-33470946101505124332007-11-15T15:41:00.000-08:002007-11-15T15:41:00.000-08:00OK, now Bryan is saying pretty much what I'm sayin...OK, now Bryan is saying pretty much what I'm saying, except with the usual tone of resigned inevitability that accompanies just about everything in our misbegotten era.<BR/><BR/>Is it impossible to reconsider this? <BR/><BR/>Why is it that we have so much difficulty being sensible?<BR/><BR/>I'd like this article to get some attention, especially outside of science. If people in the private sector understood how casually their tax dollars were allocated they might be interested in addressing the situation.<BR/><BR/>Things have not yet reached a disastrous point, but the amount of frustration among the non-anointed is getting pretty substantial.Michael Tobishttps://www.blogger.com/profile/08229460438349093944noreply@blogger.comtag:blogger.com,1999:blog-8524070301101240472.post-24512033613183931272007-11-15T13:57:00.000-08:002007-11-15T13:57:00.000-08:00Well, I didn't mean to be generous :-)There is no ...Well, I didn't mean to be generous :-)<BR/><BR/>There is no doubt that track record influences grant funding success (after all, why ask for the track record if you then ignore it).<BR/><BR/>There is also no doubt that getting a track record is getting harder and harder ... but I too dont believe it's malice, it's simply there are too many proposals and not enough fidelity in the refereeing, so panels are left with either a) too many proposals with the same high grade, or b) proposals which can be easily thrown out because one or more of the referees have said they are crap (in my experience not always correctly) ...<BR/><BR/>... but when you're sitting on a panel with proposal a with four high grades, and proposal b with two high grades and two low grades, what can you do?<BR/>Well, if you understand the science yourself (harder and harder given the breadth and depth of the material you see), then you can "moderate" the referees proposals ... but again, in my experience, that's when you get the "fund the devil you know" result.<BR/><BR/>And the same situation arises when you get two proposals with four high grades each. What are you left with to make a decision on? Track record or familiarity with the science. Same result ...<BR/><BR/>So what I meant to say was, mostly it's not enough quality reviews ...<BR/><BR/>(Which echoes Michael's point about time. I don't take enough time reviewing either, but then lately I've turned down doing them because I feel guilty about it).Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-8524070301101240472.post-63265631990154080622007-11-14T14:00:00.000-08:002007-11-14T14:00:00.000-08:00Well it seems to me, middle-of-the-roader that I a...Well it seems to me, middle-of-the-roader that I am that Bryan is too generous and James too cynical.<BR/><BR/>In my experience the more interesting the proposal less the likelihood of funding.<BR/><BR/>My experience is also that the fame of the PI has more impact than the value of the proposal, which somewhat validates James' point of view. I think the result is the same as he proposes, but the reason is just hidebound stubbornness rather than actual deliberate malice.<BR/><BR/>The fact is that as individuals we are overworked. The first thing to go is our unfunded obligation to do careful review. (Well, the first thing to go is genuine public outreach, but if that ever existed it was long gone by the time I showed up. Few hard scientists have the temerity to blog from a university account.)<BR/><BR/>I don't miss the singleminded pursuit of money in the private sector, but so many things we do in publicly funded science make so little managerial sense it is a wonder as much gets done.<BR/><BR/>Which is the main reason I'd like to see more private funding in the equation. Not because we need the money but because we need a little bit of adult supervision.<BR/><BR/>This is yet another example. Saying that it's a minority that doesn't understand the proposals at first blush seems to me tantamount to saying that the majority of proposals are pretty pedestrian. I wonder why that would be.Michael Tobishttps://www.blogger.com/profile/08229460438349093944noreply@blogger.comtag:blogger.com,1999:blog-8524070301101240472.post-36347240457560097282007-11-14T04:04:00.000-08:002007-11-14T04:04:00.000-08:00Well the official line is that we want to remove t...Well the official line is that we want to remove the possibility that if you look me in the eye I might cave in to your magnetic personality (and less magnetic science proposal) ...<BR/><BR/>... of course, this way we cave in to a minority of ill-informed reviewers who don't normally bother to read things properly.<BR/><BR/>(Did you hear about the recent palaver in the UK about junior doctor hiring in the NHS. It was anonomised (sp?) so well that the CVs had no job history, no publication record ... and in fact, no facts on which to make decisions ...)Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-8524070301101240472.post-6465637678107605982007-11-13T14:42:00.000-08:002007-11-13T14:42:00.000-08:00Oh dear. Another well-meaning gullible innocent to...Oh dear. Another well-meaning gullible innocent to the slaughter...<BR/><BR/>These systems are not in place to identify the best proposals. They are there to protect and favour the powerful and famous. Much easier to trust the established names than take a punt on some new idea from an outsider, if they only have a couple of pages to outline that new idea. I think you can work out for yourself why there is no pressure for change...<BR/><BR/>One of the major benefits of getting out of the UK is we've managed to avoid this bullshit for the past 5 years. It is widely acknowledged that our research could never have been funded under that system (I'd already had plenty of knockbacks from people who had no idea what I was talking about, and who therefore assumed that <I>I</I> must have been the clueless one). I'm guessing that we have now passed the threshold at which our ideas may be taken seriously, but this is only a guess...James Annanhttps://www.blogger.com/profile/04318741813895533700noreply@blogger.comtag:blogger.com,1999:blog-8524070301101240472.post-45218329990897367002007-11-13T08:02:00.000-08:002007-11-13T08:02:00.000-08:00Probably they think it would escalate out of hand,...Probably they think it would escalate out of hand, the presentations and wooing would get all more elaborate and the scientists' as well as the funders' time would be spent doing that discussion and presentation more than actual research or management.<BR/><BR/>Best way would be to allow some limited dialogue at least. And graphics, yes.<BR/><BR/>(I don't claim to know anything about this, just playing devil's advocate.)Anonymousnoreply@blogger.com