Thursday, February 28, 2013

The nature of publications

A paper in the journal of Genome Biology and Evolution has been doing the rounds on the internet recently and was shown to me by a friend. It is titled "On the immortality of television sets: “function” in the human genome according to the evolution-free gospel of ENCODE", by Graur et al. The title is blunt enough, but the abstract is extraordinarily so. Let me quote the entire thing here:
A recent slew of ENCODE Consortium publications, specifically the article signed by all Consortium members, put forward the idea that more than 80% of the human genome is functional. This claim flies in the face of current estimates according to which the fraction of the genome that is evolutionarily conserved through purifying selection is under 10%. Thus, according to the ENCODE Consortium, a biological function can be maintained indefinitely without selection, which implies that at least 80 – 10 = 70% of the genome is perfectly invulnerable to deleterious mutations, either because no mutation can ever occur in these “functional” regions, or because no mutation in these regions can ever be deleterious. This absurd conclusion was reached through various means, chiefly (1) by employing the seldom used “causal role” definition of biological function and then applying it inconsistently to different biochemical properties, (2) by committing a logical fallacy known as “affirming the consequent,” (3) by failing to appreciate the crucial difference between “junk DNA” and “garbage DNA,” (4) by using analytical methods that yield biased errors and inflate estimates of functionality, (5) by favoring statistical sensitivity over specificity, and (6) by emphasizing statistical significance rather than the magnitude of the effect. Here, we detail the many logical and methodological transgressions involved in assigning functionality to almost every nucleotide in the human genome. The ENCODE results were predicted by one of its authors to necessitate the rewriting of textbooks. We agree, many textbooks dealing with marketing, mass-media hype, and public relations may well have to be rewritten.

The paper that Graur et al. implicitly deride as "marketing, mass-media hype and public relations" is one of series of publications in Nature (link here for those interested) by the ENCODE consortium. I'm not going to claim any expertise in genetics, though the arguments put forward by Graur appear sensible and convincing.1 But I do think it is interesting that the ENCODE papers were published in Nature.

Nature is of course a very prestigious journal to publish in. In some fields, the presence or lack of a Nature article on a young researcher's CV can make or break their career chances. It is very selective in accepting articles: not only must contributions meet all the usual requirements of peer-review, they should also be judged to be in "the five most significant papers" published in that discipline that year. It has a very high Impact Factor rating, probably one of the highest of all science journals. In fact it is apparently one of the very few journals that does better on citation counts than the arXiv, which accepts everything.

But among some cosmologists, Nature has a reputation for often publishing claims that are over-exaggerated, describe dramatic results that turn out to be less dramatic in subsequent experiments, or are just plain wrong.One professor even once told me – and he was only half-joking – that he wouldn't believe a particular result because it had been published in Nature.

It is easy to see how such things can happen. The immense benefit of a high-profile Nature publication to a scientist's career leads to a pressure to find results that are dramatic enough to pass the "significance test" imposed by the journal, or to exaggerate the interpretation of results that are not quite dramatic enough. On the other hand, if a particular result does start to look interesting enough for Nature, the authors may be – perhaps unwittingly – less likely to subject it to the same level of close scrutiny they would otherwise give it. The journal then is more reliant on its referee's to provide the scrutiny to weed out the hype from the substance, but even with the most efficient refereeing system in the world given enough submitted papers full of earth-shattering results, some amount of rubbish will always slip through.

I was thinking along these lines after seeing Graur et al.'s paper, and I was reminded of a post by Sabine Hossenfelder at the Backreaction blog, which linked to this recent pre-print on the arXiv titled "Deep Impact: Unintended Consequences of Journal Rank". As Sabine discusses, the authors point to quite a few undesirable aspects of the ranking of journals according to "impact factor", and the consequent rush to try to publish in the top-ranked journals. The publication bias effect (and in some cases, the subsequent retractions that follow) appear to be influenced to a degree by the impact factor of the journal in which the study is published. Another thing that might be interesting (though probably hard to check) is the link between the likelihood of scientists holding a press conference or issuing a press release to announce a result, and the likelihood of that result being wrong. I'd guess the correlation is quite high!

Of course the only real reason that the impact factor of the journal in which your paper is published matters is that it can be used a proxy indication of the quality of your work for the benefit of people who can't be bothered, or are unable, to read the original work and judge it on merit.

The other yardstick by which researchers are often judged is the number of citations their papers receive, which at least has the (relative) merit of being based on those papers alone, rather than other people's papers. Combining impact factor and citation count is even sillier – unless they are counted in opposition, so that a paper that is highly cited despite being in a low-impact journal gets more credit, and a moderately cited one in a high-impact journal gets less!

Anyway, bear these things in mind if you ever find yourself making a reflexive judgement about the quality of a paper you haven't read based on where it was published.

1The paper includes a quote which pretty well sums up the problem for ENCODE:
"The onion test is a simple reality check for anyone who thinks they can assign a function to every nucleotide in the human genome. Whatever your proposed functions are, ask yourself this question: Why does an onion need a genome that is about five times larger than ours?"
2Cosmologists (the theoretical ones, at any rate) actually hardly ever publish in Nature. Even observational cosmology is rarely included. So you might regard this as a bit a of a case of sour grapes. I don't think that is the case, simply because it isn't really relevant to us. Not having a Nature publication is not a career-defining gap for a cosmologist: it's just normal.

Tuesday, February 19, 2013

Things to Read, 19th February

I have just arrived in Helsinki, where I am visiting current collaborators and future colleagues at the Helsinki Institute of Physics for a few days. I will give a talk next Wednesday, about which more later. In the meantime though, a quick selection of interesting things I have read recently:
  • Did you know that about 6 million years ago, the Mediterranean sea is believed to have basically evaporated, leaving a dry seabed? This is called the Messinian Salinity Crisis, which I first learned about from this blog. There's also an animated video showing a hypothesised course of events leading to the drying up:

    Very soon after, the Atlantic probably came flooding back in over the straits of Gibraltar – an event known as the Zanclean Flood – and, according to some models, could have refilled the whole basin back up in a very short time. Spare a thought for the poor hippopotamuses that got stuck on the seabed ...
  • A long feature in next month's issue of National Geographic Magazine is called The Drones Come Home, by John Horgan. Horgan has written a blog piece about this at Scientific American, which he has titled 'Why Drones Should Make You Afraid'. In the blog piece he has a bullet-point summary of the most disturbing facts about unmanned aircraft (military or otherwise) taken from the main piece. Some of these include:

    - "The Air Force has produced [a video showing] possible applications of Micro Air Vehicles [...] swarming out of the belly of a plane and descending on a city, where [they] stalk and kill a suspect."
    - "The Obama regime has quietly compiled legal arguments for assassinations of American citizens without a trial"
    - "The enthusiasm of the U.S. for drones has triggered an international arms race. More than 50 other nations now possess drones, as well as non-governmental militant groups such as Hezbollah."

    Scary stuff; worth reading the whole thing.
  • I wrote some time ago about Niall Ferguson's argument about economics with Paul Krugman (this was in the context of a lot of nonsense Ferguson was coming up with at the time, both in his Reith lectures for the BBC, and in other publications). I just learned (via a post by Krugman, who also just learned) that Ferguson had already apparently admitted that he got it wrong, about a year ago. Krugman's response to that is here; I'd add that I notice this admission didn't seem to stop Ferguson continuing the same economic reasoning in his Reith lectures a few months later!
  • A review of John Lanchester's new novel Capital, by Michael Lewis in the New York Review of Books. Quite often Almost always with the NYRB, I read reviews of books before I have read the actual book. In this case the result was to make me resolve to buy a copy.

Saturday, February 16, 2013

Oxford Greenland Expedition

I sometimes wonder about the mix of topics I mention on this blog – is there too much physics, or not enough? Well, today's post is about something completely different: it is a straight-up publicity plug for the Oxford Greenland Expedition. This is a sea-faring and rock-climbing expedition organised by a group of students, and is exactly the kind of exciting exploratory adventure to which the title of this blog refers, so I'd like to support it as best I can. They have a fundraising page here. As I'll try to explain, I think it is a great plan, and they deserve all the support they can get!

The climbing members of the expedition team are all current or former students of Oxford University, and members of the Oxford University Mountaineering Club, whom I have known for a varying number of years (I've climbed on the same rope as many but not all of them). Between them, they have dreamt up an outrageously bold and brilliant plan for a climbing expedition in the Arctic – as the 'Objectives' page of their website puts it, their goals are:
  1. Sail to Greenland!
  2. Climb the Horn of Upernavik!
  3. Sail north! Find an 800 m pillar! Climb it!
  4. Explore for new climbs!
  5. Sail back!
(The rest of the website is quite amusing too, if you poke around it.) The Horn of Upernavik appears to be a very dramatic 1000-metre-high piece of rock rising straight out of the sea near Uummannaq:

The Horn of Upernavik.
It has apparently also been the objective of several previous expeditions to climb the Horn, but none of them have succeeded. So if this team succeeds, they will be making (a small amount of) history ...

Although all are very good climbers and mountaineers, they are definitely firmly within the ranks of the enthusiastic amateurs rather than elite professionals. I think this makes the audacity of the undertaking all the more wonderful, and appealing to romantic ideals.

Incidentally, two of the members of this team (Ian Faulkner and Tom Codrington) were last year part of a different and equally inspiring expedition to Krygyzstan, during which they, along with Ian Cooper, made only the second ever free ascent of the Mirror Route or Rusyaev Route on Peak 4810, after Alex Lowe and Lynn Hill. A report of this climb and some other stuff they did was published here. So they've definitely got a track record with achieving amazing things!

Anyway, best of luck to the team in their efforts this summer!

Tuesday, February 12, 2013

Seeing neutrinos in the CMB

One of the really cool things about cosmology is the fact that you can look at the sky and use the cosmic microwave background to determine that there must be at least three generations of neutrinos, without having directly observed them in any ground-based experiments.

Well, actually we already knew that there were at least three neutrino generations from ground-based experiments before we had anywhere near the technological capability to determine this from the CMB, but it's arguably still pretty cool. What would inarguably be pretty amazing is if we could use CMB observations to prove that there were further, yet-undiscovered, generations of neutrinos that we could then go and look for elsewhere. Or, for a particular type of person, it might be quite satisfying to use the CMB to prove that there definitely weren't any more than the three generations we already know about, thus killing a lot of other people's pet theories stone dead.

All this is (somewhat) topical because of the release of new data from three experiments measuring the CMB: the Wilkinson Microwave Anisotropy Probe (WMAP) satellite, the Atacama Cosmology Telescope (ACT) and the South Pole Telescope (SPT), that could in principle do something like this. Given my various recent blogging delays, however, I haven't been particularly quick about writing about this, and there are other good discussions elsewhere on the web, especially at Résonaances, that many readers will already have seen. I'm also afraid I don't have time for a layperson-level introduction to the topic right now. But there's a bit of a jumble of information from the different collaborations, some of it mildly contradictory, and in this post I'd like to – partly simply as a note for myself – summarise it all and sort it into a logical order.

In case you'd like to just see the executive summary, it is that – in my opinion – the new data doesn't tell us anything very interesting. For a fuller justification of this statement, keep reading.