On the “quilt plot”

One of the big blow ups on twitter last week (at least in the circles that I follow) was the “quilt plot”, an article recently published in PLoS ONE. The quilt plot is actually just a heat map. Not a heat map with other bits removed, but a heat map. The article itself says that “they produce a similar graphical display to ‘heat maps’ when the ‘clustering’ and ‘dendrogram’ options are turned off”, but that misunderstands what a heat map is. While the wikipedia page for heat maps has a hierarchically clustered heat map as the example image at the top, the examples farther down do not. To claim that there is something new here is to fundamentally misunderstand what already existed.

It’s not really new, but scientists can’t be familiar with the entirety of the literature, and often “rediscover” old techniques from other fields. What one would hope, however, is that the journal itself is able to spot this, and respond appropriately. PLoS ONE actually has guidelines for whether new software methods should be published. Briefly, they say they must have

  1. Utility
  2. Validation
  3. Accessibility


I’ll tackle these in reverse order:

Accessibility

PLoS says:

“Software should be open source, deposited in an appropriate archive, and conform to the Open Source Definition.”

I suppose it is technically open source (though there’s no specific license for the code, I would assume that it takes on the CC-BY license that the article itself gets. It’s not really deposited in any of the normal archives (e.g. Github) anywhere I can find, but PLoS does store the supplemental files as well, so I suppose that does count.

It’s worth pointing out that in the supplemental file itself, the critical R code is actually screenshots pasted into a Word document. The Open Source definition says that it has to be “The source code must be the preferred form in which a programmer would modify the program.”

So we’re at 2/3 on accessibility. If we give them the same curve I often give undergrads, we’ll call that a C.

Validation

Again, PLoS:

“Submissions presenting methods, software, databases, or tools must demonstrate that the new tool achieves its intended purpose. If similar options already exist, the submitted manuscript must demonstrate that the new tool is an improvement over existing options in some way.”

They do show that it achieves it’s purpose. They also show, somewhat, that it’s an improvement, because it’s already configured for the adding labels, I guess. PLoS doesn’t say it has to be a significant improvement, so I guess they do meet the low bar of “in some way”.

A-, following the letter of the law here.

Utility

The tool must be of use to the community and must present a proven advantage over existing alternatives, where applicable. Recapitulation of existing methods, software, or databases is not useful and will not be considered for publication. Combining data and/or functionalities from other sources may be acceptable, but simpler instances (i.e. presenting a subset of an already existing database) may not be considered.

I suppose they do prove the advantage over the unconfigured version, but looking at the next two sentences, I think it’s fair to say that this is both a recapitulation of existing software, and that it’s a simpler instance. Thus, we have a combination of “will not be considered” and “may not be considered”. F.

So what happened? Based on this comment thread, it seems like the academic editor thought the editorial office was handling it, and the editorial office thought the academic editor was handling it, and it got approved in the shuffle. PLoS isn’t the only place I’ve heard of this happening (a friend had a paper accepted on resubmission of the original submission, then rejected, then later provisionally approved at the same journal based on what he can only surmise was based on selecting the wrong item from a dropdown). Is it good that this happened? Certainly not. Does it mean PLoS ONE is fatally flawed? I don’t think so either, and I would suspect that editors will be more careful, at least for a while until they forget about this. No one wants to be remembered as the guy who lets this kind of thing through when they shouldn’t.

Leave a Reply

Your email address will not be published. Required fields are marked *