Saturday, February 18, 2006

Peer Review & Replication: A Tale of Two Experiments

The focus of the open access movement is the peer-reviewed scholarly paper. Peer review is an important tradition in science, a way of assuring quality in a published paper through careful review and critique by the researchers peers. However, it is not the only means of ensuring quality in science, nor necessarily even the most important one!

Another way scientists assure that we are on the right track is through challenging our assumptions. This should include the assumption that peer review is truly necessary and desirable, right? It's only scientific!

Consider these two hypothetical experiments:

Experiment A: passes peer review, but cannot be replicated
The research is completed and the results written up for publication. The paper passes peer review with flying colours, and is published. However, a number of researchers attempt to replicate the results - without success!

Experiment B: replications with no peer review
The researcher posts methods and results on the world wide web. The experiment captures the imaginations of researchers around the world, and the experiment is replicated and the same results retrieved, over and over again. The researcher does not feel a need to publish in a peer-reviewed journal for career reasons; being more interested in research per se, the researcher just goes on with more experiments (posted on the web again) and never bothers with peer review at all.

The point I think I have made here is that in science, replicability of a study is a far better means of assessing the accuracy of research results than peer review. In an open source science environment, many experiments could be replicated in less than the average time it takes to publish a peer-reviewed paper. Indeed, if the experiment involves fairly simple equipment and techniques, why not have students involved in doing the replications?

In the electronic environment, it should be possible to develop means of tracking replications of a given experiment.

This does not necessarily mean that peer review would not be necessary and desirable; but perhaps it is not necessary for every piece of research.

Thanks to Drexel University's Jean-Claude Bradley and his Blogger Lab Notebook for the inspiration!

This post reflects my personal opinion only and does not represent the opinions or policy of the BC Electronic Library Network or the Simon Fraser University Library.

3 comments:

  1. Heather - the power of open source science extends even beyond that. We are getting comments from other chemists even before our experiments are completed because we are reporting our labwork in real time. Since this type of publication is constantly evolving, it will be a challenge for librarians to measure and analyse what is going on. As a scientist, the bottom line is that science is being done and communicated to others extremely quickly. See comments on this post for an example.

    ReplyDelete
  2. Jean-Claude, this strikes me as a much more useful form of peer review than the traditional one!

    For librarians, I think one important challenge we need to take on is to ensure that this work is being properly preserved, and made accessible on an ongoing basis. In other words, even though what is happening is a radical transformation in science, one of the tasks for us librarians is to return to our original roots - as collectors of information.

    Another role for librarians is helping our patrons to be aware of what is happening, and how things are different - part of our information literacy role. I suspect you and your team at Drexel don't need this kind of training, Jean-Claude, but many others could benefit from it.

    The teacher in a small, remote college, for example, who in the past may have had few or no opportunities to participate in collaborative research, now has such an opportunity. I'm guessing these folks would appreciate hearing about this new possibility from their librairans.

    ReplyDelete
  3. Here is Peter Suber's comment on Open Access News:

    Comment. Exactly. By permitting wider and more rapid dissemination than conventional publication, OA permits wider and quicker uptake, confirmation, and disconfirmation. This benefits everyone who cares about research and it doesn't exclude any form of peer review (traditional or innovative). Let a thousand forms of quality control bloom --or more simply, let's take full advantage of OA and the internet. They don't strap us into a single model of quality control but free us to try new ones and use multiple models simultaneously.
    From:
    http://www.earlham.edu/~peters/fos/2006_02_19_fosblogarchive.html#114036600733212181

    ReplyDelete

Thank you for your comment. Comments on IJPE are moderated.