Kudos to Taylor & Francis for releasing early results before they complete analysis. Following are some thoughts which I will update a bit at a time, through a series of posts which I will link to from here, in line with T&F's plans to release analysis through a series of press releases.
Attitudes and values concerning research communication
Attitudes and values regarding dissemination and licenses
Overall comments
Added March 19, 2013: this survey presents an interesting twist on CC-BY. The PDF is licensed CC-BY but when I tried to save it I got this message: "Click "sign" to fill out and sign this form. When you are done, you can save a copy by clicking "done signing".
As an open access advocate, I would like to begin with applause at what I see as a survey with significant methodological problems which nevertheless is an interesting case study in its own right, and in particular because the questions selected suggest both a tilt in bias towards open access and a will to address the practical issues of how to make open access happen. Since I am discussing bias, I should mention my own - I am a librarian and a scholar and I advocate for Freedom for Scholarship in the Internet Age, which includes open access but also scholarly communication led by and for scholars and in the public interest, and I see corporate concentration in scholarly communication as a problem, with open access corporate and corporate-like not-for-profit concentration and power only slightly less problematic than toll access.
While this survey is somewhat useful, this kind of research should be led by academics, not publishers. While publishing companies can and should do research to develop their products and services (this is reflected to some extent in this survey), basic humanities and social sciences research, and particularly research designed to inform public policy developments, should be led by academics. The fact that this survey was conducted by a multinational conglomerate corporation (using the scholar-friendly-sounding brand names Taylor & Francis and Routledge) who obviously control access to this network of tens of thousands of researchers around the world and in effect have the ability to act as gatekeepers or exert monopoly control over the ability to conduct certain kinds of research in a practical manner should ring alarm bells over the extent of power that academics have handed over to publishers. I would encourage Taylor & Francis staff to consider this seriously and establish a committee led by academics to address this issue, to have academic researchers make decisions about surveys that go beyond the practical matters of publishing.
It is unfortunate that the survey questions were developed by the publishing companies' staff, who despite what might be the best of intentions bring a certain perspective and bias to this research. It would have been better to have taken more time to develop a better survey instrument before sending this to tens of thousands of researchers. Aside from the time this has taken up for Taylor & Francis staff and the respondents, with survey research respondent fatigue is a factor to take into account. Researchers who responded to this survey might refuse to participate in another, better designed survey in future.
For example, the following 3 questions reflect a strong bias against open access (p. 6 of the survey). T&F questions are bolded.
"Open access journals are lower quality than subscription journals".
This is a leading question; it's inviting the respondent to think of open access journals as lower quality. Consider for example other possible questions, such as:
"Open access journals and subscription journals are about the same quality".
or
"Open access journals are the same or better quality than subscription journals".
The same can be said about the next two questions:
Open access journals have lower Production standards (copyediting and typesetting) than subscription journals.
Consider this possible alternative:
"Open access journals have the same or better Production standards (copyediting and typesetting) than subscription journals".
and
"There are no fundamental benefits to open access publication".
Possible alternative:
"There are fundamental benefits to open access publication".
Comment: if you're going to ask tens of thousands of scholars to respond to a survey, you should take the time to get the questions right. It would have been appropriate to have the questions developed, or at least reviewed, by academics trained in social bias analysis, and preferably without an attachment either to toll access publishing or to Taylor & Francis per se.
Update March 18 - further comments in response to a comment by Thomas Pfeiffer on Richard Poynder's Google G+ post which alerted me to this study.
Thomas Pfeiffer:
Even though I do still agree that such politically sensitive surveys should be conducted by neutral organizations, I don't really agree with your analysis of the survey. To me, the questions on page 6 of the survey merely reflect common prejudices about OA, both positive and negative, and checks how widespread they are. It is usual in social science surveys to use questions framed both positively and negatively for the same construct, which is what they are doing here.
If they reframed the negative items, they'd also have to reframe the positive ones like "Open access offers higher visibility than publication in a
subscription journal" and "Open access journals are cited more heavily than subscription journals". In fact, question 1 has six "pro OA" items and only three "anti-OA" items. I don't see how that is supposed to set an anti-OA frame.
[Disclosure: I am a scholar as well and doing this kind of research has been central to my undergrad studies as well as my current career]
My comments
In one sense I agree, Thomas. This is 9 misleading or biased questions, with more biased in favour of open access than against. This is actually useful evidence - moreso, to me, in the sense that "a large publisher is choosing to including more questions in their survey biased towards open access than against it" than in terms of making the results meaningful.
If the aim is to get at researcher's attitudes, then this is the set of questions to ask with respect to quality - strongly agree to disagree:
Open access journals are lower quality than subscription journals.
Open access journals are the same quality as subscription journals
Open access journals are higher quality than subscription journals.
Even here, there is a strong probability of order bias in the questions. If you're doing a survey that based on a 19% response rate must have been sent to a hundred thousand researchers, you should control for this (for example, by changing the order of the responses on different survey forms).
Thanks for raising this, though, I'll add this to my post.
Hello Heather, I'm one of the people at Taylor & Francis / Routledge responsible for the survey. Firstly, thank you very much for looking through the (lengthy!) document, and for your critique. I think that it's important to note that we don’t believe that this survey should replace academic research into this area, more that we wanted to get a feel for our authors' thoughts on these various issues for our own use. We thought that others might find the information useful, if not perfect, and that it might help stimulate and guide further more academically robust research. Furthermore, we didn’t feel that we could pose all the attitudinal questions as positive, negative and neutral as the survey was already quite long, as those of our authors who took the time to respond to our questions can testify. I hope that provides some clarity.
ReplyDeleteThank you for your comment, Vicky. I am sure that you and Taylor & Francis had good intentions in conducting this survey, and I very much appreciate that the questions are indicating a bit of a shift in bias towards open access. However, to me this survey is an interesting case study in its own right for a number. As T&F have said that they will release comments a bit at a time (thanks for sharing the data right away), I'll be looking at this a bit at a time as well. My next comment is substantive so I'll start a second comment so that it doesn't get lost.
ReplyDeleteHas informa.plc - the multi-national diverse for-profit company behind the scholar-friendly-sounding brands Taylor & Francis and Routledge - become a gatekeeper to conducting social sciences and humanities research? This survey is a case in point. Informa has ready access to a network of tens of thousands of scholars around the world. They control access to this network. This is not complete control, as there would be other ways of conducting this research, but they do have control over a vast network of scholars in the humanities and social sciences.
ReplyDeleteWould informa permit scholars in the humanities and social sciences to conduct research using this network? If so, who decides who gets to do research? If the answer is informa, then (as I suspect is the case) we scholars have handed over control of a major network of our communities to be controlled by corporate interests.
Clarification from informa would be welcome. Note that the company has every right to conduct research, and it is a benefit to all if they do so with the purposes of improving their products. This piece of research, however, looks like it is intended to inform public policy as well. If research to inform public policy is controlled by the corporate sector, that's a major erosion of democracy. This is an important point - freedom for scholarship isn't just about open access.