The Imaginary Journal of Poetic Economics: 2017

Sunday, December 31, 2017

Dramatic Growth of Open Access December 31, 2017

Highlights

As usual the open access movement has much to celebrate as 2017 draws to a close, and the whole world has much to look forward to from open access in 2018. As of today there are 4.6 million articles in PubMedCentral, thanks in large measure to constantly increasing participation by scholarly journals; sometime in 2018 this is likely to exceed 5 million. DOAJ added a net 1,272 journals (3.5 / day) and showed even stronger growth in article searchability; a DOAJ milestone of 3 million searchable articles in likely to come in 2018. The Directory of Open Access Books nearly doubled in size and now has more than 10,000 books from 247 publishers. Bielefeld Academic Search Engine, the best surrogate for overall growth, continues to amaze with over 120 million documents, growth of 17.3 million in 2017, a 17% growth rate on a very substantial base; a 20% growth in content providers is an indication of the overall growth of the repository movement. arXiv's growth rate was 10% while newcomer arXiv clones socRxiv grew by 187% and bioRxiv by 151%. REPEC grew by 13%, SCOAP3 by 32%. Internet Archive grew by 31 billion web pages, 4 million texts, 2.4 million images, 800,000 movies, and 600,000 audio recordings. Following are selected details indicating the content numbers at the end of 2017, 2017 growth by number, percentage, and where warranted, by day.

Full data can be downloaded from here: https://dataverse.scholarsportal.info/dataverse/dgoa

Details (selected)

Totals are from December 31, 2017. Annual growth: Dec. 31, 2017 - Dec. 31, 2017

Free journals

Directory of Open Access Journals

10,727 journals

2017 growth: 1,272 journals (3.5 / day), growth rate 13%

7,809 journals searchable at article level

2017 growth: 1,175 (3.2 / day), growth rate 18%

2,791,701 articles searchable at article level

2017 growth: 391,443 (1,072 / day), growth rate 16%

Milestone to watch for in 2018: 3 million articles searchable at article level

Electronic Journals Library

59,240 journals that can be read free of charge (2017 growth: 3,678 (10 / day), 7% growth)

Free books

OECD ilibrary

11,690 e-book titles (2017 growth 640 (2 / day), growth rate 6%

Directory of Open Access Books

10,315 academic peer-reviewed books, 247 publishers

2017 growth: 4,713 (13 / day), growth rate 84%, increase of 80 publishers

Monday, October 23, 2017

Dramatic Growth of Open Access September 30, 2017

Happy Open Access Week!

In brief: best guesstimate - there are approximately 70 million OA documents today (subset of BASE's 115 million, about 60% OA), with OA documents at BASE growing at a rate of about 1,800 OA documents per day. Where do these come from? Thousands of OA archives - with PubMedCentral the largest by far at 4.5 million articles and active participation by thousands of journals. This quarter by the numbers the DOAJ team set a new record with a net growth of 689 journals of 7.7 titles per day. However, percentage wise the most remarkable quarterly growth was all about archives, with BioRxiv and SocRXiv topping the growth list by percentage, and as usual several sections of Internet Archive well up on the growth list. On an annual basis, Directory of Open Access Books was the fastest growing in terms of both # of books and # of publishers.

To download the raw data, go to the DGOA dataverse.

Detail

Bielefeld Academic Search Engine (BASE), in addition to a great OA search engine, provides the best (if rough) guesstimate of how much we are achieving together, added 2.7 million documents this quarter for a total of 115 million. About 60% of the content in BASE is OA, so this is roughly growth of 160,000 open access items over the past quarter, or about 1,800 documents per day, with a total of about 6.9 million open access documents.

While the growth of open access is always amazing, sometimes it's more evident by the numbers, other times by the percentage. By the numbers: this quarter DOAJ net growth was 689 titles - that's 7.7 titles per day, a record for DOAJ! As of September 30, DOAJ included
10,114 titles. As the chart shows, growth in DOAJ at the searchable article level is particularly remarkable, growing from just over 60,000 in 2004 to close to 2.5 million articles today. Over at PubMedCentral there are now 4.5 million documents with close to 7 thousand journals actively contributing content.

By the percentages, it was a particularly good quarter for open access archives. Newcomers bioRxiv and SocArXiv top the quarterly growth by percentage with growth rates of 25% for bioRxiv (equivalent to doubling in a year) and 22% for SocArXiv (just under doubling in a year). bioRxiv now has 15,000 preprints, SocArXiv close to 1,500. As usual growth at Internet Archive was very impressive, 14% growth in texts (now 14.5 million free texts), 12% growth in the recently added collections category (now close to 300,000 collections) and 9% growth in software (close to 200,000). The RePEC book collection grew by 12% to over 33,000*.

On an annual basis by percentage, Directory of Open Access Books is at the top for growth both in # of books (65% growth, now close to 9,000 titles) and # of publishers (40% growth, 225 publishers). BASE continues to amaze with a 23% increase in content providers over the past year (edging up towards 6,000), and 15% growth in content (now at 115 million documents).

* The RePEC book chapter category also showed amazing growth, but perhaps this is an artefact due to a recent clean-up project as numbers were significantly down last quarter.

This post is part of the Dramatic Growth of Open Access series.

Friday, June 30, 2017

Dramatic Growth of Open Access June 30, 2017

Correction: DOAJ will soon surpass 2.5 million articles, not a quarter of a billion as originally reported.

Highlights

Open access continues to demonstrate robust growth on a global scale, in terms of works that are made available open access, ongoing growth in infrastructure (new repositories, journals, book publishers), strong growth for new initiatives such as SocArxiv, BioRxiv, the Directory of Open Access Books, SCOAP3, as well as ongoing strong growth in established services such as BASE, PubMed / PubMedCentral, Internet Archive (check out the new Collections including a Trump archive and FactChecker), DOAJ (almost 2.5 million articles searchable at the article level), RePEC and arXiv. Ongoing growth in infrastructure and OA policy give every reason to expect this growth to be ongoing.

Open Data Version

Morrison, Heather, 2014, "Dramatic Growth of Open Access", hdl:10864/10660, Scholars Portal Dataverse, V17,

Details

This edition of the Dramatic Growth of Open Access highlights two of the new kids on the OA block - SocArxiv and BioRxiv, modeled on early OA success story arXiv, topping the quarterly growth by percentage with percentage growth of about 30% each! SocArxiv now has 1,200 documents and BioRxiv 12,800.

Similarly, a relative newcomer, the Directory of Open Access Books, is in both first and second place for annual growth by percentage with 68% growth for OA books and 40% of OA publishers in the past year for a total of 8,172 open access books and 217 OA book publishers.

SCOAP3, a global initiative to transform high-energy physics publishing to open access, is showing remarkable growth, 39% in the last year and 8% in the last quarter for a total of 15,790 articles funded.

To celebrate the growth of all OA services two pictures are presented of the growth of the largest collective OA search engine that I am aware of. Together, the 5,000 content providers who contribute metadata to the Bielefeld Academic Search Engine (BASE) have made available over 112 million documents. Around 60% of these are open access, so the number of OA documents in the world can be said to be somewhere about 67 million. BASE also posts their own online statistics table and chart - check it out here.

I wish I had the time to applaud and celebrate the growth of each and every OA service, but with 5,000 services contributing to BASE (and others that don't), if I worked on this 365 days a year I would have to cover 14 initiatives every day. So please feel free to help out by applauding and celebrating the services most relevant to you - the journals in your discipline, your institutional repository, the services you find most helpful to search.

Below you will find tables listing the top services by quarterly (5% or more) and annual growth (10% or more). For the full numbers download the open data version (link above). As usual Internet Archive is well represented, with 5 items in the list of the top 13 services by quarterly growth and the top 18 services by annual growth. Internet Archive also offers 2 intriguing new services under Collections - a Trump Archive with over a thousand videos and a Fact Checker collection with over 400 items, available at https://archive.org/details/tv https://archive.org/details/tv

Of course PubMed and PubMedCentral are up there in the growth charts, in this quarter for total number of items (5% quarterly growth) as well as what looks (to me) like hesitant new steps by a substantial number of journals, with a 26% increase in the number of contributing journals that provide some OA and a 14% increase in the number of journals that provide OA to selected articles. The number of journals providing immediate free access and/or all articles open access continues to increase, so this is clearly growth, not backsliding.

DOAJ is included in the top growth services with 14% growth in the number of articles searchable at article level. DOAJ now has over 2.49 million articles searchable at the article level and should soon surpass 2.5 million articles.

arXiv and RePEC are on the list for strong growth in articles, and ROARMAP for growth in OA policies.

-->

Quarterly growth (percentage)		June 2017
32%	SocArxiv preprints	1,200
29%	BioRxiv all articles	12,280
18%	# of academic peer-reviewed books (DOAB)	8,172
18%	# publishers (DOAB)	217
8%	SCOAP3 articles	15,790
8%	Internet Archive Software	178,635
7%	Video (movies) (Internet Archive)	3,437,542
7%	Texts (Internet Archive)	12,821,051
5%	Images (Internet Archive)	1,476,743
5%	# of content providers (BASE)	5,621
5%	Audio (recordings) (Internet Archive)	3,477,033
5%	Webpages (Internet Archive) (in billions)	298
5%	PubMedCentral (number of items)	4,400,000

-->

Annual growth (percentage)		06/30/17
68%	# of academic peer-reviewed books (DOAB)	8,172
40%	# publishers (DOAB)	217
39%	SCOAP3 number of archives	15,790
34%	Video (movies) (Internet Archive)	3,437,542
33%	Internet Archive: Software	178,635
29%	# of content providers (BASE)	5,621
27%	Texts (Internet Archive)	12,821,051
26%	PMC journals some OA	609
25%	Internet Archive: Images	1,476,743
20%	# of documents (BASE)	112,458,360
17%	Audio (recordings) (Internet Archive)	3,477,033
17%	RePEc journal articles	1,491,037
14%	# of articles searchable at article level (DOAJ)	2,493,835
14%	PMC select deposit journals	4,296
13%	RePEC downloadable	2,143,844
13%	Total Policies (ROARMAP)	872
13%	PMC # items	4,400,000
10%	arXiv http://arxiv.org/	1,278,739

This post is part of the Dramatic Growth of Open Access Series

Feel free to copy and share - with love. Note that images are compressed by the software to reduce file size, and they are also quickly outdated. You are welcome to use the images, but my recommendation is to download the data and make your own graphics. It's easier than you think with tools like modern spreadsheet software.

Wednesday, June 28, 2017

Critical Data Literacy, why and how: an Open Education Resource (OER)

This OER was developed for presentation at the Data Power 2017 conference held at Carleton University, Ottawa, Ontario June 22 - 23. This is primarily a framework for how to go about teaching critical data literacy in the student-centered tradition of Freire, supplemented by the work of Tygel and colleagues. A sample introduction developed for Canadian university students, and a few references, are included. My definition of critical data literacy as used in this OER is:

critical data literacy is the ability to understand and critique how the beliefs and values of people and groups (including government) influence what data is created, how it is shared and how it used by to tell compelling stories by storytellers whose beliefs and values shape the kind of stories they choose to tell and how they tell the stories. Critical data literacy also means having the ability to create and tell one's own stories using data.

This OER is released under the terms of copy and share - with love, my latest statement on sharing which can be found at the bottom of this post. The Freire tradition of popular education involves starting with the lived experience of students. In this context, following is what I recommend for anyone who wishes to develop a full critical data literacy program based on the framework. I think that this framework could be adapated for teaching at any level, from community-based learning (led by community groups or organizers or as a participatory action research project) to graduate classes (that's where I teach). Some of the details would change. For example, if you are teaching at a university, some parts of the process are likely to involve formal evaluation (marking), but if you are teaching to the general public or a community group, this would not make sense. Please adjust as needed for your own context.

The overall approach:

Identify your student group. Think about what kinds of issues or problems they might have that could potentially be helped by data, the kind of data stories they might be familiar with.
Develop an introduction to critical data literacy. Tygel and colleagues (2015, 2016) found that this was necessary. One way to think about the difference between critical data literacy and basic literacy (reading) is that people who do not know how to read in recent history are likely to be aware of the existence of reading as something that other people do. Data literacy / critical data literacy is not at this point in time as broadly understood as reading.
Plan the 3 phases of the framework that follow directly from the Freire tradition: investigation, thematisation, and problematisation. In these phases, students should lead the learning process (active learning), pursuing problems and questions of their own devising. The teacher's role is to provide support.
Plan a systematisation (synthesis) wrap-up approach that makes sense for your student group. In some cases this might be left for the students to decide the approach, and the teacher only helps to guide the students towards this closure. In a formal educational setting, this might involve a pre-determined assignment.
Implement!

The 5 phases are: introduction, investigation, thematisation, problematisation, and systematization (synthesis). Details follow. The introduction section is the most fully developed as this is the only teaching portion that involves imparting knowledge; all others begin with the student.

Introduction

As noted above, it will not be obvious to everyone what data literacy or critical data literacy is or why they should learn about it, as discovered by Tygel and colleagues (2015, 2016). For this reason, an introduction to the topic may be helpful. In this phase one might invite in guest speakers from the community who use data in their storytelling and/or to provide examples of data storytelling. This is also where definitions of critical data literacy could be introduced. In addition to my definition (see above), I like this definition of data literacy from the Data Journalism Handbook because it includes the element of critical thinking; not every definition that I have seen includes this, to me a significant omission.

data literacy is the ability to consume for knowledge, produce coherently and think critically about data [emphasis added] (Grey, Bounearu & Chambers (2012)

Following is a sample introduction developed for an audience of Canadian university students. If you are teaching a different type of student group, I recommend that you develop your own introduction tailored to your group. If you do and you are willing to share this with others, please send me a link (via e-mail to Heather dot Morrison at uottawa dot ca) or as a comment to this post and I will include a link to your work in this post. If you would like to use this introduction as is, please see the link to the full presentation.

Introduction slide 1

This slide presents two conflicting stories that are told using basically the same underlying data. One of these (tax freedom day) will be very familiar to the audience, while the other will not as it is relatively new.

This slide illustrates two very different perspectives on taxation in Canada. On the left, we see the Fraser Institute’s Tax Freedom Day. The Fraser Institute, a right-wing think tank, uses data to tell their story of over-taxed Canadians, working more than half the year for the government before earning a dime for themselves. The idea of tax freedom day has been very effective in Canada over the past few decades. On the right, we see one of the images from the Broadbent Institute’s report The Brass Tax which was published very recently. The left-wing Broadbent Institute challenges the numbers behind the Fraser Institute’s analysis, argues that Canadian taxation is pretty reasonable compared to other countries, and presents a different picture. In this case this graph illustrates Canada’s progressive approach to taxation and makes the point that people with little to no income pay no income tax and only a small percentage of Canadians age 25 to 54 are in the top income tax bracket, paying more than 30% of income in taxes. These are 2 groups of people with a different vision of what society should be like, using the same underlying data to tell 2 very different stories. If we go directly to the data source, will this eliminate the impact of the storyteller? Let’s see.

The following two slides might be more effective as a live demo or in-class lab activity.

One of the underlying datasets used by both groups is the statistics provided by OECD. If you go to the OECD website there are some neat online tools that let us quickly visualize data in different ways. One of the elements of the data story told by the Fraser Institute is that individual families pay too much in taxes. I wondered if there has been any change in the portion of tax revenue contributed through personal and corporate taxes over the years. Here is what I found using the OECD website. It seems that more tax is gathered from personal rather than corporate taxes, but over the past few years the portions don’t seem to have changed much. This is the default view that shows trends from 2000 – 2015. If this had fit what I already believed, I suspect I would have stopped here. But I seem to recall a relative decrease in corporate taxation over the past few decades so I decided to slide the years covered…

And this is what I found. If we slide the start date of the visualization tool back to 1965, it does appear that there has been a relative increase in tax revenue from personal sources and a relative decrease in tax revenue from corporate sources. This shows how easy it would be for two people with different perspectives on what a data trend is likely to be to go to exactly the same dataset and make a slight change to how the data is visualized to tell two very different stories.

Kaulfuss uses OECD data to tell a story about U.S. health care spending on a blog called Beyond Economics. The story is that the U.S. spends two and a half times the OECD average on health. It doesn’t surprise me that the U.S. spends more than the OECD average on health, but I am surprised that the difference is this much. What I found even more intriguing is the author’s claim that U.S. public spending on health is above the OECD average. Who knew? Disclaimer: what I am doing here is presenting stories told through data, I have not examined the data itself so cannot comment on the accuracy of the story.

Wikipedia has a section called Health Care in Canada. Here in Canada many of us – I include myself – think highly of our public health care system, and I think I see this perspective here. This section states that “most health statistics in Canada are at or above the G8 average” in a paragraph that is followed by the table pictured above. The table draws from a number of data sources and appears to me to demonstrate above-average data literacy skills. However…

When you look at the statistics that are presented and calculate the averages, Canada is above average on 3 of 8 measures. This is not “most”. This suggests a need for data literacy. If you look at the specific measures where we are above average, an argument can be made that being above average in life expectancy is a good thing. However, an above-average infant mortality rate is probably not such a good thing. We are also slightly above average on % of government revenue spent on health, but what does this mean and is it a good thing? Looking at some of the areas where we are below average –such as the # of doctors & nurses per population & % of health costs paid by government – might give one reason to re-consider our narrative that we Canadians are above average in public health. This illustrates a need for critical data literacy. In other words, our beliefs might be getting in the way of understanding what is our existing data tells us.

Some approaches and suggestions for creating a meaningful introduction

The reason for the introduction section is because as Tygel and colleagues found there is a need to start with some explanation about what data is and how people use it. There are many potential approaches to introducing the topic such as having guest speakers come to explain how they make use of data and data visualization.

Suggested sample activity

One activity that would fit here is to have students create their own demonstrations. In the case of tax data, students could do a google search for tax data and limit to images. This search will yield lots of material to work on. The idea is to have students find out who created the visualization and what the story behind the visualization is. If this is done for evaluation purposes, I recommend a pass/fail approach because student success will depend a lot on which images are selected. Being there to hear the findings of all the students is sufficient for this learning exercise. A teacher in an area where computers are not readily available could bring in copies of materials to work with. This introductory phase may be more relevant for some student groups than others, for example university students. If this doesn’t seem to fit, you could skip this stage.

Investigation, Thematisation & Problematisation

Two key points to keep in mind in these 3 phases: 1) the core focus should be lived experience not imparting abstract knowledge and 2) teaching involves helping people seek and find answers. This is important because in teaching data literacy one might be tempted by starting with the data, teaching people how to understand and work with data. Keynote speaker Gwen Phillips (and BC First Nations data activist) at the Data Power 2017 provided a brilliant example of why not to start with the data: the existing data might not be what is wanted at all. As Gwen said, we should measure what do want (e.g. youth vitality) not just what we don't want (e.g. teen suicide). This introduces a challenge to develop new metrics, but one that seems worthy of pursuit. If we start by teaching about existing data we risk missing the opportunity to identify gaps like this.

Disclosure: in understanding the following 3 phases, it may be helpful to know that although I teach at a university and am very engaged in pedagogy, I do not have an education degree and do not consider myself an expert on pedagogy. If you would like to know more about how to teach in the Freire tradition, I suggest starting with the Tygel references below and if desired supplementing with general educational books and articles covering the Freire tradition. My contributions below are limited to providing a very quick introduction and making the connection with critical data literacy.

Investigation

The investigation phase is the first of 3 phases that follow the Freire tradition. The idea is to begin with lived experience, with real-world problems. If this approach is used for self-teaching by community groups independently or with an academic consultant as a participatory action research project, this is closest to the classic Freire scenario and the best example of a pure investigation stage. To modify this for an education setting, students could either choose problems or issues of direct interest to them, for example student debt, or they might brainstorm a particular target group whose problems they are familiar with such as First Nations, a salient issue here in Canada as many of us struggle to implement the recommendations of our Truth & Reconciliation commission. Classroom activities could include a brainstorm session, individual or small group reflection, and/or presentation of the results of the investigation stage.

Thematisation

Thematisation is the first analytic stage. Before searching for what data is available, the idea is to focus on the real-world issue and figure out what kind of data might help to understand or resolve the issue. Examples based on today’s case studies on taxation and health spending could include learning what sorts of taxes are collected and by which governments, or comparing public collective health spending with individual spending.

Problematisation

After thematisation, with some back-and-forth, comes problematisation. This is where we get into research on what kinds of data actually exist that is relevant to the problem, who collects the data and why. Some examples of the types of data sources students might look into at this point if they choose to focus on taxation and spending:

Canada Revenue Agency
OECD
Federation and provincial budgets
Academic Research
NGO / Think Tank research (e.g. Fraser Institute and Broadbent Institute)

One question that might be raised is whether the existing data is actually sufficient or not, that is, the scope of the inquiry is not focused just on understanding what data is available. but rather what is needed to understand and resolve the problem of interest.

Systematization

Finally, in the systematization stage we put what we have together to come up with an action plan. The nature of the action plan might vary quite a bit depending on the students. An activist community group might want to develop an action campaign or an infographic or other data story to facilitate an existing action campaign. One approach to action could involve citizen data collection. In a graduate class on information policy, like the classes that I teach at the University of Ottawa's School of Information Studies, developing a policy briefing and recommendations for evaluation as academic work might make sense.

References

Fraser Institute (n.d.). Tax freedom day calculator. Retrieved June 9, 2017 from https://www.fraserinstitute.org/tax-freedom-day-calculator

Grey, J., Bounegru, L., & Chambers, L. (2012). Data Journalism Handbook. OKFN. (as cited in Tygel & Kirsch 2016)

Kaulfuss, R. (2017). Health care: human right or expensive entitlement? Beyond economics. Retrieved June 15, 2017 from https://beyondeconomics.org/2017/03/15/health-care-human-right-or-expensive-entitlement/

OECD (2017), Tax revenue (indicator). doi: 10.1787/d98b8cf5-en (Accessed on 15 June 2017)

Shillington, R. & Shaban, R. (2017). The brass tax: busting myths about overtaxed Canadians. Ottawa: Broadbent Institute. Retrieved June 9, 2017 from http://www.broadbentinstitute.ca/the_brass_tax

Tygel, A.; Campos, M.; De Alvear, C. (2015). Teaching open data for social movements: a research strategy. The Journal of Community Informatics 11:3. Retrieved June 19, 2017 from http://ci-journal.net/index.php/ciej/article/view/1220/1165

Tygel, A.; Kirsch, R. (2016). Contributions of Paulo Freire for a critical data literacy: a popular education approach. The Journal of Community Informatics 12:3 pp. 108 – 121. Retrieved June 19, 2017 from http://ci-journal.net/index.php/ciej/article/view/1296.

Wikipedia (n.d.). Healthcare in Canada. Retrieved June 15, 2017 from https://en.wikipedia.org/wiki/Healthcare_in_Canada

Terms: Please copy and share with love.

What does this mean? In brief, I have no interest in using intellectual property law to prevent anyone from using or re-using my work with intentions such as furthering the collective knowledge of humanity (truth with justice and compassion), protecting or restoring the environment or making the conditions of life of humanity better. That is what I mean by with love. If your motives in using my work are something other than love, such as making a profit for yourself or a corporation that you work for, subverting truth, justice, or compassion, then note that I reserve all rights under copyright. Please use attribution as appropriate. For example, if you use my work in an academic or journalist context, you need to acknowledge me as author in order to avoid plagiarism (and confusion).

This post is part of the Creative Globalization series.

Tuesday, March 28, 2017

Novel processes and metrics for a scientific evaluation: preliminary reflections

Reflections on Michaël Bon, Michael Taylor, Gary S. McDowell. “Novel processes and metrics for a scientific evaluation rooted in the principles of science - Version 1”. SJS (26 Jan. 2017)
<http://www.sjscience.org/article?id=580>

Following are my initial reflections on what I would describe as a ground-breaking effort toward articulating a radically transformation of scholarly communication, a transformation that I regard as much needed and highly timely as the current system is optimized for the technology of the 17th century (printing press and postal system) and is far from taking full advantage of the potential of the internet.

The basic idea described by the authors is to replace the existing practices of evaluation of scholarly work with a more collaborative and open system they call the Self-Journals of Science .

Comments

The title Self-journals of science: I recommend coming up with a new name. The name is likely to give the impression of vanity publishing, even though this is not what the authors are suggesting, which appears to be more along the lines of a new form of collaborative organization of peer review.

Section 1 Introduction: the inherent shortcomings of an asymetric evaluation system appears to attempt to describe how scientific communication works, its purpose, and critique, with citations, in just a few pages. This is sufficient to tell the reader where the authors are coming from, but too broad in scope to have much depth or accuracy. I am not sure that it makes sense to spend a lot of time further developing this section. For example, the second paragraph refers to scientific recognition as artificially turned into a resource of predetermined scarcity. I am pretty sure that further research could easily yield evidence to back up this statement - e.g. Garfield's idea of the "core journals" to eliminate the journals one needn't bother buying or reading, and the apparently de facto assumption that a good journal is one with a high rejection rate. On page 3, first paragraph, 4 citations are given for one statement. A quick glance at the reference list suggests that this may be stretching what the authors of the cited works have said. For example, at face value it seems unlike that reference 4 with a title of "Double-blind review favours increased representation of female authors" actually supports the author's assertion that "Since peer-trials necessarily involve a degree of confidentiality and secrecy..many errors, biases and conflicts of interest may arise without the possibility of correction". It seems that the authors of the cited article are making exactly the opposite argument, arguing that semi-open review results in bias. If I was doing a thorough review, I would look up at least a few of the cited works and if the arguments cited are not justified in the cited works I hand the work of reading the works cited and citing appropriately back to the authors.

The arguments presented are provocative and appropriate for initiating an important scholarly discussion. Like any provocative work, the arguments may be relatively stronger for the task of initiating needed discussion but somewhat weak due to lack of counter-argument. For example, the point of Section 1.4 is that "scientific conservatism is placing a brake on the pace of change". Whether anything is placing a brake on the pace of change in 2017 is, I believe, arguable. However, the authors also do not address the benefits of scientific conservatism here, although the arguments made elsewhere e.g. "The validity of an article is established by a process of open and objective debate by the whole community" are arguments for scientific conservatism (or so I argue). The potential benefits of scientific conservatism are not addressed. For example, one needs to understand this tendency of science to fully appreciate the current consensus on climate change.

Section 2 defines scientific value as validity and importance

There are some interesting ideas here, however the authors conflate methodological soundness with validity. A research study can reflect the very best practices in today's methodology and present logical conclusions based on existing knowledge while still being incorrect or invalid (lacking external validity) for such reasons as limitations on our collective knowledge. A logical argument based on a premise incorrectly perceived to to be true can lead to logical but incorrect conclusions.

The authors state that "the validity of an article is established by a process of open and objective debate by the whole community". This is one instance of what I see as overstatement of both current and potential future practice. Only in a very small scholarly community would it be possible for every member of the community to read every article, never mind have an open and objective debate about each article. I think the authors have a valid point here, but direct this at the wrong level. This kind of debate occurs with the big picture paradigmatic issues such as climate change, not at the level of the individual article.

Perceived importance of an article is given along with validity as the other measure for evaluation of an article. This argument needs work and critique. I agree with the author (and Kuhn) about the tendency towards scientific conservatism, and I think we should be aware of bias in any new system, especially one based on open review. People are likely to perceive articles as more important if they advance work that falls within an existing paradigm or a new one that is gaining traction than truly pioneering work. With open review, I expect that authors with existing high status are more likely to be perceived to be writing important work while new, unknown, female authors or those from minority groups are more likely to have their work perceived as unimportant.

I do not wish to dismiss the idea of importance, rather I would like to suggest that this needs quite a bit of work to define appropriately. For example, if I understand correctly replication of previous experiments is perceived as a lesser contribution than original work. This is a disincentive to replication that seems likely to increase the likelihood of perpetuating error. Assuming this is correct, and we wish to change the situation, what is needed is something like a consensus that replication should be more highly valued, otherwise if we rely on perceived importance this work is likely to continue to be undervalued.

Section 2.2 Assessing validity by open peer review

This section presents some very specific suggestions for a review system. One comment that I have is that this approach reflects a particular style. The idea of embedded reviews likely appeals more to some people than to others. Journals often provide reviewers with a variety of options depending on their preferred style; a written review like this, or go through the article and track changes. The + / - vote system for reviews strikes me as a tool very likely to reflect the personal popularity of reviewers and/or particular viewpoints rather than adding to the quality of reviews. There are advantages and disadvantages to authors being able to select the reviews that they feel are of most value to them. The disadvantage is that authors with a blind spot or conscious bias are free to ignore reviews that a really good editor would force them to address before a work could be published.

Section 3 Benefits of this evaluation system

Here the authors argue that this evaluation system can be transformed into metrics for the purpose of evaluation (number of scholars engaged in peer review, fraction that consider the article is up to scientific standards) and for importance (the number of authors that have curated the article in their self-journal). Like the authors, I think we need to move away from publishing in high impact factor journals as a surrogate of quality. However, I argue against metrics-based evaluation, period. This is a topic that I will be writing about more over the coming months. For now, suffice it to say that quickly moving to new metrics-based evaluation systems appears to me likely to create worse problems than such a move is meant to solve. For example, if we assume that scientific conservatism is a thing and is a problem, isn't a system where people are evaluated based on the number of people who review one's work and find it up to standards likely to increase conservatism?

Some strengths of the article:

recognizing the need for change and hopefully kick-starting an important discussion
starting with the idea that we scholars can lead the transformation ourselves
focus on collaboration rather than competition

To think about from my perspective:

researcher time: realism is needed. An article that is reviewed by two or three people who are qualified to judge soundness of method, logic of arguments and clarity of writing should be enough. It isn't a good use of the time of researchers to have a whole lot of people looking at whether a particular statistical test was appropriate or not.
this is work for scholarly communities, not individuals. The conclusion speaks to the experience of arXiv. arXiv is a service shared by a large community and supported by a partnership of libraries that has staff and hosting support.
the Self-Journals of Science uses the CC-BY license as a default. Many OA advocates regard this license as the best option for OA, however I argue that this is a major strategic error for the OA movement. My arguments on the overlap between open licensing and open access are complex and can be found in my series Creative Commons and Open Access Critique. To me this is a central issue that the OA movement must deal with, and so I raise it here and continue to avoid participating in services that require me to use this license for my work.

Key take-aways I hope people will get out of this review:

forget metrics - don't come up with a replacement for impact factor, let's get out of metrics-based evaluation altogether
look for good models, like arXiv because communities are complicated. What works?
let's talk - some of us may want immediate radical transformation of scholarly communication, but doing this well is going to take some time, to figure out the issues, come up with potential solutions, let people try stuff and see what works and what doesn't, and research too
be realistic about time and style - researchers have limited time, and people have different preferred styles. New approaches need to take this into account.

For more on this topic, watch for my keynote at the What do rankings tell us about higher education? roundtable at UBC this May.

The Imaginary Journal of Poetic Economics

Sunday, December 31, 2017

Dramatic Growth of Open Access December 31, 2017

Monday, October 23, 2017

Dramatic Growth of Open Access September 30, 2017

Friday, June 30, 2017

Dramatic Growth of Open Access June 30, 2017

Wednesday, June 28, 2017

Critical Data Literacy, why and how: an Open Education Resource (OER)

Tuesday, March 28, 2017

Novel processes and metrics for a scientific evaluation: preliminary reflections

Series

Blog Archive

Terms

OA conference in Tunisia

About IJPE (includes commenting policy)

Total Pageviews

Labels

Education is a Public Good - not a commercial activity!

Haiti basic health info

Sitemeter