Could the reddit model replace the current scientific publication system?

Just today, Subhajit Ganguly put up a blog post that piqued my interest: “Imminent Changes In The Publication Process In Sciences.” In this blog post, Subhajit repeats the battle cry we so often hear from the Open Science movement: traditional for-profit journals have outgrown their usefulness, they benefit only a few individuals, and it’s time to abandon traditional journals and move to a completely Open Science model, such as posting all of your scientific output on arXiv or figshare for everyone to see and critique.

The benefits of the Open Science model

There are clear benefits to openly posting all of your scientific output instead of relying on for-profit journals:

1) We open the possibility of receiving feedback and criticism from everyone in the scientific community, and not just a select few who were chosen to review the article. Presumably this will lead to a more fair treatment of everyone’s scientific output, since the rating of a manuscript will be based on the opinion of hundreds (or even thousands) of people with a broad range of viewpoints and backgrounds.

2) We cut out the middleman (the for-profit publisher). We won’t have to pay someone to disseminate our manuscripts any more. Most scientists find and retrieve other scientist’s manuscripts online nowadays. Why are we paying someone thousands of dollars to host our manuscript(s) on a web server when we can host them on figshare/arXiv for free? Then we can freely promote our article through standard means: conferences, press releases, blog posts, Twitter, etc.

3) We increase the visibility of our scientific output. There has been a big uproar in the Open Science community claiming that “hiding your scientific output behind a paywall is immoral.” Posting your manuscripts on figshare/arXiv immediately alleviates this concern. Additionally, anyone can access your manuscript(s) at any point in time since they’re just a click away on figshare/arXiv, which directly translates into increased potential visibility of your scientific output.

I’m sure there are many more benefits, but those are the major ones that stick out to me.

Let’s be real

My biggest criticism of moving to the model that Subhajit et al. argue for is about benefit #1. It assumes that we have hundreds (or even thousands) of scientists chomping at the bit to thoughtfully critique everyone’s scientific output. Realistically, journals already have enough trouble finding just three scientists to thoughtfully review each manuscript, and that’s with the added benefit that the reviewers can claim they performed a service to the journal and scientific community on their CV. If we take that benefit away, will we really average three or more reviewers per manuscript?

Beyond the above, is it really practical to rely on the standard means of dissemination to get your article noticed? Publishing a manuscript in high-profile journals is a reliable way to ensure that your manuscript is seen by a good portion of the scientific community. If we take away the journals, what do we have left to make your manuscript stand out among the rest?

We have some popular blogs, such as Haldane’s Sieve, that highlight quality manuscripts and give them the attention they deserve. But what if the people running those blogs don’t think your manuscript is good enough to highlight in their blog? What about the thousands of manuscripts that would fall into that category? We run into the exact same problem as we have with for-profit journals: when only a few people are in charge of deciding which manuscripts are “highlight-worthy,” not all manuscripts are going to be given a fair rating.

Could the reddit model replace the current scientific publication system?

For those of you who are unfamiliar with reddit, reddit is one of the most-visited social news web sites in the world. reddit has been tremendously influential on the web as the self-proclaimed “front page of the Internet.”

The reddit model works as follows. Users who sign up on reddit are given four abilities:

  1. submit links
  2. comment on links and other comments
  3. upvote links and comments they think are constructive, and
  4. downvote links and comments they think are unconstructive.

Links and comments that receive a higher score (score = upvotes minus downvotes) are ranked higher in the queue when people view the list of links on reddit. Typically high-scoring links are only ranked high for 24 hours after they are posted, after which time they rapidly decline in the queue.

Top 5 posts on /r/science on March 23, 2013

Top 5 posts on /r/science on March 23, 2013

Shown above, reddit already has an active scientific community in /r/science. There, users submit links to scientific articles, upvote or downvote them depending on their quality and relevance, and discuss their methods and implications in the comments. Seeing this community got me thinking: could the entire scientific community share research with each other this way?

With this model, there’s no more need for formal peer review. Instead, peer review is done by upvoting or downvoting the link and discussing the manuscript in the comments. The constructive comments would be upvoted to the top, while the unconstructive comments fall to the bottom. The entire process is transparent to anyone visiting the web site, and the discussion is archived if anyone ever wants to find that comment that Dr. So-and-so made about a manuscript four years ago.

A few subreddits that you can subscribe to on reddit

A few subreddits that you can subscribe to on reddit

There would be separate “subreddits,” or sub-forums for each field of research that you could optionally subscribe to. If you’re only interested in sequence alignment algorithms, there could be a subreddit dedicated specifically to manuscripts detailing the latest sequence alignment algorithms. Similarly, if you have broader interests such as Evolutionary Biology in general, there could be a subreddit dedicated to discussing the major breakthrough manuscripts in Evolutionary Biology. The research community would be able to create and abandon these subreddits as they wish, allowing the system to reflect the interests of the research community as a whole.

The citation system would work as usual, except the journal would be replaced by the preprint manuscript hosting service you use. Instead of reporting on your CV that you published two papers in such-and-such high-profile journal, you would report the Sciddit (“Scientific reddit”) score your manuscript received.

The reddit model seems like a much more scalable model for the widespread dissemination, peer review, and discussion of scientific output online. The only thing left is to give it a try. Will someone step up to the challenge?

Update (03/25/2013): It looks like someone already created a reddit interface for arXiv. It’s a little buggy, but this is a great first step toward making this idea a reality! http://arxaliv.org/

Randy is a PhD candidate in Michigan State University's Computer Science program. As a member of Dr. Chris Adami's research lab, he studies biologically-inspired artificial intelligence and evolutionary processes.

Posted in open science, philosophy, reddit Tagged with: , , ,
  • http://depth-first.com Rich Apodaca

    This reminds me of a publishing model suggested by Yann LeCun:

    http://yann.lecun.com/ex/pamphlets/publishing-models.html

  • Charlie Barlie

    An interesting question is whether to try to insist that comments be signed (with real names.) I think it would be best to allow both. Encourage signed comments, and provide a RealName feature like Amazon reviews do, but do not discourage anonymous ones. If people want to do a lot of posting with a consistent pseudonym, that’s fine, and they can accumulate positive reputation for the pseudonym. A stray comment by Anon is fine too, but some readers may want to set their options not to display comments by unrated anonymous people.

    The substantive issue is: there ARE good reasons for anonymity: eg junior people may see a big flaw in work by someone very prominent in their field and fears the ire of that person. If you insist they sign their names, they just won’t post. But there may be plenty of people who are happy to speak their minds and accumulate both positive and negative reputational consequences–so why not let them?

    • http://www.randalolson.com Randy Olson

      Very good point! I think the reddit model handles this exceptionally well: by default, everyone is anonymous. They have the option to verify an email, which then shows a tag on their profile that their email is verified. This is just a few lines of code away from making that verified email option and making it a “verified identity” option using .edu email addresses.

  • http://twitter.com/eperlste Ethan Perlstein (@eperlste)

    Nice post! I like the idea of self publishing on one’s lab website/blog and then soliciting peer review on reddit, on the content page itself, or anywhere on the Internet. The challenge, obviously, is capturing and archiving all that fragmented Midrash. But the upvote/downvote model is attractive. My only concern is would a critical mass of academics change their behavior in spite of the Tenure Games.

    • http://www.randalolson.com Randy Olson

      But the upvote/downvote model is attractive. My only concern is would a critical mass of academics change their behavior in spite of the Tenure Games.

      Totally agree that that is going to be the hardest part. There needs to be a serious incentive system in place to get people to even try out the new model. Are the incentives inherently there in the model (e.g., fairer peer review, increased manuscript visibility)? Or do more incentives need to be offered (i.e., something you can be proud to put on your CV)?

  • Chris Fong

    I agree only if including a TLDR is mandatory :D

  • http://twitter.com/mfenner Martin Fenner (@mfenner)

    I like the idea, but don’t think the Reddit model alone will work. Two reasons: a) it is too different from what researchers do now b) The Reddit model is all about short-term attention, not long-term impact.
    Disclaimer: I work in the PLOS Article-Level Metrics project, and Reddit is on the list of sources we want to add.

  • http://egtheory.wordpress.com Artem Kaznatcheev

    Hey Randy!

    I actually came across your post via reddit. However, in the wee-hours of the morning I didn’t realize that we met at Swarmfest last year (to jog your memory, I presented on evolution of ethnocentrism). To reiterate (and expand on) what I said on reddit:

    This is already done, and the reddit aspect is not really having much of an impact. Many of the categories on the ArXiv have their own subreddits. For instance, I follow quant-ph. If you look there, voting barely exists and there are no comments. From my personal experience (I was at two leading quantum research centres — QIC and CQT — for a while and this is based on my assessment of culture there), the subreddit is having absolutely no impact on quantum information/computing — one of the most frequent topics on quant-ph.

    It takes a lot of effort to read and evaluate papers, it isn’t as simple as just up voting your favourite gif or lolcat. I see very little benefit to be gained from a reddit approach, even if you could convince scientists to participate.

    Should we even aim for convincing scientists to take on reddit-like approach? I fear not. Scientists are already being forced to defend their reason for having grants from critiques of uninformed and unqualified pundits that read the title of the proposal only. If evaluating research based on the title and a short snippet alone as is the case on reddit. For example, the reason I didn’t realize it was you that wrote the post when I commented on reddit this morning was because I didn’t actually read it, I was responding to the title, alone.And, I am probably not the shortest-attention-span redditor in the world.

    If this system was popularized, then it would force even more “over-selling” or research, which overtime would lead to (a) a decrease in interest and funding of basic (as opposed to applied) research, (b) reinforcement of popular misconceptions (look at all the neurobullocks articles shared on reddit), (c) exacerbation of over-publishing and in the inequality of attention and citations (as measured by Sciddit) and (d) eventual loss-of-confidence in science by the public as the scientists repeatedly.

    I agree with you that it is important to move away from the closed system of publishing we have right now. However, I really don’t think reddit is the best way. I am personally a fan of advocating for people to host research blogs to replace the 3rd and 4th tiers of journals that currently publish so much unread work with weak reviewing standards. Why keep incremental results as pdfs of arXiv when you can have more accessible images, multimedia, and discussion on WordPress?

    • http://www.randalolson.com Randy Olson

      Good to hear from you again!

      I think you raise some good concerns that we would definitely need to keep in mind when designing “sciddit.” However, I’m not entirely convinced that the reddit model is designed to select for content appealing to the lowest common denominator. If we take /r/science as an example, with proper rules and moderation in place it is entirely possible to have a community dedicated to thoughtful discussion of the latest research.

      Should we even aim for convincing scientists to take on reddit-like approach? I fear not. Scientists are already being forced to defend their reason for having grants from critiques of uninformed and unqualified pundits that read the title of the proposal only. If evaluating research based on the title and a short snippet alone as is the case on reddit. For example, the reason I didn’t realize it was you that wrote the post when I commented on reddit this morning was because I didn’t actually read it, I was responding to the title, alone.And, I am probably not the shortest-attention-span redditor in the world.

      That brings up an important point. If scientists are funded by the public, should they not have to convince the people funding them that their research is worthwhile? Obviously a NSF/NIH/etc. committee of scientists agreed that their research is worth funding with public money, so it shouldn’t be terribly difficult to come up with a reasonable case for the public to fund your research. I think it is very important for science to be more connected–and more understandable–to the public. The more the public understands science and why we do it, the better off we are as a society.

      I am personally a fan of advocating for people to host research blogs to replace the 3rd and 4th tiers of journals that currently publish so much unread work with weak reviewing standards. Why keep incremental results as pdfs of arXiv when you can have more accessible images, multimedia, and discussion on WordPress?

      I agree with you in a sense: science should be less publication-focused and more progress-focused. I think that having a lengthy preparation, review, and publication process (sometimes taking over a year to get a manuscript from finished to published) slows down scientific progress. Rather, we should encourage scientists to share their findings as soon as they feel they’re ready to be publicly consumed. If we did so via a reddit model, everyone could immediately read the results, evaluate them, and discuss them in the comments. I think having regular small research updates is far more consumable than trying to comprehend a year or more of work crammed into a single 8-page manuscript.

  • http://www.moneyscience.com Jacob

    It’s feasible that a sophisticated model of social capital could go some way to replicating some salient features of peer review, but the current Reddit model comes nowhere close.

    Reddit’s model is built for simplicity, easy of use and to accomplish a very specific task; namely to raise the profile of content which is judged by a sufficiently large group of people to be noteworthy.

    The purpose of Peer Review is clearly somewhat more nuanced; to assess the accuracy or scientific value of a specific piece of research.

    Online comments from a suitably qualified subset of users could conceivably replace the traditional reviews provided by a referees, and voting on those comments could provide a crude metric of their value, but such a system is patently open to abuse. Online commenting generally is something of a minority pursuit and fraught with questions, indeed, research recently showed that online comments tend to hurt the understanding of science (http://bit.ly/YtpZTg).

    In order to provide a reliable model for which participants would be both happy to assess and be assessed, you need a transparent means by which they can assess each other’s value – so it’s a chicken and egg problem.

    ‘The wisdom of crowds’ argument does not hold unless the crowd has sufficient confidence that the results of their activity will provide good results, and not simply results from a self-selecting group of engaged participants.

    Traditional peer review is essentially a heuristic for optimising an assessment process of experts. It is not perfect, but unless a model of social review can impose reliable constraints on the length of the process, and create a consensus of confidence in the results, then it will not be able to compete.

    • http://www.randalolson.com Randy Olson

      Great comments! All of the discussions I’ve been having with people about this topic lately have gotten me thinking: maybe we need to start doing some rigorous analyses of the reddit model. Could we build a simplified version of the “sciddit” model and see how the community evolves? How people act in certain situations (such as those you mention above)? How prone the system is to abuse? I’d imagine such a study could be done for only a couple hundred dollars using Amazon Mechanical Turk.

  • http://jasonpriem.org Jason Priem

    If you haven’t already, check out http://arxaliv.org/, a Reddit clone that pulls in new ArXiv articles as they’re published.

    I like the idea (as both a researcher and an avid redditor), but would post two cautions:

    1. This one’s obvious: it’s tough to get scholars to donate their intellectual labor unless they’re getting something back. You’re suggesting scholars could be paid in “karma” (to put it in Reddit terms), and I agree. BUT until the karma-recognizing community is big enough, the exchange rate for that currency is going to be pretty darn low…you’re paying in low-value scrip. To improve the value of that scrip, you need a bigger, more important community…and then you’ve got the classic chicken-and-egg problem of online communities.

    This isn’t insoluble. Reddit’s founders solved it be operating dozens of sockpuppet accounts apiece, all posting interesting content, to bootstrap the karma economy. More relevantly for scholarship, MathOverflow solved it by using personal connections to recruit top mathematicians to the community from the get-go. But as the long and growing list of somnolent online-peer-review-community-things evinces, these solutions are not easy.

    2. Even if making wild success for this kind of system were easy, I’m not convinced it’d be desirable. Do we really want to go from 25,000 journals to 10 versions of SciRedditThing.com(tm), controlled by the same oligarchy of powerful publishers and complicit tenure committees? I think no. I think we want more diversity, more publications outlets, more conversations in more places. We want smaller pieces, looslier joined.

    I think the scholcomm future will probably have some Reddit-lookin’ things in it, and I’m glad for that. But expect that rather than anchoring the system, instead these Redditesques will be yet another input in the Research-Wide Web of aggregated interactions: reads, discussions, saves, cites, and recommendations on Twitter, blogs, reference managers, social networks, and more. Instead of building systems that ask scholars to come play in our backyards, we’ll be listening to the conversations they’re already having in their own places, on their own scholarly networks, wherever those may be. Cameron Neylon argued this well, back in 2010. I think the growth in the altmetrics movement since then offers a chance to make this actually happen in the next few years.

    • http://www.randalolson.com Randy Olson

      http://arxaliv.org/ looks awesome! It looks almost exactly like I was talking about in this post. It just needs people to actually use it…

      +1 about point #1. As mentioned in other comments here, it would definitely take a lot to 1) get a critical mass of the scientific community using the system and 2) make the “internet points” actually valuable to the scientific community. I think that’s a non-obvious problem to overcome, and you mentioned a few ways that it has been overcome in the past. I think whoever ends up making “sciddit” will have to give some serious thoughts to making the system inherently beneficial to the scientific community, such that it makes what scientists already do easier, and thus an obvious choice for a scientist to use.

      Regarding point #2: That’s the idea behind “sciddit”: the scientific community would be in charge of how everything is organized. There’d be a few mods to help overcome abuse, but they would be members of the community, selected by the community. Most of the moderation would be performed by the community via upvotes/downvotes.

      • http://jasonpriem.org Jason Priem

        regarding #2: I agree with you, a Reddit-style approach lets the community do a lot. But one thing we’ve learned from Reddit is that the community is not autonomous, but rather curated. The same is true of my other go-to example of Awesome Online Communities, Stackoverflow. It’s got a very strict set of rules and norms that don’t work for everyone.

        And I think most of us would agree this is totally ok. If you don’t like the direction the community is headed, you can go somewhere else. But given the inevitable reality that communities are controlled, it becomes really important that you have a somewhere else to go to. And that’s tricky, because online communities want to move toward being only place you go. They want to be as big as they can, to take advantage of positive network externalities; Facebook works better when everyone is on Facebook. And, to explicitly state the more ominous implication, when no one is anywhere else.

        This isn’t necessarily a bad thing; the web evolves new species pretty quickly, and so far we’ve seen lots of healthy churn in networks. But in a narrow domain like scholarship–where we’ve shown ourselves all too willing to hand over our social infrastructure to monolithic corporations already–the idea of an Online Network To Rule Them All ought to, I think make us a bit nervous. So while I totally support the idea of a sciddit, I do have reservations about there being a the sciddit, even one run by the benevolent dictatorship of the scholarly proletariat.

        • http://jasonpriem.org Jason Priem

          Drat, seems I forgot to close a link tag in there, and there’s no way to edit. Apologies.

          • http://www.randalolson.com Randy Olson

            Fixed it for ya. :-)

  • Pingback: Could the reddit model replace the current scientific publication system? | openingscience.org()

About this blog

The data visualizations on this blog are the result of my “data tinkering” hobby, where I tackle a new data analysis problem every week. If I find something interesting, I report my findings here to share with the world.

If you would like to use one of my graphs on your website or in a publication, please email me.

Archives

Enter your email address to subscribe to this blog and receive notifications of new posts by email.