This is a quick writeup about a project I began during my PhD that I planned to continue afterwards as one possible future. Now that I decided to work at Trisk, futures are being shuffled around. My hope is that this idea evolves into an important project. I want to see it blossom, and so the right thing to do is share it with others and get the snowball rolling.

First things first, shoutout to Matt Phillips for the initial spark, and to Peter Vincent for wonderful discussions about the concept.

The basic idea is: the journal article remains the object of work for researchers. As a scientist, you do a bit of work, you write up your results (in a variety of formats and bespoke processes), and eventually (one-side distributed around 6-12 months) you put it on arXiv. You might then submit to a “real journal” or a conference and wait a while for the work to be reviewed. The review might be an iterative process, or it might just be a go/no-go. Either way, it's rarely very transparent, and it's rarely much fun.

Typically, by the time the review process happens, everyone in your (sub)field tends to know about the work already, and the only thing that changes is some grad student is relieved that she might have a slightly higher chance of getting a job somewhere (because her contract is only for a year or two).

Where this goes wrong in my view is the reviewing step. The wildest part of the process (to me) is the lack of focus on the reviewers. As far as I know, reviewers are not paid for their time, they get virtually nothing for what they do except maybe some clout and the ability to pull favors when it's their turn to submit. None of this feels very transparent, collaborative, empowering, or even useful. Aren't these the values we want in Science?

Solution: prioritize the reviewer by creating a market for reviews. The market can be blind or not, the same way twitter has realnames and alts. That’s part of the platform’s beauty, and likewise reviewers can be anon or realname as well. The first trick is to provide a market for clout on the platform that's measured in actual, useful output. The second trick is to build a community around discussing and reviewing scientific work in a generative fashion, the way HN is a community built upon discussing software in a generative fashion. The same way stack overflow has its queens and kings with millions of stars and answers etc. Why do these people do this work, much of it painstaking and requiring the patience of Job? Because they love it. Do we care if these people have PhDs or high-ranking posts? Nope.

So what does this look like logistically? First, attach reviews to papers and let them evolve together. Soon, this will change this will change the way we think about publishing itself. The ML field has already build this through constant conferences, but this isn't exactly scalable. It’s also becoming a balkanised system of FAANG-funded recruitment events rather than a focus on scientific-technical progress to some extent. So open it up! Those conferences will eventually leave openreview and use a platform like I'm describing anyway.

Eventually, we’ll forget about papers, and we’ll think in terms of something we've been calling research atoms— units of work that are bonded to reviews and other atoms to create scientific narratives (compounds?). A dataset can even be an atom. A PhD is a collection of atoms. Atoms have collaborators, not authors, there’s no need for author order, and the insights put into atoms are tracked as those atoms evolve (the same way open source projects are tracked). Open Science Framework is building part of this infrastructure, but I don’t think they’re leaning into the reviewing/reviewer angle enough. The atom platform like this can easily be linked to twitter, osf, etc.

I've been calling the general platform where atoms and their reviews live and evolve Sifter.

The hurdle to overcome is to gain enough momentum that the platform clout (much like twitter followers, which are for the most part scientifically meaningless (stack overflow is a decent metric of someones expertise however!)) actually means something in terms of funding, jobs, etc.

I recently spoke to someone that helped found ResearchGate. The conclusion: it’s a solid idea that ResearchGate dreamed about in it's beginning as well. However, publishing is huge business and dark legal clouds emerged pretty rapidly once RG's vision became clear the incumbents. Spooky. The way around this is the empower the will of the people, stay open and transparent at every step, and let people vote with their clicks.

There are other precedents as well. Yann LeCun has also proposed a similar system for publishing, particularly the market-for-reviews idea. I bet he would even join the project in some minor capacity if he thought it had serious legs.

So what next?

I’d be very happy to advise a team to develop this concept. I think it has the potential to change research publication for the better. Here’s the splash page I whipped up.