9 Reviewing and Publishing
After we finish writing a paper, we send it for review to an appropriate publication venue, usually a journal or a conference. If the outcome of the review process is eventually positive, it is published in the given venue.
9.2 Choosing a Publication Venue
Many publication venues exist, differing in topics, acceptable contribution types, quality, and review process. We should always read the instructions on the website of the given venue to decide whether our paper fits it. Reading a few papers published in the venue gives us a feeling of what will be expected from us. In fact, it is best to submit to a venue whose papers we often read, provided the quality of our work suffices.
The two most common types of publication venues are journals and conferences. Computer science journals are typically suitable for mature research results with complete evaluation. On the other hand, many conferences accept also works with preliminary evaluation, particularly in their Early Research Achievement (ERA), New Ideas, “short papers”, or similar tracks, poster sessions, and co-located workshops or working conferences. In computer science, many researchers publish a paper at a conference first, and then they submit a significantly extended version to a journal, taking into account the rules of the publishers on what “significantly extended” means.
Other types of publications are non-conference proceedings and books. Non-conference proceedings are collections of papers, usually published annually, that do not involve presenting at a conference. In general, their quality criteria tend to be less strict than that of journals and conferences. Books in computer science are traditionally used to summarize highly developed topics, only rarely for original primary research.
Generally, we cannot publish the exact same content twice, which makes sense. Copying large portions from one of our published articles to another is also forbidden, unless we clearly claim what is the original contribution of the new paper with respect to the old one (and both publishers’ copyright rules allow it). However, bachelor’s, master’s, and doctoral theses submitted to a university library are non-archival publications, i.e., they are not considered formal academic publications per se. This practically means we should be allowed to publish their large portions, such as chapters, as original research papers in a journal or conference. Other non-archival documents include technical reports, which are simply articles uploaded on the web, lacking any review process.
When selecting a publication venue, one of the criteria we could take into account is its acceptance rate. It is defined as the number of accepted papers divided by the number of submitted papers, as a percentage. Rather shocking for newcomers, the acceptance rate of top-ranked conferences and journals can be as low as 10 to 20%. Some journals display acceptance rates on their publisher’s website, but many do not. For selected conferences, the Computer Science Conference Statistics shows acceptance rates.
Given the low acceptance rates, we could be tempted to submit one paper to multiple journals or conferences at once, without waiting until it is rejected in the first venue. However, simultaneous submission is strictly forbidden in the academic sphere, and it can result in a permanent publishing ban. We should also refrain from submitting our papers to so-called predatory journals that publish low-quality papers, fake the review process, spam researchers, or have hidden fees. Beall’s List can act as one of the hints to decide.
9.3 Process Overview
This section summarizes the whole process from the point of finishing a paper draft to the point of having the paper published. Although we mention two typical procedures, one for journals and one for conferences, in reality many venues use something in between. We always need to consult the venue’s website for instructions.
9.3.1 Journals
This is a typical process applied in journals, which often lasts from a few months to a year:
- We select an appropriate journal and modify our draft to use the supplied template.
- When we think the paper is ready for review, we submit it using the journal’s online submission system.
- The editor of the journal sends review invitations to researchers that could be interested in the paper. The selection is performed using personal knowledge, the submission system’s database, paper references, or our suggestions.
- Reviewers that accept the invitation read our article and write textual reviews. A typical number of reviewers is two to four, usually three. Review deadlines typically range from two weeks to two months.
- The reviewers also supply suggestions to accept, reject or revise the paper. The term major revision is sometimes used if the reviewer will need to see the revised paper to decide, while minor revision means the reviewer is already decided the paper should be accepted after a revision.
- The editor then makes a final decision based on the recommendations of the reviewers.
- If the decision is to revise the paper, we make the suggested changes to the paper. We submit it along with a Response to Reviews document, which should contain answers to the reviewer’s comments linked with the descriptions of the corresponding changes made in the paper.
- The editor sends the revised version of the paper to the reviewers, as necessary. The process repeats from point 4 until the paper is accepted or rejected.
- In the case of rejection, the process ends here. We should substantially improve it and re-submit it or submit it to another journal. If the paper is accepted, it is sent for “production” (copyediting and typesetting).
- The accepted paper is put online by the publisher.
- The paper is assigned to a specific issue of the journal. At this point, it is considered published.
9.3.2 Conferences
For conferences, the typical process is a bit different:
- We find a conference with suitable deadlines, topic, and location.
- We select a suitable track of the main conference (Research Papers, Tool Papers, ERA, etc.) or a co-located conference or workshop.
- Our manuscript must be modified to respect the template and the page count specified by the conference track. We submit it using the submission system until a firm deadline.
- Members of the program committee bid for the papers, expressing how much they want to review them by numeric scores.
- Usually three reviewers are assigned to our paper using these scores.
- Each reviewer submits a textual review and a recommendation on a given scale, such as reject, weak reject, weak accept, or accept.
- The reviewers discuss until they come to a decision: either accept or reject the paper.
- Based on the discussion, program chairs make the final decision.
- Until the approximate deadline displayed on the website, we receive a notification containing the decision and the text of the reviews.
- If the paper is rejected, the process ends here; we can improve it and submit elsewhere. In the case of acceptance, we submit a final “camera-ready” version until a deadline. This version is not checked by the reviewers again.
- We register for the conference, travel to the given location, and present the paper.
- The conference proceedings with all presented papers are published.
In recent years, some top-tier conferences have adopted a two-phase review process with revisions, similar to journals.
9.4 Reviewing
In academia, there is no single authority that reviews papers; instead, other researchers from our field (peers) do it. Therefore, we call it peer review.
The purpose of peer review is twofold. First, we decide whether the paper is ready for publication. This should, in theory, prevent incorrect or irrelevant papers from appearing in journals or conferences. Second, we try to increase the quality of the paper by providing comments.
Journals and conferences often list specific criteria upon which the reviewers should judge papers. As an example, we can mention the ICSE 2020 criteria: soundness, significance, novelty, verifiability, and presentation. In general, we can try to answer the following questions (loosely based on Zobel, 2014):
- Is the motivation, contribution, and research question described clearly?
- Is the contribution significant enough to be published?
- Is the method described clearly? Could we perform the study ourselves given such a description?
- Is the methodology valid?
- Do the results seem correct?
- Does the conclusion answer the research question?
- Is related work described in sufficient quantity and quality?
- Is the paper logically structured and well-written?
Some venues provide a specific structure that the text of the review should have, but generally we can expect a brief summary of the paper, a list of its strong points, description of its weak points, and suggestions for improvement.
Traditionally, the review process is single-blind (or single-anonymous), which means reviewers can see the authors’ names, but not vice versa. In double-blind review, the authors have to anonymize their names in the paper and refer to their own works in a way that does not reveal their identity. The texts of the reviews are private in either case. Open review, which is still relatively rare, publishes the reviews alongside the papers, sometimes even with the reviewers’ names. Open reviews for some conferences are available at OpenReview.net. In the PeerJ Computer Science journal, reviews are available thanks to the “Read the peer review reports” button at the beginning of the corresponding article’s page. We recommend reading a few reviews to see how they typically look.
Regarding the ethical issues, we should not review papers for which we have a conflict of interest: advisorship, sharing the same institution, or a recent co-authorship with one of the authors. The papers that are currently under review are usually confidential, i.e., we should not disclose their content to others or publish the discussed ideas in any way.
9.5 Publication Access Models
In the past, publishers physically printed journals or conference proceedings and sold them to institutions and readers. Typesetting, printing, and distributing the physical copies incurred substantial costs. Institutions, such as universities, subscribed to journal bundles and paid for these costs. The publishers did not pay authors as the profit margins were not extremely large, and the authors were interested in reputation more than profit. To ensure profitability, the publishers required the authors to transfer their copyright to them or sign an exclusive copyright license, which means their rights to distribute papers were very limited, even the ones they wrote themselves.
With the advancement of the Internet, the publishers started to replace printed publications with digital libraries. The costs decreased dramatically, but the subscription prices were growing steadily. The publishers did not want to give up their subscription-based access model, based on paywalls. The readers need to access papers through a subscribed institution, otherwise they would need to buy individual articles for inappropriately high prices, of which nothing goes to the authors (and reviewers).
The open access principle tries to change this situation with multiple new models. Green open access means a paper is published as usual, behind a paywall, but the authors can put a free copy of their manuscript (so-called pre-print or post-print) on their website or selected repositories. Usually the publisher imposes a list of requirements, such as the version of the paper that must be used and phrases that have to be pasted into the paper. The authors should always read a specific license they are signing, but the Jisc Open policy finder is useful for an overview. For all researchers, we definitely recommend creating your own website, however trivial, and uploading all papers compatible with green open access (nowadays almost all) to it. Additionally, we recommend uploading to arXiv, which makes the paper permanently accessible, provided the publisher allows this.
In gold open access, the publisher itself makes the papers accessible to everyone, without a paywall. Many publishers, however, ask an article processing charge (APC) for such a service, which is often an exorbitant sum.
9.6 Referencing Published Papers
After a paper is published, we can refer to it using a bibliographic citation. Its format depends on the type of the publication. For both journal and conference articles, the reference contains the authors’ names, the title of the article, the year, and page numbers or article number. In addition to this, a journal article reference includes the journal’s name and volume and/or issue, while a conference paper reference contains the name of the conference proceedings.
To globally identify a paper (or any document), publishers assign it a Digital Object Identifier (DOI). It is a unique permanent handle in the form 10.xxxx/.... Visiting the DOI’s corresponding URL, e.g., https://doi.org/10.1007/s10664-021-10083-5, redirects us to the official publisher’s version of the paper. Including a DOI in bibliographic citations is recommended or required by many publishers.
A journal as a whole, i.e., not its specific issue or paper, can be identified by an International Standard Serial Number (ISSN). It is an 8-digit number assigned to periodical publications. A conference proceedings volume is assigned an International Standard Book Number (ISBN), which has 10 or 13 digits.
9.7 Data Deposition
Even after research findings are published, they should be subject to further scrutiny by other researchers to confirm their correctness. This is achieved through reproduction and replication of studies. According to ACM Artifact Review and Badging Version 1.1, reproducibility is the ability to obtain the same results by a different team with the same experimental setup, which includes the artifacts (programs and data) supplied by the original authors. On the other hand, replicability means the ability to obtain similar results using artifacts developed by the new team independently.
Publishing the artifacts, i.e., programs and data, used during research along with the paper is therefore becoming the norm. Uploading the data to a personal website or a shared drive and linking it in the paper’s text is better than nothing. However, these links often become invalid after some time. Thanks to specialized research data repositories, such as Zenodo, OSF, or Figshare, we can deposit data permanently and even assign it a DOI.
When depositing programs and data, we may be faced with the question of a license selection. To understand this issue, we have to explain the notion of copyright. Copyright is a legal protection of creative works such as text, images and software. Contrary to a popular belief, the purpose of a license is not to restrict otherwise unlimited rights. Instead, all new creative works are copyrighted by default. To allow others to reasonably use and reuse our works, we must explicitly declare a license. For software, we may use a website to choose an open source license. For text or images, Creative Commons offers a variety of licenses with different terms. If we are not the sole authors of the artifacts, for instance, if we transformed existing data, we need to take the original license into account during our decision.
Exercises
- What are the publication types of these documents?
- How could we speed up the traditional submission and publication process of journals or conferences without compromising review quality?
- Why do you think simultaneous submission is prohibited?
- Is the current peer review practice effective in preventing papers with factual errors from appearing? Why or why not?
- What are the advantages and disadvantages of single- and double-blind review?
- Why do you think the subscription-based model and the gold open access model with high APCs are still widely used when nowadays the price of hosting papers online is negligible?
- Find a published replication study in your field. Did the replication confirm the original results?