3 Working with Literature

The techniques for finding relevant literature depend on whether we would like to get an overview of a broad area or find papers related to a specific idea. Learning how to manage a large quantity of papers and read them effectively is also important. This chapter is concluded by an overview of systematic reviews, which are a special kind of studies summarizing literature on a given topic.

3.1 Exploring the Field

For aspiring researchers, such as master’s students thinking about their potential thesis topics or first-year PhD students, it is beneficial to see the big picture of their field. This allows them to get an intuition of what was already done years or decades ago, what the current hot topics are, and how the papers in their area typically look.

Research in computer science is typically published as a paper either in a scientific journal or conference proceedings. Therefore, the first step is to prepare a list of the best journals and conferences in the given field. For journals, there exist catalogs filtrable by sub-categories, such as Journal Citation Reports (accessible only from the university network), Scimago Journal Rank, and Research.com journal rankings. For conferences, there is CORE ranking and Research.com conference ranking. The list of top publication venues at Google Scholar mixes journals and conferences together.

For each journal we are interested in, we should find its official homepage, find a few latest issues, scroll through the list of paper titles, and download a few of them to get a feeling of what constitutes top-notch research in the given area.

For conferences, we should also open their official homepages. Most conferences are annual, so we focus on the last two or three years. Top-tier conferences tend to be large and contain multiple tracks, which consist of sessions. It is advantageous to open the main research track’s page and write down the names of its sessions (excluding lunches and coffee breaks, of course). Suppose we would like to explore the field of software engineering. At the ICSE 2024 website, we click Program / ICSE Program, expand the “Filter Program” section, and select Tracks / ICSE Research Track. Then we can see the list of days, each containing multiple sessions, in our case: AI & Security 1, Evolution & AI, Testing 1, Analysis 1, Human and Social 1, Generative AI studies, and many others. Constructing such a list helps us understand the current trends in the given field. For the sessions we are interested in, we can also inspect the list of the corresponding papers.

3.2 Getting an Overview of the Topic

If we narrowed our interest to a more specific topic, such as “time-traveling debugging” or “plant identification using computer vision”, we may benefit from reading secondary studies, i.e., articles summarizing existing papers on a given topic. The easiest way to find them is to append the keywords “review”, “survey”, “systematic literature review”, and “systematic mapping study” to the name of our topic and enter them into Google Scholar. Trying synonyms of words in the topic increases the chances of finding the relevant studies. We can also make the topic more specific or general as necessary.

For instance, if we are interested in declarative debugging, we may try:

“declarative debugging” review
declarative debugging survey
algorithmic debugging “systematic mapping study”

Note that some terms are not generally synonyms: algorithmic and declarative are different terms, but algorithmic and declarative debugging is approximately the same topic. It is thus important to get acquainted with the concepts used by other researchers.

3.3 Searching for Related Papers

Suppose we already have a relatively specific research idea, e.g., “time-traveling debugging using a graphical processor” or “an experimental comparison of cybersecurity education using a traditional and flipped classroom”. How do we determine if someone has already published a similar paper and find related papers that we can use for inspiration or for comparison in our Related Work section?

3.3.1 Academic Search Engines

We may use traditional web search engines such as Google or Bing, which return also research papers among other results. However, specialized academic search engines aim to limit the results only to academic literature and offer useful tools such as forward reference lists, which speeds up the process. We will mention some of the more popular ones.

Google Scholar has high document coverage, which is its main benefit and drawback at the same time. Practically all academic documents are indexed – not only journal and conference papers, but also theses, books, and technical reports. It does not exclude low-quality publications, and occasionally includes also files such as manuals and company whitepapers. No subscription is required.

Semantic Scholar is a tool similar to Google Scholar, with a slightly lower document coverage. The usage is free.

Scopus, on the other hand, is a paid service, so it is accessible only from the subscribed universities’ networks or after registration with the university e-mail. Coverage is limited to papers passing certain quality criteria. It offers two features indispensable for systematic literature reviews: advanced search based on tens of properties combinable using a custom query language; and export of all search results into a structured file. Sorting the results by relevance is not on par with the other engines, though.

Web of Science is another paid service. It boasts of strict quality criteria for inclusion. Despite its popularity in many fields, its usefulness for computer science research is severely limited. Waiting times for the indexing of new journals and conference proceedings are as long as a few years. Many conferences are not indexed at all.

OpenAlex is a free alternative website, offering export of the results and an API.

Searching for related papers using AI chatbots usefully supplements academic search engines. However, these tools often do not have full-text access to paywalled articles, so using them as a sole source does not suffice. Care must be also taken to verify if the given results are actually relevant to our prompt.

3.3.2 Searching Process

To find as many related works as possible, it is advised to use a high-coverage academic search engine, such as Google Scholar, with a systematic approach. The general procedure is as follows:

Enter a search phrase describing your idea succinctly yet precisely.
Read the titles of the documents on the displayed results page.
For each document that seems relevant based on title, read its abstract. If the abstract seems relevant, inspect the full text quickly: skim the Introduction, Conclusion, figures, and tables. Search the full text for clues – e.g., if finding an experiment with humans, search the terms such as “human”, “subject”, “participant”, or “enrolled”. If the paper is still relevant, download it and save it to your reference manager.
Go to the next pages of results until you keep seeing relevant papers, repeating steps 2–3.
For each saved paper, look at the backward references. These are the works listed in its References section and thus older than the given article. If any of them seems relevant, perform step 3 on it.
For each saved paper, list its forward references, i.e., newer papers that cite it. Google Scholar names this feature “Cited by …”. Find and save relevant papers in this list.
Repeat steps 5 and 6 on all articles saved in the reference manager but not yet processed. Stop when you seem to be approaching infinite recursion, i.e., the documents you see start repeating too much.
Read the full text of the saved papers that seem the most relevant.
Repeat steps 1–8 with alternative search terms: synonyms or other aspects of the problem. When forming the search terms, use the knowledge you acquired so far.

Obviously, searching for related works is a long-term process. After performing a five-minute search, we should not claim a paper describing ideas similar to our own does not yet exist.

Tip: We can save time by configuring search keywords for academic search engines in a browser (Firefox, Chrome). For instance, after assigning the keyword “s” to Google Scholar, we can simply type “s motion tracking” into the address bar.

3.4 Accessing Papers behind a Paywall

Some papers are inaccessible for general public without paying. As we will explain later in this book, paying for papers as individuals is not reasonable. Instead, we should apply the following tips.

First, on the publisher’s websites, there are always the final versions available, so it is a preferred way if we have access to the given paper. Many universities have pre-paid access to digital libraries such as IEEE Xplore, the ACM Digital Library (free access from 2026), Elsevier ScienceDirect, Springer Link, and Wiley Online Library.

If we are outside the university network, we can try a VPN to remotely access the given resource. An alternative is to register on the given site with a university e-mail, or apply for the CVTI remote access available in Slovakia.

If the publisher’s version is inaccessible, we should search for the authors’ archived version. For example, on Google Scholar, we can click the “[PDF]” or “All versions” buttons. There are also browser plugins that automate this, most notably Unpaywall.

As a last resort, we can try e-mailing the corresponding author of the paper.

3.5 Reference Management

A reference manager is a program designed to add, edit and view a database of bibliographic citations to research-related documents, along with associated metadata, such as notes, tags, or full text links.

3.5.1 Advantages

Using a reference manager makes a huge difference.

First, we build a database of all papers relevant to us for some reason. Most reference manager have some tagging or categorization capability, so papers related to various projects or topics can be distinguished easily.

Second, papers already read can be marked as such. This may seem useless at first, but after reading at least tens of papers, forgetting is easy.

Third, we can, and absolutely should, make notes about papers. This is related to the previous point – having a paper marked as “already read” is fine, but having a short comment summarizing the main points of the paper is even more useful. Instead of objectively summarizing the facts, we should try to focus on exactly how the paper is relevant to us: How could we use the results? What hinders building upon this paper? Is there any research method used that we could use as inspiration?

Finally, with a reference manager, we can cite effortlessly. Reference managers with BibTeX integration offer the possibility of having one central .bib file on a computer, which can be included in each paper via \bibliography{path/to/file} in LaTeX. Citing is then as simple as pressing a keyboard shortcut and pasting \cite{citationKey} into the source code. Unfortunately, during collaboration this does not work as easily since everyone has a separate BibTeX file.

3.5.2 Software

There are many reference managers available. We will mention a few popular ones.

JabRef is a Java-based desktop application. BibTeX is its native storage format, so all information we edit is actually saved in this structured text file. It offers rich customization options. For example, we can set up automated citation key generation for bib file entries, based on a pattern. Adding custom fields with user-defined names to entries is also possible. An older version with a classical look and feel can be downloaded too.

Zotero offers, among other features, PDF file annotation and simple citation import from websites via web browser plugins. Although it uses an opaque data storage format, the database can be synchronized with a bib file via the Better BibTeX plugin.

KBibTeX is an alternative suitable mainly for Linux distributions with KDE. For macOS, there is also BibDesk.

As a minimalistic alternative, using a spreadsheet or a structured text file is better than nothing.

3.5.3 Citation Sources

Citation entries for the same article from various websites vary in quality. Although it is possible to download a BibTeX entry from Google Scholar by clicking “Cite” and then “BibTeX” under the given document, such a citation often misses necessary fields, such as the issue number of a journal or a Digital Object Identifier (DOI). Downloading citation information directly from the publisher’s website usually provides all available information. Alternatively, we can use sites such as the ACM Guide to Computing Literature, which contain citation records from multiple publishers.

If the downloaded BibTeX record lacks some necessary information, it is sometimes necessary to correct it manually.

3.6 Reading Papers

Now, let us consider an important motivational point: Can someone be a good film director without watching a lot of movies first? How would such a director even know what is a good or bad movie? Similarly, can we be a good researcher without reading a lot of papers first?

However, since research papers are usually long and dense pieces, we should not read them like news articles. Instead, we should adopt the three-pass approach (Keshav, 2007):

The first reading pass should take at most 5-10 minutes:

Read the title, abstract, and introduction quickly.
Skim headings, tables, figures.
Briefly read the conclusion.
Mark important references.

If the paper is relevant, we can continue with the second pass. We try to read the rest of the paper from start to end in about one hour (of course, this depends on the length), ignoring not immediately important content and information requiring too much time to comprehend: complicated equations not essential to the topic, mathematical proofs, and low-level details including the description of a tool implementation, hardware specification in benchmarks, etc.

If the paper is really important, e.g., we plan to extend it, we consider applying the described methodology, or we deem it one of the three most relevant papers for our upcoming research projects, the third pass is recommended. During it, we can imagine being the author writing this paper. We try to comprehend all details and resolve ambiguities: How exactly would I perform this (unclear) step in the method? How was this result computed? What are the shortcomings? How could I do it better? This pass takes multiple hours.

Sometimes, the second and third passes blend together, particularly if we know beforehand that the third pass will be necessary.

3.7 Systematic Literature Studies

A systematic literature study (often called simply a systematic review) is a research study that aims to collect all works relevant to given questions using a rigorous protocol, analyze them, and synthesize results characterizing them. We recognize the two most common types of systematic literature studies:

Systematic literature reviews (SLRs) in their narrow sense analyze the collected articles deeply, often producing quantitative results by aggregating the results of the original (primary) works. They answer specific research questions, such as: What portion of bugs in Android applications can be resolved using automated tools? Which parsing algorithm is the most efficient for context-free languages?
Systematic mapping studies (SMSs) identify sub-topics, categories, trends, or research gaps. They answer broader, less focused research questions, e.g.: How can automated bug resolution techniques be categorized? Which parsing algorithms exist for context-free languages and what are their shortcomings?

Multiple guidelines exist for conducting systematic reviews, including the ones by Carrera-Rivera et al. (2022), Kitchenham & Charters (2007), and Wohlin et al. (2024). The best way to get an intuition about how a systematic review is performed is to read specific examples of high-quality papers of this kind, e.g., by Hall et al. (2012), Shahin et al. (2014), Inayat et al. (2015), or Arvanitou et al. (2021) in the area of software engineering or many other papers in the respective fields.

To conduct a systematic review, it is first necessary to pose research questions that we would like to answer.

Next, we specify a search strategy, which is a process of the retrieval of all works potentially relevant to our research questions. There are three major search methods: manual search, automated search, and snowballing.

Manual search is performed on the lists of all papers published in relevant journals and conference proceedings. Relevant journals and conferences are selected by personal knowledge of the research field or from catalogs. We then manually scan the lists of all published papers in the given year range of these journals and conferences, while selecting only papers relevant for the study based on titles, abstracts, or full texts if necessary.

Automated search is done by entering keywords into the search boxes of digital libraries and/or academic search engines. First, we carefully select keywords, considering all aspects of our research questions and including synonyms, e.g., for tree and graph-based software visualization it may look similar to (software OR program) AND visualization AND (tree OR graph). A combination of multiple digital libraries or an academic search engine with sufficient coverage is usually used, in a way that maximizes the union of papers covered and minimizes their intersection. It is beneficial if the used websites offer the export of results, otherwise it would be difficult to obtain a deterministic list of papers that can be inspected non-sequentially or by multiple researchers. Similarly to manual search, we select relevant papers based on titles, abstracts, and eventually full texts.

Snowballing means each relevant paper is scanned for backward and forward references. This extends the set of relevant papers. The added papers can again be searched for references, similarly to like a snowball grows.

These search methods are appropriately combined into a search strategy. For instance, we can perform manual search in five journals and seven conferences, then automated search using a database of academic papers, and finally perform backward and forward snowballing with a depth of two on the union of the results from the manual and automated search. During this process, it is necessary to remove duplicates, so that we do not assess one article twice.

To decide which papers are relevant for our study, we use inclusion and exclusion criteria. Each paper included in the study has to fulfill all inclusion criteria and must not fulfill any exclusion criterion. A simple example of such a set of criteria is:

I1: It is a journal or conference paper published between 2015 and 2024.
I2: The paper includes an empirical evaluation of a tree or graph-based software visualization.
E1: Editorials, essays, secondary studies, and tool papers are excluded.
E2: UML-based visualizations are excluded.

The criteria have to be as precise and unambiguous as possible. Ideally, two researchers should be able to independently include/exclude a set of papers with a perfect overlap.

After constructing a list of papers relevant to our study, data extraction is performed. We construct a table where we assign each paper numerical, categorical, or other properties (e.g., year: 2020, type: “graph”, evaluation: “controlled experiment”). The set of potential categories for each categorical property is either defined beforehand or refined gradually during the process of data extraction.

Finally, we perform data synthesis, where we statistically derive conclusions from the extracted data, such as percentages of papers pertaining to each category. The details of data extraction and synthesis depend on the research questions that we posed in the beginning.

Exercises

Choose one of the leading conferences in your favorite subfield of computer science. Open the websites of its last two or three years and find the main research tracks. Compile a list of 10–12 session names per year. If the conference does not have named sessions, list selected paper titles instead.
Select a specific enough idea for which you would like to find related papers. For instance, it can be related to your thesis or any paper that you liked. Describe the idea in one or two sentences. Perform the searching process from Section 3.3.2, ignoring step 4, severely limiting or skipping the recursion in step 7, and ignoring step 8. Stop after saving about 10–12 actually relevant papers. Which part of the process was the most efficient in finding relevant papers? Are you satisfied with the results? Do you have any suggestions to improve this process?
Use an AI chatbot to find the works related to the idea from the previous exercise. Are the results more or less relevant?
Which of the mentioned advantages of reference managers is the most important to you? Can you find any other advantages?
Which reference manager do you use and why?
Find three papers relevant for you that you have never read at all. Read them with the first pass only. After reading each paper, close it and write down all the information that you remembered. How would you characterize the remembered information? For instance, is it somehow important for your research?
(This exercise does not require home preparation.) During your lesson, the teacher will show you a screenshot. Imagine you have an idea that could be explained by a screenshot you see. Your task is to find if a similar idea was already described in research paper(s) and find such a paper. Warning: Do not search for exact texts from the screenshot. Imagine it represents only an abstract idea in your head.