Assignments

Assignment 1: List of Papers for a Systematic Review

Imagine you are doing a systematic review (see Systematic Literature Studies). Leave out data analysis and synthesis – perform only the initial steps:

  • define the topic and research question(s),
  • specify the search strategy in detail,
  • define inclusion and exclusion criteria,
  • perform the search,
  • and list the found relevant articles matching the criteria.

The narrower the topic, the less work you will potentially have during filtering of the papers; recall that a systematic review aims to ideally include all relevant works. However, there should be at least 20 relevant papers in the resulting list. If inclusion criteria specify a time range, it should span at least 5 years.

Your goal is to achieve as high recall and as high precision as possible with respect to all published research papers relevant to the given topic/question and matching the criteria. Since the set of all relevant published papers is unknown, the reviewer (teacher) can only guess it by assessing the quality of the research method and/or by performing random keyword searches.

You can use the fact that you are working in teams to your advantage: for example, label a subset of papers by multiple persons, compute inter-rater reliability, and argue with this for the quality of your study. Alternatively, label all papers by two people and then resolve disagreements.

Semi-automated tools are allowed and encouraged. However, simply entering your research question into a chatbot and copying all results is definitely not enough. As the internal workings of the tool are opaque, you have to ensure the research method is valid, a large portion of relevant papers are included and irrelevant ones excluded.

The assignment should be submitted as a report in PDF format, containing a clear description of the method (including a diagram is helpful) and the list of relevant papers as bibliographic citations.

Assignment 2: Controlled Experiment Data Processing

Your task is to design and describe an imaginary controlled experiment. It will not be actually executed, so you can make up the data and state this in the report. Although unethical in practice, as an assignment this is a practical approach since executing a controlled experiment is often resource-intensive, and finding existing raw data from a controlled experiment without a paper already describing this experiment in detail is rare.

The report should describe at least:

  • the research question, the null and alternative hypotheses,
  • variables (including their scales),
  • experimental design,
  • details of the procedure,
  • results (the made-up data, effect size, statistical testing),
  • threats to validity,
  • and conclusion.

The topic of the experiment should be related to computer science (interdisciplinarity is accepted). Quasi-experiments are allowed, but this has to be clearly stated in the report.

The experiment should be reported using a computational notebook such as Jupyter, marimo, or Observable Notebooks. A ZIP file should be submitted, containing:

  • notebook source files (*.ipynb, *.py, etc.),
  • data files, if any (e.g., *.csv),
  • an HTML (including images) or PDF export of the notebook.

Assignment 3: Review of a Paper

Traditionally, a review contains a plain-text description of the paper’s main point, strengths, weaknesses, and suggestions for improvement (see Reviewing). The paper is judged using coarse-grained and vague criteria, such as novelty, validity, and presentation. This leads to very subjective reviews and a low level of agreement between multiple researchers reviewing the same paper.

To tackle this, a team of software engineering researchers led by Paul Ralph devised the ACM SIGSOFT Empirical Standards. Although the standards are primarily intended for software engineering, they encompass a wide range of research methods, which makes them usable for many other subfields of computer science. The Empirical Standards are based on checklists. First, the reviewer selects research methods that the paper uses. The tool then generates a list of specific attributes, such as “states formal hypotheses” or “justifies sample size”. The reviewer responds “yes” or “no” to each item; in the case of a negative answer, the system asks further questions. After submitting the form, the review result (acceptance or rejection) is generated. To learn more about the standards, see, for instance, a paper by Arshad et al. (2021).

In this assignment, you will select a computer science research paper from arXiv:

  • at least 10 single-column or 8 two-column pages long,
  • less than 12 months old,
  • that probably has not yet been reviewed.

The last point can only be guessed, but generally this paper should not have an “accepted/published at” comment on arXiv, and searching the web for its title should not lead to a final published version. Then generate a checklist by selecting all methods that apply to the given paper. At least two methods must be checked; otherwise, please select another paper for this assignment.

Next, as the primary purpose of this assignment is not to accept or reject a paper but rather to assess your thinking, we will proceed a bit differently. Copy the generated attributes into a text document. In addition to choosing “yes” or “no” for each attribute, append a short note explaining your decision, such as the positions in the text where this criterion is fulfilled or a statement on why it is unfulfilled.

Grading will take into account whether the appropriate methods were selected in the generator and whether the decisions about attributes are correctly justified. You can ignore “desirable” and “extraordinary” attributes in the checklist. The maximum number of points for the assignment will then be limited to 12.

You should read your selected paper carefully, so you will be able to evaluate the attributes. You can use AI chatbots to learn, for example, what a given empirical standards attribute means in general, provided you verify the sources. However, it is forbidden to upload the paper to a chatbot and copy-and-paste its decisions on attributes.

The assignment should be submitted as a PDF file containing the paper’s title, a link to it, the list of methods, and the review checklist as described above.