Assignments

Assignment 1: List of Papers for a Systematic Review

Imagine you are doing a systematic review (see Systematic Literature Studies). Leave out data analysis and synthesis – perform only the initial steps:

  • define the topic and research question(s),
  • specify the search strategy in detail,
  • define inclusion and exclusion criteria,
  • perform the search,
  • and list the found relevant articles matching the criteria.

The narrower the topic, the less work you will potentially have during filtering of the papers; recall that a systematic review aims to ideally include all relevant works. However, there should be at least 20 relevant papers in the resulting list. If inclusion criteria specify a time range, it should span at least 5 years.

Your goal is to achieve as high recall and as high precision as possible with respect to all published research papers relevant to the given topic/question and matching the criteria. Since the set of all relevant published papers is unknown, the reviewer (teacher) can only guess it by assessing the quality of the research method and/or by performing random keyword searches.

You can use the fact that you are working in teams to your advantage: for example, label a subset of papers by multiple persons, compute inter-rater reliability, and argue with this for the quality of your study. Alternatively, label all papers by two people and then resolve disagreements.

Semi-automated tools are allowed and encouraged. However, simply entering your research question into a chatbot and copying all results is definitely not enough. As the internal workings of the tool are opaque, you have to ensure the research method is valid, a large portion of relevant papers are included and irrelevant ones excluded.

The assignment should be submitted as a report in PDF format, containing a clear description of the method (including a diagram is helpful) and the list of relevant papers as bibliographic citations.

Assignment 2: Controlled Experiment Data Processing

Your task is to design and describe an imaginary controlled experiment. It will not be actually executed, so you can make up the data and state this in the report. Although unethical in practice, as an assignment this is a practical approach since executing a controlled experiment is often resource-intensive, and finding existing raw data from a controlled experiment without a paper already describing this experiment in detail is rare.

The report should describe at least:

  • the research question, the null and alternative hypotheses,
  • variables (including their scales),
  • experimental design,
  • details of the procedure,
  • results (the made-up data, effect size, statistical testing),
  • threats to validity,
  • and conclusion.

The topic of the experiment should be related to computer science (interdisciplinarity is accepted). Quasi-experiments are allowed, but this has to be clearly stated in the report.

The experiment should be reported using a computational notebook such as Jupyter, marimo, or Observable Notebooks. A ZIP file should be submitted, containing:

  • notebook source files (*.ipynb, *.py, etc.),
  • data files, if any (e.g., *.csv),
  • an HTML (including images) or PDF export of the notebook.