4  Choosing Research Questions and Methods

After we select a specific research topic and idea, it is necessary to specify the research question(s) and/or hypotheses and choose a research method to answer them.

4.1 Research Questions

A research question (RQ) is a clearly formulated question that we would like to answer in our study using research methods. In general, a good research question should be:

• Focused: If we have multiple research questions per paper, they should all relate to the main goal.
• Researchable and feasible: We must be able to answer it scientifically and using reasonable resources.
• Specific: A research question should not be vague.
• Complex enough and relevant: We should not study questions that will be useless for sure.

To provide more specific examples, Begel & Zimmermann (2014) and Huijgens et al. (2020) provide lists of questions that industrial software engineers find relevant. Similar lists may exist for other subfields.

Sometimes, a research question (or a hypothesis) is only implicit in a paper. This is the case mainly for papers whose main contribution is an artifact – the research question then might be “Is it feasible to design X?” or “How to design an approach X so that it has some property Y?”. However, particularly for empirical studies, we should always state a research question explicitly.

4.2 Hypotheses

A hypothesis is a declarative sentence whose truthfulness is yet unknown. It often specifies a relationship between variables, e.g., “A high-contrast mouse cursor improves the clicking precision on a target rectangle on a screen.” A hypothesis must be falsifiable, which means we can reject it using an empirical research method.

Any hypothesis can be written as a research question, but the reverse does not always hold. For instance, the question “How do developers unit test database modules in procedural languages?” cannot be written as a hypothesis. If a statement is statistically testable, the hypothesis is a preferred form.

A hypothesis is a statement about a more specific phenomenon, such as “When debugging, having dynamic information from Senseo (Rothlisberger et al., 2012) available reduces the time for solving maintenance tasks.” In contrast, a theory is a proposition explaining a certain general phenomenon. It is usually based on the generalization of a large quantity of evidence in the area. For example, an information foraging theory for debugging (Lawrance et al., 2013), explains how the developers navigate the program, similarly to a predator following prey, using concepts such as scent, proximal cues, or topology.

4.3 Operationalization

We know that a research question or a hypothesis should be specific and precisely defined. However, even well-stated hypotheses and RQs contain terms with many possible interpretations. Consider the hypothesis:

Using a time-traveling debugger improves developers’ efficiency in fixing bugs.

There exist multiple time-traveling debuggers with different features. Many people consider themselves developers: from high-school students coding for fun to senior developers in corporations. What actually constitutes efficiency is a notoriously debated problem.

We thus have to provide operational definitions of important terms. An operational definition expresses how the given term will be interpreted in this specific study, i.e., how it will be measured, observed, or applied.

For example, we will use a specific time-traveling debugger TimeDebugX, implementing certain features. By “developers”, we will understand professional programmers with at least one-year industrial Java experience. We will measure “efficiency” as the number of successfully solved bug-fixing tasks per hour.

4.4 Choosing a Research Method

Many beginning researchers make a mistake of selecting a research method without having any specific hypothesis or RQ in mind. For instance, they design a new approach, which they would like to empirically evaluate. They start thinking about a questionnaire survey as a relatively easy way to collect data from multiple respondents. They start adding many questions to the survey, ranging from demographic details of every kind to many unclear and suggestive questions about how the respondents liked various aspects of the newly designed system.

This is definitely a wrong practice. When doing research, we should always define RQs/hypotheses first and only then proceed to choosing research methods. Otherwise, we risk that the method will answer only questions that are irrelevant or will not reliably answer any research question at all.

There are multiple guides and overviews suggesting the most suitable research methods for given types of research questions, e.g., by Wohlin & Aurum (2015) or Easterbrook et al. (2008). In the next chapters, we will describe some of the frequently used research methods, including controlled experiments, surveys, interviews, case studies, and benchmarking studies among others. For each method, we will mention examples of RQs it is suitable for answering. However, in practice, the methods and their principles are sometimes not clearly separated, so we may use, e.g., the concept of sampling known from surveys in a benchmarking study.

During a research project, it is often beneficial to use multiple methods or data sources in a practice called triangulation. For example, we can obtain requirements for our new approach using a survey, assess its machine-based efficiency by a benchmark, find its advantages and disadvantages in a case study, and finally assess its human-based efficiency using a controlled experiment with humans. This increases the validity of the study and provides a more comprehensive understanding of the subject.

Exercises

1. Which of the following are good research questions and why?
   1. Why do students fail in the operating systems bachelor’s courses?
   2. How to determine if a program written in C terminates without running it?
   3. What proportion of C++ Gists on GitHub is compilable without downloading third-party libraries?
2. Which of the following are good hypotheses? Why?
   1. Java is a fast programming language.
   2. An LL parser is faster than an LR parser for converting JSON into an in-memory tree structure.
3. Find a paper in your field that explicitly lists operational definitions of terms used in a hypothesis or a research question.

Begel, A., & Zimmermann, T. (2014). Analyze this! 145 questions for data scientists in software engineering. Proceedings of the 36th International Conference on Software Engineering, 12–23. https://doi.org/10.1145/2568225.2568233

Easterbrook, S., Singer, J., Storey, M.-A., & Damian, D. (2008). Selecting empirical methods for software engineering research. In F. Shull, J. Singer, & D. I. K. Sjøberg (Eds.), Guide to advanced empirical software engineering (pp. 285–311). Springer London. https://doi.org/10.1007/978-1-84800-044-5_11

Huijgens, H., Rastogi, A., Mulders, E., Gousios, G., & Deursen, A. van. (2020). Questions for data scientists in software engineering: A replication. Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, 568–579. https://doi.org/10.1145/3368089.3409717

Lawrance, J., Bogart, C., Burnett, M., Bellamy, R., Rector, K., & Fleming, S. D. (2013). How programmers debug, revisited: An information foraging theory perspective. IEEE Transactions on Software Engineering, 39(2), 197–215. https://doi.org/10.1109/TSE.2010.111

Rothlisberger, D., Harry, M., Binder, W., Moret, P., Ansaloni, D., Villazon, A., & Nierstrasz, O. (2012). Exploiting dynamic information in IDEs improves speed and correctness of software maintenance tasks. IEEE Transactions on Software Engineering, 38(3), 579–591. https://doi.org/10.1109/TSE.2011.42

Wohlin, C., & Aurum, A. (2015). Towards a decision-making structure for selecting a research design in empirical software engineering. Empirical Softw. Engg., 20(6), 1427–1455. https://doi.org/10.1007/s10664-014-9319-7