Quality of scientific evidence (in social sciences) and research integrity
Despite the rigor of the scientific process, the quality and credibility of its outcomes vary. Research credibility depends on a network of interconnected factors and a multiverse of pathways. In this course, we will untangle this network to better understand the process of scientific knowledge generation. We will place particular focus on its weak spots, which is crucial for critically evaluating evidence quality and improving one’s own research endeavors. The course will cover essential topics such as confirmatory vs. exploratory research, measurement problems, issues related to sampling and careless responding, questionable research practices, as well as broader issues of reproducibility, empirical robustness, and replicability. Subsequently, we will discuss evidence synthesis in both quantitative and qualitative research settings, and methods for assessing study quality and informativeness. A portion of the course will be dedicated to research integrity and open science, focusing on methods to improve the trustworthiness of findings. We will conclude with the topic of the responsible use of generative AI (LLMs), showcasing its potential and applications in research settings. The entire course will be a mix of meta-science, research ethics, and research methodology combined with a gentle touch of statistics. Each topic will be demonstrated through practical examples, minigames, and heuristics. By the end of the course, students will be equipped with a broader understanding of how the scientific process works and with skills which can be applied both to evaluate the credibility of published papers and improve their own research.
The course will be conducted in English and will be contact teaching only, with no remote option available. It is intended primarily for BA and MA students interested in current issues in social science research, including the credibility/replication crisis, research methodology, research integrity, and meta-science. Please note that around 90% of the course will primarily concern quantitative research. The course will help students critically evaluate scientific outputs and improve their own research practices.
Requirements: Students are expected to have experience reading at least five quantitative/empirical papers in the field of social sciences. They should be familiar with basic statistical concepts (e.g., standard deviation, confidence intervals), but advanced knowledge of statistics, research methodology, or coding is not required.
Assignment: In addition to lecture participation (at least 2 out of 3), the task is to conduct a post-publication peer review of a pre-selected manuscript in small teams (3–4 people). Examples of peer reviews will be provided, as well as a list of recommended reading. The review should be written as a report and briefly presented (~10 minutes per team) on the last lecture day.
Pre-course reading: A list of suggested readings will be made available later. Although reading these is optional, I highly recommend going through at least some of them to facilitate understanding and discussion during the course.
Lecturer: Matúš Adamkovič