SMALL STUDY BIAS:
A bias relevant to studies of knowledge synthesis (e.g. systematic review). A bias in a meta-analysis, or statistical summary of results on a topic, due to the inclusion of studies with small sample sizes1,2.
Small Study Bias should not to be confused with Small Trial Bias; a type of Interpretive Bias that occurs when one makes decisions about the quality of a study based solely on the fact that the size of the sample is small. Small Study Bias refers to the mathematical and clinical implications of having too little data to draw meaningful conclusions. Whereas Small Trial Bias, refers to a biased opinion someone has about the quality of study from observing its superficial characteristics.
It is generally agreed that small studies, although useful for generating new hypotheses or discussions on a topic, are not valid for drawing meaningful conclusions on cause-effect relationships between exposures and diseases; because there are too few data points to conduct any meaningful evaluations or analyses. However, the definition as to what is considered small is highly variable across different research scenarios. A study that is considered small in one scenario may not be considered small in another (there is no universally accepted definition of small for all medical research scenarios).
In general, small needs to be determined by consensus amongst experts, and is depended on the size of the relationship that one expects to see in a study, the variability/range of the data that came from the measurements, how much error one is willing to accept when determining if there is a relationship (often denoted as alpha and beta), as well as what clinically meaningful size of relationship between variables would be reasonable, in the specific scenario, irrespective of any statistical observations. Also see: Publication Bias, Small Trial Bias, Significance Bias, and Empiricism-Narcissism Bias.
References:
1. Song F, Parekh S, Hooper L, Loke YK, Ryder J, Sutton AJ, et al. Dissemination and publication of research findings: an updated review of related biases. Health Technol Assess. 2010;14(8):iii, ix-xi, 1-193. (Link to Reference)
2. Müller D, Pullenayegum E, Gandjour A. Impact of small study bias on cost-effectiveness acceptability curves and value of information analyses. The European journal of health economics : HEPAC : health economics in prevention and care. 2015;16(2):219-23. (Link to Reference)