POST-HOC SIGNIFICANCE BIAS:

When decision levels for the reasonable amount of Type I (alpha) and Type II error (beta) are selected after the data have been examined, conclusions may be biased¹. Type I and Type II errors affect all studies; because, in most cases, it is infeasible/impossible to study an entire population. It is considered best practice to select a reasonable level/amount of Type I or Type II error at the start of a study, to convince readers that the statistical significance (or lack of) observed in the study could not have been ‘rigged’ or directed by the data (i.e. biased). Although not obligatory, use of a conventional level (e.g. 0.05 for alpha) often helps convince readers, when there is a lack of evidence for another value. If the scientist selects a level of error after the study is completed, and especially if the level changes for different analyses, then inevitably they could prove anything they wanted to prove to suit the scientist’s preference; thus negating the ‘objectiveness’ of the study. Also see: Insensitive Measure Bias, and Wrong Sample Size Bias.

Reference:

1. Sackett DL. Bias in analytic research. J Chronic Dis. 1979;32 (1-2):51-63. (Link to Reference)