Experiment (or controlled experimente) in software engineering is an empirical inquiry that manipulates one factor or variable of the studied setting. Based in randomization, different treatments are applied to or by different subjects, while keeping other variables constant, and measuring the effects on outcome variables. In human-oriented experiments, humans apply different treatments to objects, while in technology-oriented experiments, different technical treatments are applied to different objects.

C. Wohlin, P. Runeson, M. Höst, M. C. Ohlsson, B. Regnell, and A. Wesslén, “Experimentation in Software Engineering,” Springer, 2012. [Springer]

Experiments are concerned with assigning subjects to different treatments by random. Control is a key issue in experimentation. The objective is to manipulate one or more variables and control all other variables at fixed levels. The effect of the manipulation is measured, and based on this a statistical analysis can be performed. In some cases it may be impossible to use true experimentation, we may have to use quasi-experiments. The latter term is often used when it is impossible to perform random assignment of the subjects to the different treatments. An example of an experiment in software engineering may be to compare two different methods for inspections. For this type of studies methods for statistical inference are applied with the purpose of showing with statistical significance that one method is better than the other […].

Wohlin, C., Runeson, P., Höst, M., Ohlsson, M., Regnell, B., & Wesslén, A. (2000). Introduction to Experimentation in Software Engineering. [-]