Regulatory experimentation case studies

A/B testing

A simplified form of a randomised control trial (RCT) where participants are randomly shown one of two versions (A or B) of a webpage, app or other product.

These tests often focus on optimising a single outcome. They can also be iterative: once version A wins, it may be compared with a new version C. This type of study is typically easy to implement, especially with digital tools and platforms that automate the process.

Before-and-after-study

A study that measures and compares outcomes before and after a treatment or intervention.

These studies are relatively simple to design and analyse. However, a key limitation is that results can be influenced by external factors (for example, changes in the weather across the study period might affect outcomes), and this cannot be distinguished from the effect of the intervention.

Discrete choice experiment (DCE)

A research method used to elicit preferences by asking participants to choose between sets of alternative options that have varying features.

This approach helps in understanding decision-making and the value placed on different features. While DCEs are not as resource-intensive as RCTs, they require careful design and sophisticated analysis to accurately interpret the trade-offs participants are willing to make.

Online experiment

A study conducted over the internet where participants complete tasks or respond to questions remotely. This allows researchers to reach a larger and more diverse group of participants rapidly, compared to traditional lab settings.

A common limitation of online experiments is reduced external validity - the extent to which findings can be generalised to real-world settings. This is because online experiments often do not reflect the real-world context of the behaviour being studied. However, when the behaviour of interest is performed online, this gap is reduced. External validity is less of an issue in these cases, especially when using a simulated online environment that closely mimics real-world settings.

Quasi-experiment

A research design that resembles an experiment but does not involve randomly assigning participants to different groups.

Due to the lack of randomisation, quasi-experiments are susceptible to various biases. One common bias is self-selection bias, where participants who choose to take part might be more motivated and therefore perform differently - either better or worse - based on their reasons for participating. These biases can complicate the establishment of cause-and-effect relationships.

Randomised control trial (RCT)

A study in which participants are randomly assigned to either a treatment group or a control group.

This randomisation allows researchers to measure the causal effect of the treatment by comparing outcomes between the groups. RCTs are considered the gold standard for determining causal relationships.