Flexible replication

A recent study (1) highlights the difficulty in replicating scientific experiments to prove the initial conclusions in the presence of large amounts of data, changing environments and technologies. This is fair enough but these conditions also point to the flexibility afforded to scientists in experiments. Industries with high information content created by experiments in the presence of uncertainty, such as pharmaceuticals, show significant bias toward confirmation of earlier results in repeated experiments. Replication, thus, has become easier and not harder with large amounts of data and improving technologies to collect and store data. In systems with high uncertainty and non-linearity, where conclusions are reached by statistical testing, the growth of data in repeated experiments have allowed a higher flexibility in replication.

Thus, replication is not necessarily challenging when data grow and conditions change. In many cases, these make replication easier. At the limit, then, one has to question if experimentation is a good way to reach conclusions. As has been shown many times in the past, statisticians, given sufficient time, can prove anything. Experiment designers, given sufficient time and money, can also replicate anything. This is a significant problem for the advancement of science and knowledge. Most educational systems around the world are training the next generation of scientists on how to collect and analyze data according to traditional principles. It is, however, unclear, that traditional principles are good enough to reach conclusions and to create insights.

It is not the difficulty in replicating scientific experiments we have to worry about. It is the flexibility afforded by data and technology to prove anything that is more troublesome.

(1) Again, and Again, and Again …

Science 2 December 2011: Vol. 334 no. 6060 p. 1225
DOI: 10.1126/science.334.6060.1225 Barbara R. Jasny, Gilbert Chin, Lisa Chong, Sacha Vignieri