Cite this as: Strupler, N. 2021 Re-discovering Archaeological Discoveries. Experiments with reproducing archaeological survey analysis, Internet Archaeology 56. https://doi.org/10.11141/ia.56.6
This article describes an attempt to reproduce the published analysis from three archaeological field-walking surveys by using datasets collected between 1990 and 2005 which are publicly available in digital format. The exact methodologies used to produce the analyses (diagrams, statistical analysis, maps, etc.) are often incomplete, leaving a gap between the dataset and the published report. By using the published descriptions to reconstruct how the outputs were manipulated, I expected to reproduce and corroborate the results. While these experiments highlight some successes, they also point to significant problems in reproducing an analysis at various stages, from reading the data to plotting the results. Consequently, this article proposes some guidance on how to increase the reproducibility of data in order to assist aspirations of refining results or methodology. Without a stronger emphasis on reproducibility, the published datasets may not be sufficient to confirm published results and the scientific process of self-correction is at risk.
Corresponding author: Néhémie Strupler
McDonald Institute for Archaeological Research, Cambridge & Institut Français d'Études Anatoliennes, Istanbul.
Figure 1: An attempt to reproduce a figure from the Boeotia Survey book showing discrepancies in the ID numbers: Figure 1a: the reproduced map; Figure 1b: figure 1.4 from Bintliff et al. (2007). ID divergence can be seen at the top of the figure where transects 501 and 502 should be 177 and 178.
Figure 2: A second attempt to reproduce a figure from the Boeotia Survey book, which also reveals discrepancies in the ID numbers. Figure 2a: the reproduced map; Figure 2b: figure 1.6 from Bintliff et al. (2007).
Figure 3: Pottery density map generated by the author on the basis of the data. It shows random distribution that does not match the reported results which is showing clearly delimited sites. The legend (top right) presents the colour for each class as well as the associated interval, from low density in blue (between 0 and 214 sherds) to high density in red (between 8839 and 11288 sherds).
Figure 5: Plot of linear regression of 'Ground Visibility' and 'Adjusted Visibility' showing an almost perfect linear relation (Sydney Cyprus Survey Project).
Figure 6: Plot of the published 'Ground Visibility' and 'Adjusted Visibility' data, as well as a reproduced 'Adjusted Visibility' ('bgc' stands for 'background confusion') (Sydney Cyprus Survey Project).
Figure 8: Plot of the visibility percentage by units (Pyla-Koutsopetria Archaeological Project) as published in the book (Figure 8a) and as reproduced (Figure 8b). The two figures show a strong similarity, even if the shapes of the units are not identical (if some units are oriented to the North and have a regular shape, multiple units were adapted to the terrain and their individual shape can not be exactly emulated with the information provided).
Figure 9: Plot of the Late Bronze Age artefacts (Pyla-Koutsopetria Archaeological Project) as published in the book (Figure 9a) and as reproduced (Figure 9b). The two figures show that the units with points are the same but the number of (visible) points displayed (i.e. artefacts) differs considerably.
Table 2: Head of the tabular data of the units file (Sydney Cyprus Survey Project. Explanation of variable names is provided in the text files of the project archive.
Table 4: The reproduced analysis (Pyla-Koutsopetria Archaeological Project) showing the individual numbers as well as sums for each period, proving that the same subset of 205 sherds (7+6+19+173) is being used as in the monograph publication.
Baddeley, A., Ege R. and Rolf T. 2015 Spatial Point Patterns: Methodology and Applications with R, London: Chapman and Hall/CRC Press. https://doi.org/10.1201/b19708
Baddeley, A. and Turner, R. 2005 'spatstat: An R Package for analyzing spatial point patterns.' Journal of Statistical Software 12(6), 1–42. https://doi.org/10.18637/jss.v012.i06
Banning, E.B. 2002 Archaeological Survey, Manuals in Archaeological Method, Theory, and Technique, New York: Kluwer Academic. https://doi.org/10.1007/978-1-4615-0769-7
Bintliff, J.L., Howard, P. and Snodgrass, A. (eds) 2007 Testing the Hinterland: The Work of the Boeotia Survey (1989–1991) in the Southern Approaches to the City of Thespiai, Cambridge: McDonald Institute Monographs.
Bintliff, J.L., Farinetti, E., Slapšak, B. and Snodgrass, A. 2017 The City of Thespiai : Survey at a Complex Urban Site, Cambridge: McDonald Institute Monographs.
Bollen, K., Cacioppo, J.T., Kaplan, R., Krosnick, J. and Olds, J.L. 2015 Social, Behavioral, and Economic Sciences Perspectives on Robust and Reliable Science, Arlington, VA: National Science Foundation.
Bruner, J.P. and Holman, B. 2019 'Self-Correction in Science: Meta-Analysis, Bias and Social Structure', Studies in History and Philosophy of Science Part A. https://doi.org/10.1016/j.shpsa.2019.02.001
Caraher, W.R., Moore, R.S. and Pettegrew, D.K. 2013 Pyla-Koutsopetria Archaeological Project, Open Context. https://doi.org/10.6078/M7B56GNS
Caraher, W.R., Moore, R.S. and Pettegrew, D.K. 2014 Pyla-Koutsopetria I: Archaeological Survey of an Ancient Coastal Town, Boston: American Schools of Oriental Research. https://doi.org/10.5615/j.ctvj7wksw
Chang, A.C. and Li, P. 2015 'Is Economics Research Replicable? Sixty published papers from thirteen journals say 'Usually Not', Finance and Economics Discussion Series 2015-83, Board of Governors of the Federal Reserve System (U.S.). https://doi.org/10.17016/FEDS.2015.083 [PDF]
Dieudonné, N. 1989 'La Prospection Au Sol: Étude Bibliographique', Revue Archéologique Du Centre de La France 28(2), 217–28. https://doi.org/10.3406/racf.1989.2597
Ebersole, C.R., Axt, J.R. and Nosek, B.A. 2016 'Scientists' reputations are based on getting it right, not being right', PLOS Biology 14(5), 1–7. https://doi.org/10.1371/journal.pbio.1002460
Fidler, F. and Wilcox, J. 2018 'Reproducibility of Scientific Results' in E.N. Zalta (ed) The Stanford Encyclopedia of Philosophy, Winter 2018, Metaphysics Research Lab, Stanford University. https://plato.stanford.edu/archives/win2018/entries/scientific-reproducibility/
Given, M. and Knapp, A.B. (eds) 2003 The Sydney Cyprus Survey Project: Social Approaches to Regional Archaeological Survey, Los Angeles: Cotsen Institute of Archaeology.
Goodman, S.N., Fanelli, D. and Ioannidis, J.P.A. 2016 'What does research reproducibility mean?', Science Translational Medicine 8(341). https://doi.org/10.1126/scitranslmed.aaf5027
Haas, T. de and Leusen, M. van 2020 'FAIR Survey: Improving Documentation and Archiving Practices in Archaeological Field Survey Through Cidoc Crm', FOLD&R: The Journal of Fasti on-Line. http://www.fastionline.org/docs/FOLDER-sur-2020-12.pdf [PDF]
Ioannidis, J.P.A. 2005 'Why most published research findings are false', PLOS Medicine 2(8), e124. https://doi.org/10.1371/journal.pmed.0020124
Ioannidis, J.P.A. 2012 'Why Science is not necessarily self-correcting', Perspectives on Psychological Science 7(6), 645–54. https://doi.org/10.1177/1745691612464056
Jamieson, K.H. 2018 'Crisis or Self-Correction: Rethinking media narratives about the well-being of science', Proceedings of the National Academy of Sciences 115(11), 2620–7. https://doi.org/10.1073/pnas.1708276114
Jenks, G.F. 1977 'Optimal Data Classification for Choropleth Maps' in Department of Geography Occasional Paper No. 2, Kansas: University of Kansas.
Kansa, E.C., Kansa, S.W. and Arbuckle, B. 2014 'Publishing and Pushing: Mixing Models for Communicating Research Data in Archaeology', International Journal of Digital Curation 9(1), 57–70. https://doi.org/10.2218/ijdc.v9i1.301
Kelty, C.M. 2005 'Free Science' in J. Feller, B. Fitzgerald, S.A. Hissam and K.R. Lakhani (eds) Perspectives on Free and Open Source Software, Cambridge, MA: MIT Press. 415–30.
Knapp, A.B. and Given, M. (eds) 2003 Archive: The Sydney Cyprus Survey Project. York: Archaeology Data Service [distributor] https://doi.org/10.5284/1000208
Lee, C.J., Sugimoto, C.R., Zhang, G. and Cronin, B. 2013 'Bias in Peer Review', Journal of the American Society for Information Science and Technology 64(1), 2–17. https://doi.org/10.1002/asi.22784
Makel, M.C., Plucker, J.A. and Hegarty, B. 2012 'Replications in Psychology Research: How often do they really occur?' Perspectives on Psychological Science 7(6), 537–42. https://doi.org/10.1177/1745691612460688
Marwick, B. 2016 'Computational Reproducibility in Archaeological Research: Basic Principles and a Case Study of their implementation', Journal of Archaeological Method and Theory 24, 424–50. https://doi.org/10.1007/s10816-015-9272-9
Marwick, B., d'Alpoim Guedes, J., Barton, C.M., Bates, L.A. et al. 2017 'Open Science in Archaeology', SAA Archaeological Record 17(4), 8–14. https://faculty.washington.edu/bmarwick/PDFs/Marwick_et_al_2017_SAA_Record_Sept.pdf
Meyer N. and Gregory T. E. 2003 'Pottery Collection, Pottery Analysis, and GIS Mapping' in M. Given and A.B. Knapp (eds) The Sydney Cyprus Survey Project: Social Approaches to Regional Archaeological Survey, Los Angeles: Cotsen Institute of Archaeology. 48–52.
Niiniluoto, I. 2019 'Scientific Progress' in E.N. Zalta (ed) The Stanford Encyclopedia of Philosophy, Winter 2019, Metaphysics Research Lab, Stanford University. https://plato.stanford.edu/archives/win2019/entries/scientific-progress/
Noorden, R. Van. 2014 'The scientists who get credit for peer review', Nature, 9 October 2014. https://doi.org/10.1038/nature.2014.16102
Nosek, B.A., Spies, J.R. and Motyl, M. 2012 'Scientific Utopia: II. Restructuring incentives and practices to promote truth over publishability', Perspectives on Psychological Science 7(6), 615–31. https://doi.org/10.1177/1745691612459058
Nuzzo, R. 2015 'How scientists fool themselves – and how they can stop', Nature 526, 182–5. https://doi.org/10.1038/526182a
Peng, R.D. 2011 'Reproducible Research in Computational Science', Science 334(6060), 1226–7. https://doi.org/10.1126/science.1213847
Pettegrew, D.K. 2010 'Book review of Testing the Hinterland: The Work of the Boeotia Survey (1989–1991) in the Southern Approaches to the City of Thespiai by John Bintliff, Phil Howard, and Anthony Snodgrass', American Journal of Archaeology 114(1). https://doi.org/10.3764/ajaonline114.1.Pettegrew
R Core Team 2020. R: A Language and Environment for Statistical Computing, Wien: R Foundation for Statistical Computing. https://www.R-project.org
Resnik, D.B. and Shamoo, A.E. 2016 'Reproducibility and Research Integrity', Accountability in Research 24(2), 116–23. https://doi.org/10.1080/08989621.2016.1257387
Sarabipour, S., Debat, H.J., Emmott, E., Burgess, S.J. et al. 2019 'On the value of preprints: an Early Career Researcher Perspective', PLOS Biology 17(2), 1–12. https://doi.org/10.1371/journal.pbio.3000151
Schmidt, S. 2009 'Shall we really do it again? The powerful concept of Replication is neglected in the Social Sciences', Review of General Psychology 13(2), 90–100. https://doi.org/10.1037/a0015108
Silberzahn, R. and Uhlmann, E.L. 2015 'Crowdsourced Research: Many hands make tight work', Nature 526, 189–91. https://doi.org/10.1038/526189a
Strupler, N. 2018 Project Panormos Archaeological Survey: Satellite Image (gis-copernicus). https://doi.org/10.5281/zenodo.1185044
Strupler, N. and Wilkinson, T.C. 2017 'Reproducibility in the Field: Transparency, Version Control and Collaboration on the Project Panormos Survey', Open Archaeology 3(1), 279–304. https://doi.org/10.1515/opar-2017-0019
Vazire, S. 2018 'Implications of the Credibility Revolution for productivity, creativity, and progress', Perspectives on Psychological Science 13(4), 411–17. https://doi.org/10.1177/1745691617751884
Wilensky, U. and Rand, W. 2007 'Making models match: replicating an agent-based model', Journal of Artificial Societies and Social Simulation 10(4, 2). http://jasss.soc.surrey.ac.uk/10/4/2.html
Wilkinson, M.D., Dumontier, W., Aalbersberg, I., Appleton, G. et al. 2016 'The FAIR Guiding Principles for Scientific Data Management and Stewardship', Scientific Data 3(1), 160018. https://doi.org/10.1038/sdata.2016.18
Zwaan, R.A., Etz, A., Lucas, R.E. and Donnellan, M. 2018 'Making replication mainstream', Behavioral and Brain Sciences 41, e120. https://doi.org/10.1017/S0140525X17001972
Internet Archaeology is an open access journal based in the Department of Archaeology, University of York. Except where otherwise noted, content from this work may be used under the terms of the Creative Commons Attribution 3.0 (CC BY) Unported licence, which permits unrestricted use, distribution, and reproduction in any medium, provided that attribution to the author(s), the title of the work, the Internet Archaeology journal and the relevant URL/DOI are given.
Internet Archaeology content is preserved for the long term with the Archaeology Data Service. Help sustain and support open access publication by donating to our Open Access Archaeology Fund.