Re-discovering Archaeological Discoveries. Experiments with reproducing archaeological survey analysis

• Disclaimer
• 1. Introduction
• 2. The Boeotia Survey
• 3. The Sydney Cyprus Survey Project (SCSP)
  • 3.1 'Ground' and 'Adjusted' visibility
  • 3.2 Adjusted pottery count
  • 3.3 SCSP summary
• 4. The Pyla-Koutsopetria Archaeological Project (PKAP)
  • 4.1 Reading and shaping the data
  • 4.2 Format of units data
  • 4.3 The main finds dataset
  • 4.4 General remarks on the PKAP dataset
• 5. Discussion: Values and Prospects of Reproducibility in Archaeology
  • 5.1 Promoting reproducibility as a valuable scientific practice
  • 5.2 Promoting reproducibility returns for reproducers
  • 5.3 Providing guidelines: how to report an error
    • 5.3.1 Publish, push, reproduce and update?
    • 5.3.2 Good communication is key
    • 5.3.3 Full disclosure versus coordinate disclosure
  • 5.4 Clarifying responsibility
• 6. Closing remarks
• Appendix: Code used to reproduce the analyses
  • The Boeotia Project
  • The Sydney Cyprus Survey Project
  • The Pyla-Koutsopetria Archaeological Project
• Acknowledgements
• Glossary
• Footnotes

Figure 1: An attempt to reproduce a figure from the Boeotia Survey book showing discrepancies in the ID numbers: Figure 1a: the reproduced map; Figure 1b: figure 1.4 from Bintliff et al. (2007). ID divergence can be seen at the top of the figure where transects 501 and 502 should be 177 and 178.

Figure 2: A second attempt to reproduce a figure from the Boeotia Survey book, which also reveals discrepancies in the ID numbers. Figure 2a: the reproduced map; Figure 2b: figure 1.6 from Bintliff et al. (2007).

Figure 3: Pottery density map generated by the author on the basis of the data. It shows random distribution that does not match the reported results which is showing clearly delimited sites. The legend (top right) presents the colour for each class as well as the associated interval, from low density in blue (between 0 and 214 sherds) to high density in red (between 8839 and 11288 sherds).

Figure 4: Map of the sherd density per unit (Sydney Cyprus Survey Project) on a Raster background derived from Copernicus data (see Strupler 2018).

Figure 5: Plot of linear regression of 'Ground Visibility' and 'Adjusted Visibility' showing an almost perfect linear relation (Sydney Cyprus Survey Project).

Figure 6: Plot of the published 'Ground Visibility' and 'Adjusted Visibility' data, as well as a reproduced 'Adjusted Visibility' ('bgc' stands for 'background confusion') (Sydney Cyprus Survey Project).

Figure 7: Plot of the adjusted pottery count (Sydney Cyprus Survey Project). Figure 7a: as published; Figure 7b: as reproduced.

Figure 8: Plot of the visibility percentage by units (Pyla-Koutsopetria Archaeological Project) as published in the book (Figure 8a) and as reproduced (Figure 8b). The two figures show a strong similarity, even if the shapes of the units are not identical (if some units are oriented to the North and have a regular shape, multiple units were adapted to the terrain and their individual shape can not be exactly emulated with the information provided).

Figure 9: Plot of the Late Bronze Age artefacts (Pyla-Koutsopetria Archaeological Project) as published in the book (Figure 9a) and as reproduced (Figure 9b). The two figures show that the units with points are the same but the number of (visible) points displayed (i.e. artefacts) differs considerably.

Table 1: Screenshot of the tabular data from the CD-ROM in Bintliff et al. (2007).

Table 2: Head of the tabular data of the units file (Sydney Cyprus Survey Project. Explanation of variable names is provided in the text files of the project archive.

Table 3: Table published in the monograph (Caraher et al. 2014, 203). It is divided into four main periods, each subdivided into three or four chronotypes.

Table 4: The reproduced analysis (Pyla-Koutsopetria Archaeological Project) showing the individual numbers as well as sums for each period, proving that the same subset of 205 sherds (7+6+19+173) is being used as in the monograph publication.

Baddeley, A., Ege R. and Rolf T. 2015 Spatial Point Patterns: Methodology and Applications with R, London: Chapman and Hall/CRC Press. https://doi.org/10.1201/b19708

Baddeley, A. and Turner, R. 2005 'spatstat: An R Package for analyzing spatial point patterns.' Journal of Statistical Software 12(6), 1–42. https://doi.org/10.18637/jss.v012.i06

Banning, E.B. 2002 Archaeological Survey, Manuals in Archaeological Method, Theory, and Technique, New York: Kluwer Academic. https://doi.org/10.1007/978-1-4615-0769-7

Bintliff, J.L., Howard, P. and Snodgrass, A. (eds) 2007 Testing the Hinterland: The Work of the Boeotia Survey (1989–1991) in the Southern Approaches to the City of Thespiai, Cambridge: McDonald Institute Monographs.

Bintliff, J.L., Farinetti, E., Slapšak, B. and Snodgrass, A. 2017 The City of Thespiai : Survey at a Complex Urban Site, Cambridge: McDonald Institute Monographs.

Bollen, K., Cacioppo, J.T., Kaplan, R., Krosnick, J. and Olds, J.L. 2015 Social, Behavioral, and Economic Sciences Perspectives on Robust and Reliable Science, Arlington, VA: National Science Foundation.

Bruner, J.P. and Holman, B. 2019 'Self-Correction in Science: Meta-Analysis, Bias and Social Structure', Studies in History and Philosophy of Science Part A. https://doi.org/10.1016/j.shpsa.2019.02.001

Caraher, W.R., Moore, R.S. and Pettegrew, D.K. 2013 Pyla-Koutsopetria Archaeological Project, Open Context. https://doi.org/10.6078/M7B56GNS

Caraher, W.R., Moore, R.S. and Pettegrew, D.K. 2014 Pyla-Koutsopetria I: Archaeological Survey of an Ancient Coastal Town, Boston: American Schools of Oriental Research. https://doi.org/10.5615/j.ctvj7wksw

Chang, A.C. and Li, P. 2015 'Is Economics Research Replicable? Sixty published papers from thirteen journals say 'Usually Not', Finance and Economics Discussion Series 2015-83, Board of Governors of the Federal Reserve System (U.S.). https://doi.org/10.17016/FEDS.2015.083 [PDF]

Dieudonné, N. 1989 'La Prospection Au Sol: Étude Bibliographique', Revue Archéologique Du Centre de La France 28(2), 217–28. https://doi.org/10.3406/racf.1989.2597

Ebersole, C.R., Axt, J.R. and Nosek, B.A. 2016 'Scientists' reputations are based on getting it right, not being right', PLOS Biology 14(5), 1–7. https://doi.org/10.1371/journal.pbio.1002460

Fidler, F. and Wilcox, J. 2018 'Reproducibility of Scientific Results' in E.N. Zalta (ed) The Stanford Encyclopedia of Philosophy, Winter 2018, Metaphysics Research Lab, Stanford University. https://plato.stanford.edu/archives/win2018/entries/scientific-reproducibility/

Given, M. and Knapp, A.B. (eds) 2003 The Sydney Cyprus Survey Project: Social Approaches to Regional Archaeological Survey, Los Angeles: Cotsen Institute of Archaeology.

Goodman, S.N., Fanelli, D. and Ioannidis, J.P.A. 2016 'What does research reproducibility mean?', Science Translational Medicine 8(341). https://doi.org/10.1126/scitranslmed.aaf5027

Haas, T. de and Leusen, M. van 2020 'FAIR Survey: Improving Documentation and Archiving Practices in Archaeological Field Survey Through Cidoc Crm', FOLD&R: The Journal of Fasti on-Line. http://www.fastionline.org/docs/FOLDER-sur-2020-12.pdf [PDF]

Ioannidis, J.P.A. 2005 'Why most published research findings are false', PLOS Medicine 2(8), e124. https://doi.org/10.1371/journal.pmed.0020124

Ioannidis, J.P.A. 2012 'Why Science is not necessarily self-correcting', Perspectives on Psychological Science 7(6), 645–54. https://doi.org/10.1177/1745691612464056

Jamieson, K.H. 2018 'Crisis or Self-Correction: Rethinking media narratives about the well-being of science', Proceedings of the National Academy of Sciences 115(11), 2620–7. https://doi.org/10.1073/pnas.1708276114

Jenks, G.F. 1977 'Optimal Data Classification for Choropleth Maps' in Department of Geography Occasional Paper No. 2, Kansas: University of Kansas.

Kansa, E.C., Kansa, S.W. and Arbuckle, B. 2014 'Publishing and Pushing: Mixing Models for Communicating Research Data in Archaeology', International Journal of Digital Curation 9(1), 57–70. https://doi.org/10.2218/ijdc.v9i1.301

Kelty, C.M. 2005 'Free Science' in J. Feller, B. Fitzgerald, S.A. Hissam and K.R. Lakhani (eds) Perspectives on Free and Open Source Software, Cambridge, MA: MIT Press. 415–30.

Knapp, A.B. and Given, M. (eds) 2003 Archive: The Sydney Cyprus Survey Project. York: Archaeology Data Service [distributor] https://doi.org/10.5284/1000208

Lee, C.J., Sugimoto, C.R., Zhang, G. and Cronin, B. 2013 'Bias in Peer Review', Journal of the American Society for Information Science and Technology 64(1), 2–17. https://doi.org/10.1002/asi.22784

Makel, M.C., Plucker, J.A. and Hegarty, B. 2012 'Replications in Psychology Research: How often do they really occur?' Perspectives on Psychological Science 7(6), 537–42. https://doi.org/10.1177/1745691612460688

Marwick, B. 2016 'Computational Reproducibility in Archaeological Research: Basic Principles and a Case Study of their implementation', Journal of Archaeological Method and Theory 24, 424–50. https://doi.org/10.1007/s10816-015-9272-9

Marwick, B., d'Alpoim Guedes, J., Barton, C.M., Bates, L.A. et al. 2017 'Open Science in Archaeology', SAA Archaeological Record 17(4), 8–14. https://faculty.washington.edu/bmarwick/PDFs/Marwick_et_al_2017_SAA_Record_Sept.pdf

Meyer N. and Gregory T. E. 2003 'Pottery Collection, Pottery Analysis, and GIS Mapping' in M. Given and A.B. Knapp (eds) The Sydney Cyprus Survey Project: Social Approaches to Regional Archaeological Survey, Los Angeles: Cotsen Institute of Archaeology. 48–52.

Niiniluoto, I. 2019 'Scientific Progress' in E.N. Zalta (ed) The Stanford Encyclopedia of Philosophy, Winter 2019, Metaphysics Research Lab, Stanford University. https://plato.stanford.edu/archives/win2019/entries/scientific-progress/

Noorden, R. Van. 2014 'The scientists who get credit for peer review', Nature, 9 October 2014. https://doi.org/10.1038/nature.2014.16102

Nosek, B.A., Spies, J.R. and Motyl, M. 2012 'Scientific Utopia: II. Restructuring incentives and practices to promote truth over publishability', Perspectives on Psychological Science 7(6), 615–31. https://doi.org/10.1177/1745691612459058

Nuzzo, R. 2015 'How scientists fool themselves – and how they can stop', Nature 526, 182–5. https://doi.org/10.1038/526182a

Peng, R.D. 2011 'Reproducible Research in Computational Science', Science 334(6060), 1226–7. https://doi.org/10.1126/science.1213847

Pettegrew, D.K. 2010 'Book review of Testing the Hinterland: The Work of the Boeotia Survey (1989–1991) in the Southern Approaches to the City of Thespiai by John Bintliff, Phil Howard, and Anthony Snodgrass', American Journal of Archaeology 114(1). https://doi.org/10.3764/ajaonline114.1.Pettegrew

R Core Team 2020. R: A Language and Environment for Statistical Computing, Wien: R Foundation for Statistical Computing. https://www.R-project.org

Resnik, D.B. and Shamoo, A.E. 2016 'Reproducibility and Research Integrity', Accountability in Research 24(2), 116–23. https://doi.org/10.1080/08989621.2016.1257387

Sarabipour, S., Debat, H.J., Emmott, E., Burgess, S.J. et al. 2019 'On the value of preprints: an Early Career Researcher Perspective', PLOS Biology 17(2), 1–12. https://doi.org/10.1371/journal.pbio.3000151

Schmidt, S. 2009 'Shall we really do it again? The powerful concept of Replication is neglected in the Social Sciences', Review of General Psychology 13(2), 90–100. https://doi.org/10.1037/a0015108

Silberzahn, R. and Uhlmann, E.L. 2015 'Crowdsourced Research: Many hands make tight work', Nature 526, 189–91. https://doi.org/10.1038/526189a

Strupler, N. 2018 Project Panormos Archaeological Survey: Satellite Image (gis-copernicus). https://doi.org/10.5281/zenodo.1185044

Strupler, N. and Wilkinson, T.C. 2017 'Reproducibility in the Field: Transparency, Version Control and Collaboration on the Project Panormos Survey', Open Archaeology 3(1), 279–304. https://doi.org/10.1515/opar-2017-0019

Vazire, S. 2018 'Implications of the Credibility Revolution for productivity, creativity, and progress', Perspectives on Psychological Science 13(4), 411–17. https://doi.org/10.1177/1745691617751884

Wilensky, U. and Rand, W. 2007 'Making models match: replicating an agent-based model', Journal of Artificial Societies and Social Simulation 10(4, 2). http://jasss.soc.surrey.ac.uk/10/4/2.html

Wilkinson, M.D., Dumontier, W., Aalbersberg, I., Appleton, G. et al. 2016 'The FAIR Guiding Principles for Scientific Data Management and Stewardship', Scientific Data 3(1), 160018. https://doi.org/10.1038/sdata.2016.18

Zwaan, R.A., Etz, A., Lucas, R.E. and Donnellan, M. 2018 'Making replication mainstream', Behavioral and Brain Sciences 41, e120. https://doi.org/10.1017/S0140525X17001972