Mini journal logo  Home Issue Contents All Issues

Re-discovering Archaeological Discoveries. Experiments with reproducing archaeological survey analysisOpen Materials

Néhémie Strupler

Cite this as: Strupler, N. 2021 Re-discovering Archaeological Discoveries. Experiments with reproducing archaeological survey analysis, Internet Archaeology 56.


This article describes an attempt to reproduce the published analysis from three archaeological field-walking surveys by using datasets collected between 1990 and 2005 which are publicly available in digital format. The exact methodologies used to produce the analyses (diagrams, statistical analysis, maps, etc.) are often incomplete, leaving a gap between the dataset and the published report. By using the published descriptions to reconstruct how the outputs were manipulated, I expected to reproduce and corroborate the results. While these experiments highlight some successes, they also point to significant problems in reproducing an analysis at various stages, from reading the data to plotting the results. Consequently, this article proposes some guidance on how to increase the reproducibility of data in order to assist aspirations of refining results or methodology. Without a stronger emphasis on reproducibility, the published datasets may not be sufficient to confirm published results and the scientific process of self-correction is at risk.

  • Google Scholar
  • Keywords: archaeology, data, reuse, reproducibility, methodology, reanalysis
  • Accepted: 18 March 2021. Published: 14 June 2021
  • Funding: The publication costs of this article were paid by the Swiss National Science Foundation (SNSF).
  • PDF download (main article text only)

Corresponding author: Néhémie StruplerORCID logo
McDonald Institute for Archaeological Research, Cambridge & Institut Français d'Études Anatoliennes, Istanbul.

Full text

Figure 1: An attempt to reproduce a figure from the Boeotia Survey book showing discrepancies in the ID numbers: Figure 1a: the reproduced map; Figure 1b: figure 1.4 from Bintliff et al. (2007). ID divergence can be seen at the top of the figure where transects 501 and 502 should be 177 and 178.

Figure 2: A second attempt to reproduce a figure from the Boeotia Survey book, which also reveals discrepancies in the ID numbers. Figure 2a: the reproduced map; Figure 2b: figure 1.6 from Bintliff et al. (2007).

Figure 3: Pottery density map generated by the author on the basis of the data. It shows random distribution that does not match the reported results which is showing clearly delimited sites. The legend (top right) presents the colour for each class as well as the associated interval, from low density in blue (between 0 and 214 sherds) to high density in red (between 8839 and 11288 sherds).

Figure 4: Map of the sherd density per unit (Sydney Cyprus Survey Project) on a Raster background derived from Copernicus data (see Strupler 2018).

Figure 5: Plot of linear regression of 'Ground Visibility' and 'Adjusted Visibility' showing an almost perfect linear relation (Sydney Cyprus Survey Project).

Figure 6: Plot of the published 'Ground Visibility' and 'Adjusted Visibility' data, as well as a reproduced 'Adjusted Visibility' ('bgc' stands for 'background confusion') (Sydney Cyprus Survey Project).

Figure 7: Plot of the adjusted pottery count (Sydney Cyprus Survey Project). Figure 7a: as published; Figure 7b: as reproduced.

Figure 8: Plot of the visibility percentage by units (Pyla-Koutsopetria Archaeological Project) as published in the book (Figure 8a) and as reproduced (Figure 8b). The two figures show a strong similarity, even if the shapes of the units are not identical (if some units are oriented to the North and have a regular shape, multiple units were adapted to the terrain and their individual shape can not be exactly emulated with the information provided).

Figure 9: Plot of the Late Bronze Age artefacts (Pyla-Koutsopetria Archaeological Project) as published in the book (Figure 9a) and as reproduced (Figure 9b). The two figures show that the units with points are the same but the number of (visible) points displayed (i.e. artefacts) differs considerably.

Table 1: Screenshot of the tabular data from the CD-ROM in Bintliff et al. (2007).

Table 2: Head of the tabular data of the units file (Sydney Cyprus Survey Project. Explanation of variable names is provided in the text files of the project archive.

Table 3: Table published in the monograph (Caraher et al. 2014, 203). It is divided into four main periods, each subdivided into three or four chronotypes.

Table 4: The reproduced analysis (Pyla-Koutsopetria Archaeological Project) showing the individual numbers as well as sums for each period, proving that the same subset of 205 sherds (7+6+19+173) is being used as in the monograph publication.

Baddeley, A., Ege R. and Rolf T. 2015 Spatial Point Patterns: Methodology and Applications with R, London: Chapman and Hall/CRC Press.

Baddeley, A. and Turner, R. 2005 'spatstat: An R Package for analyzing spatial point patterns.' Journal of Statistical Software 12(6), 1–42.

Banning, E.B. 2002 Archaeological Survey, Manuals in Archaeological Method, Theory, and Technique, New York: Kluwer Academic.

Bintliff, J.L., Howard, P. and Snodgrass, A. (eds) 2007 Testing the Hinterland: The Work of the Boeotia Survey (1989–1991) in the Southern Approaches to the City of Thespiai, Cambridge: McDonald Institute Monographs.

Bintliff, J.L., Farinetti, E., Slapšak, B. and Snodgrass, A. 2017 The City of Thespiai : Survey at a Complex Urban Site, Cambridge: McDonald Institute Monographs.

Bollen, K., Cacioppo, J.T., Kaplan, R., Krosnick, J. and Olds, J.L. 2015 Social, Behavioral, and Economic Sciences Perspectives on Robust and Reliable Science, Arlington, VA: National Science Foundation.

Bruner, J.P. and Holman, B. 2019 'Self-Correction in Science: Meta-Analysis, Bias and Social Structure', Studies in History and Philosophy of Science Part A.

Caraher, W.R., Moore, R.S. and Pettegrew, D.K. 2013 Pyla-Koutsopetria Archaeological Project, Open Context.

Caraher, W.R., Moore, R.S. and Pettegrew, D.K. 2014 Pyla-Koutsopetria I: Archaeological Survey of an Ancient Coastal Town, Boston: American Schools of Oriental Research.

Chang, A.C. and Li, P. 2015 'Is Economics Research Replicable? Sixty published papers from thirteen journals say 'Usually Not', Finance and Economics Discussion Series 2015-83, Board of Governors of the Federal Reserve System (U.S.). [PDF]

Dieudonné, N. 1989 'La Prospection Au Sol: Étude Bibliographique', Revue Archéologique Du Centre de La France 28(2), 217–28.

Ebersole, C.R., Axt, J.R. and Nosek, B.A. 2016 'Scientists' reputations are based on getting it right, not being right', PLOS Biology 14(5), 1–7.

Fidler, F. and Wilcox, J. 2018 'Reproducibility of Scientific Results' in E.N. Zalta (ed) The Stanford Encyclopedia of Philosophy, Winter 2018, Metaphysics Research Lab, Stanford University.

Given, M. and Knapp, A.B. (eds) 2003 The Sydney Cyprus Survey Project: Social Approaches to Regional Archaeological Survey, Los Angeles: Cotsen Institute of Archaeology.

Goodman, S.N., Fanelli, D. and Ioannidis, J.P.A. 2016 'What does research reproducibility mean?', Science Translational Medicine 8(341).

Haas, T. de and Leusen, M. van 2020 'FAIR Survey: Improving Documentation and Archiving Practices in Archaeological Field Survey Through Cidoc Crm', FOLD&R: The Journal of Fasti on-Line. [PDF]

Ioannidis, J.P.A. 2005 'Why most published research findings are false', PLOS Medicine 2(8), e124.

Ioannidis, J.P.A. 2012 'Why Science is not necessarily self-correcting', Perspectives on Psychological Science 7(6), 645–54.

Jamieson, K.H. 2018 'Crisis or Self-Correction: Rethinking media narratives about the well-being of science', Proceedings of the National Academy of Sciences 115(11), 2620–7.

Jenks, G.F. 1977 'Optimal Data Classification for Choropleth Maps' in Department of Geography Occasional Paper No. 2, Kansas: University of Kansas.

Kansa, E.C., Kansa, S.W. and Arbuckle, B. 2014 'Publishing and Pushing: Mixing Models for Communicating Research Data in Archaeology', International Journal of Digital Curation 9(1), 57–70.

Kelty, C.M. 2005 'Free Science' in J. Feller, B. Fitzgerald, S.A. Hissam and K.R. Lakhani (eds) Perspectives on Free and Open Source Software, Cambridge, MA: MIT Press. 415–30.

Knapp, A.B. and Given, M. (eds) 2003 Archive: The Sydney Cyprus Survey Project. York: Archaeology Data Service [distributor]

Lee, C.J., Sugimoto, C.R., Zhang, G. and Cronin, B. 2013 'Bias in Peer Review', Journal of the American Society for Information Science and Technology 64(1), 2–17.

Makel, M.C., Plucker, J.A. and Hegarty, B. 2012 'Replications in Psychology Research: How often do they really occur?' Perspectives on Psychological Science 7(6), 537–42.

Marwick, B. 2016 'Computational Reproducibility in Archaeological Research: Basic Principles and a Case Study of their implementation', Journal of Archaeological Method and Theory 24, 424–50.

Marwick, B., d'Alpoim Guedes, J., Barton, C.M., Bates, L.A. et al. 2017 'Open Science in Archaeology', SAA Archaeological Record 17(4), 8–14.

Meyer N. and Gregory T. E. 2003 'Pottery Collection, Pottery Analysis, and GIS Mapping' in M. Given and A.B. Knapp (eds) The Sydney Cyprus Survey Project: Social Approaches to Regional Archaeological Survey, Los Angeles: Cotsen Institute of Archaeology. 48–52.

Niiniluoto, I. 2019 'Scientific Progress' in E.N. Zalta (ed) The Stanford Encyclopedia of Philosophy, Winter 2019, Metaphysics Research Lab, Stanford University.

Noorden, R. Van. 2014 'The scientists who get credit for peer review', Nature, 9 October 2014.

Nosek, B.A., Spies, J.R. and Motyl, M. 2012 'Scientific Utopia: II. Restructuring incentives and practices to promote truth over publishability', Perspectives on Psychological Science 7(6), 615–31.

Nuzzo, R. 2015 'How scientists fool themselves – and how they can stop', Nature 526, 182–5.

Peng, R.D. 2011 'Reproducible Research in Computational Science', Science 334(6060), 1226–7.

Pettegrew, D.K. 2010 'Book review of Testing the Hinterland: The Work of the Boeotia Survey (1989–1991) in the Southern Approaches to the City of Thespiai by John Bintliff, Phil Howard, and Anthony Snodgrass', American Journal of Archaeology 114(1).

R Core Team 2020. R: A Language and Environment for Statistical Computing, Wien: R Foundation for Statistical Computing.

Resnik, D.B. and Shamoo, A.E. 2016 'Reproducibility and Research Integrity', Accountability in Research 24(2), 116–23.

Sarabipour, S., Debat, H.J., Emmott, E., Burgess, S.J. et al. 2019 'On the value of preprints: an Early Career Researcher Perspective', PLOS Biology 17(2), 1–12.

Schmidt, S. 2009 'Shall we really do it again? The powerful concept of Replication is neglected in the Social Sciences', Review of General Psychology 13(2), 90–100.

Silberzahn, R. and Uhlmann, E.L. 2015 'Crowdsourced Research: Many hands make tight work', Nature 526, 189–91.

Strupler, N. 2018 Project Panormos Archaeological Survey: Satellite Image (gis-copernicus).

Strupler, N. and Wilkinson, T.C. 2017 'Reproducibility in the Field: Transparency, Version Control and Collaboration on the Project Panormos Survey', Open Archaeology 3(1), 279–304.

Vazire, S. 2018 'Implications of the Credibility Revolution for productivity, creativity, and progress', Perspectives on Psychological Science 13(4), 411–17.

Wilensky, U. and Rand, W. 2007 'Making models match: replicating an agent-based model', Journal of Artificial Societies and Social Simulation 10(4, 2).

Wilkinson, M.D., Dumontier, W., Aalbersberg, I., Appleton, G. et al. 2016 'The FAIR Guiding Principles for Scientific Data Management and Stewardship', Scientific Data 3(1), 160018.

Zwaan, R.A., Etz, A., Lucas, R.E. and Donnellan, M. 2018 'Making replication mainstream', Behavioral and Brain Sciences 41, e120.

Internet Archaeology is an open access journal based in the Department of Archaeology, University of York. Except where otherwise noted, content from this work may be used under the terms of the Creative Commons Attribution 3.0 (CC BY) Unported licence, which permits unrestricted use, distribution, and reproduction in any medium, provided that attribution to the author(s), the title of the work, the Internet Archaeology journal and the relevant URL/DOI are given.

Terms and Conditions | Legal Statements | Privacy Policy | Cookies Policy | Citing Internet Archaeology

Internet Archaeology content is preserved for the long term with the Archaeology Data Service. Help sustain and support open access publication by donating to our Open Access Archaeology Fund.