Psychology Research and Applications

Download PDF (182.7 KB) PP. 29 - 40 Pub. Date: June 30, 2021

DOI: 10.22606/pra.2021.32001

• Walter R. Schumm^*
  College of Health and Human Sciences, Kansas State University, Justin Hall, Manhattan, Kansas, USA
• Duane W. Crawford
  College of Health and Human Sciences, Kansas State University, Justin Hall, Manhattan, Kansas, USA
• Lorenza Lockett
  College of Health and Human Sciences, Kansas State University, Justin Hall, Manhattan, Kansas, USA
• Abdullah AlRashed
  College of Health and Human Sciences, Kansas State University, Justin Hall, Manhattan, Kansas, USA
• Asma bin Ateeq
  College of Health and Human Sciences, Kansas State University, Justin Hall, Manhattan, Kansas, USA

Some scientists have fabricated their data, yet have published their fake results in peer-reviewed journals. How can we detect patterns typical of fabricated research? Nine relatively less complex ways for detecting potentially fabricated data in small samples (N < 200), are presented, using data from articles published since 1999 as illustrations. Even with smaller samples, there are several ways in which scholars, as well as their undergraduate and graduate students, can detect possible fabrication of data as well as other questionable research practices (QRPs). However, with larger samples, other techniques may be needed.

research integrity, fraud, research misconduct, anomalous results, methods for detecting fraudulent research

[1] Appelbaum, M., Cooper, H., Kline, R. B., Mayo-Wilson, E., Nezu, A. M., & Rao, S. M. (2018). Journal article reporting standards for quantitative research in psychology: The APA Publications and Communications Board Task Force Report. American Psychologist, 73(1), 3-25.

[2] Brainard, J., You, J., & Bonazzi, D. (2018). Rethinking retractions. Science, 362(6413), 390-393. Brown, A. W., Kaiser, K. A., & Allison, D. B. (2018). Issues with data and analyses: Errors, underlying themes, and potential solutions. PNAS, 115(11), 2563-2570.

[3] Brown, N. J. L., & Heathers, J. A. J. (2017). The GRIM test: A simple technique detects Numerous anomalies in the reporting of results in psychology. Social Psychological and Personality Research, 8(4), 363-369.

[4] Brown, N. J. L., & Heathers, J. A. J. (2019). Rounded input variables, exact test statistics (RIVETS): A technique for detecting hand calculated results in published research. Unpublished paper, Bouve College of Health Sciences, Northeastern University, 360 Huntington Avenue, Boston, MA 02115.

[5] Bouter, L. M., & Hendrix, S. (2017). Both whistleblowers and the scientists they accuse are vulnerable and deserve protection. Accountability in Research, 24(6), 359-366.

[6] Cohen, J. (1992a). A power primer. Psychological Bulletin, 112, 155-159.

[7] Cohen, J. (1992b). Statistical power analysis. Current Directions in Psychological Science, 1, 98-101.

[8] Crocker, J., & Cooper, M. L. (2011). Addressing scientific fraud. Science, 334(6060), 1182. doi: 10.1126/science.1216775.

[9] Fanelli, Daniele. (2009). How many scientists fabricate and falsify research? A systematic review and meta-analysis of survey data. PloS One, 4, e5738.

[10] Fanelli, D. (2013). Why growing retractions are (mostly) a good sign. PLoS Medicine, 10(12), e1001563. doi: 10.1371/journal.pmed.1001563.

[11] Gartrell, N., Bos, H., & Koh, A. (2018). National Longitudinal Lesbian Family Study – Mental health of adult offspring. New England Journal of Medicine, 379(3), 297-299.

[12] Hartgerink, C. H. J., Voelkel, J. G., Wicherts, J. M., van Assen, M. A. L. M. (2019). Detection of data fabrication using statistical tools. Unpublished report. Tilburg, Netherlands: Tilburg University.

[13] Hartgerink, C. H. J., & Wicherts, J. M. (2016). Research practices and assessment of research misconduct. ScienceOpenResearch. Doi: 10.14293/S2199-1006.1.SOR-SOCSCI.ARYSBI,v1.

[14] Hartgerink, C. H. J., Wicherts, J. M., & van Assen, M. A. L. M. (2016). The value of statistical tools to detect data fabrication. Research Ideas and Outcomes, 2, e8860, 1-17.

[15] Heathers, J. A. J., Anaya, J., van der Zee, T., & Brown, N. J. L. (2018). Recovering data from summary statistics: Sample parameter reconstruction via iterative techniques (SPRITE). Peerj Preprints. doi.org/10.7287/peerj.preprints.26968v1.

[16] Heathers, J. A. J., & Brown N. J. L. (2019) DEBIT: A simple consistency test for binary data. Unpublished paper, Bouve College of Health Sciences, Northeastern University, 360 Huntington Avenue, Boston, MA 02115.

[17] Koh, A., Bos, H. M. W., & Gartrell, N. K. (2019). Predictors of mental health in emerging adult offspring of lesbian-parent families. Journal of Lesbian Studies, 23(2), 257-278.

[18] Kuroki, T., & Ukawa, A. (2018). Repeating probability of authors with retracted scientific publications. Accountability in Research, 25(4), 212-219.

[19] LaCour, M. J., & Green, D. P. (2014). When contact changes minds: An experiment on transmission of support for gay equality. Science, 346(6215), 1366-1369. doi: 10.1126/science.1256151.

[20] Lanier, W. L. (2020). Dealing with inappropriate-, low-quality-, and other forms of challenging peer review, including hostile referees and inflammatory or confusing critiques: Prevention and treatment. Accountability in Research, online advance.

[21] Malek, J. (2010). To tell or not to tell? The ethical dilemma of the would-be whistleblower. Accountability in Research, 17, 115-129.

[22] Marcus, A., Oransky, I. (2018). www.sciencemag.org/news/2018/02/meet-data-thugs-out-exposes-hoddy-and-questionable-research.com.

[23] McNutt, M. (2015a). Editorial expression of concern. Science, 348(6239), 1100. Doi: 10.1126/Science.aac.6184.

[24] McNutt, M. (2015b). Editorial retraction. Science, 348(6239), 1100. doi: 10.1126/science.aac6638.

[25] Milevsky, A. (2019). Parental factors, psychological well-being, and sibling dynamics: A meditational model in emerging adulthood. Marriage & Family Review, 55(5), 476-492.

[26] Milevsky, A. (2020). Sibling dynamics in adulthood: A qualitative analysis. Marriage & Family Review, 56(2), 91-108.

[27] Mistry, V., Grey, A., & Bolland, M. J. (2019). Publication rates after the first retraction for biomedical researchers with multiple retracted publications. Accountability in Research, 26(5), 277-287.

[28] Mosimann, J. E., Dahlberg, J. E., Davidian, N. M., & Krueger, J. W. (2002). Terminal digits and the examination of questioned data. Accountability in Research, 9, 75-92.

[29] Mosimann, J. E., Wiseman, C. V., & Edelman, R. E. (1995). Data fabrication: Can people generate random digits? Accountability in Research, 4, 31-55.

[30] Nurunnabi, M., & Hossain, M. A. (2019). Data falsification and question on academic integrity. Accountability in Research, 26(2), 108-122.

[31] Penders, B., & Shaw, D. M. (2020). Civil disobedience in scientific authorship: Resistance and insubordination in science. Accountability in Research, online advance.

[32] Pickett, J. T. (2020). The Stewart retractions: A quantitative and qualitative analysis. Econ Watch Journal, 17(1), 152-190.

[33] Pickett, J. T., & Roche, S. P. (2018). Questionable, objectionable or criminal? Public opinion on data fraud and selective reporting in science. Science and Engineering Ethics, 24, 151-171.

[34] Reisig, M. D., Holtfreter, K., & Bersofsky, M. E. (2020). Assessing the perceived prevalence of research fraud among faculty at research-intensive universities in the USA. Accountability in Research, advance online.

[35] Regnerus, M. (2012). How different are the adult children of parents who have same-sex relationships? Findings from the New Family Structures Study. Social Science Research, 41, 752-770.

[36] Ruggiero, K. M., & Marx, D. M. (1999). Less pain and more to gain: Why high-status group members blame their failure on discrimination. Journal of Personality and Social Psychology, 77, 774-784. doi: 10.1037/0022-3514.77.4.774.

[37] Ruggiero, K. M., & Marx, D. M. (2001). Less pain and more to gain: Why high-status group members blame their failure on discrimination: Retraction. Journal of Personality and Social Psychology, 81, 178. doi: 10.1037/0022-3514.81.2.178.

[38] Ruggiero, K. M., Steele, J., Hwang, A., & Marx, D. M. (2000). “Why did I get a ‘D’?” The effects of social comparisons on women’s attributions to discrimination. Personality and Social Psychology Bulletin, 26, 1271-1283. doi: 0.1177/0146167200262008.

[39] Ruggiero, K. M., Steele, J., Hwang, A., & Marx, D. M. (2001). “Why did I get a ‘D’?” The effects of social comparisons on women’s attributions to discrimination: retraction. Personality and Social Psychology Bulletin, 27, 1237.

[40] Ruys, K. L., & Stapel, D. A. (2008). Emotion elicitor or emotion messenger?: Subliminal priming reveals two faces of facial expressions [retracted]. Psychological Science, 19(6), 593-600.

[41] Schumm, W. R. (2021). Confirmation bias and methodology in social science: an editorial. Marriage & Family Review, 57(4), 285-293. Doi: 10.1080/01494929.2021.1872859.

[42] Schumm, W. R., Bosch, K. R., & Doolittle, A. (2009). Explaining the importance of statistical variance for undergraduate students. Psychology and Education – An Interdisciplinary Journal, 46(3/4), 1-7.

[43] Schumm, W. R., & Crawford, D. W. (2020). Is research on transgender children what it seems? Comments on recent research on transgender children with high levels of parental support. Linacre Quarterly, 87(1), 9-24.

[44] Schumm, W. R., Crawford, D. W., Fawver, M. M., Gray, N. K., Niess, Z. M., & Wagner, A. D. (2019). Statistical errors in major journals: Two case studies used in a basic statistics class to assess understanding of applied statistics. Psychology and Education – An Interdisciplinary Journal, 56, 1/2, 35-42.

[45] Schumm, W. R., Crawford, D. W., Higgins, M., Lockett, L., AlRashed, A., & Ateeq, A. B. (2018). Estimating the standard deviation from the range: A replication of analysis of demographic data reported in Marriage & Family Review, 2016-2017. Marriage & Family Review, 54, 777-792.

[46] Schumm, W. R., Crawford, D. W., & Lockett, L. (2019a). Using statistics from binary variables to detect data anomalies, even possibly fraudulent research. Psychology Research and Applications, 1(4), 112-118.

[47] Schumm, W. R., Crawford, D. W., & Lockett, L. (2019b). Patterns of means and standard deviations with binary variables: A key to detecting fraudulent research. Biomedical Journal of Scientific and Technical Research, 23(1), 17151-17153.

[48] Schumm, W. R., Higgins, M., Lockett, L., Huang, S., Abdullah, N., Asiri, A., Clark, K., & McClish, K. (2017). Does dividing the range by four provide an accurate estimate of a standard deviation in family science research: A teaching editorial. Marriage & Family Review, 53, 1-23. doi: 10.1080/0149.

[49] Schumm, W. R., Nazarinia, R. R., & Bosch, K. R. (2009). Unanswered questions and ethical issues concerning U.S. biodefence research. Journal of Medical Ethics, 35, 594-598.

[50] Simonsohn, U. (2013). Just post it: The lesson from two cases of fabricated data detected by statistics alone. Psychological Science, 24(10), 1875-1888. doi: 10.1177/0956797613480366.

[51] Stapel, D. A., & Lindenberg, S. (2011a). Coping with chaos: How disordered contexts promote stereotyping and discrimination. Science, 332, 251. doi: 10.1126/science.1201068.

[52] Stapel, D. A., & Lindenberg, S. (2011b). Retraction. Science, 334, 1202.

[53] Steen, R. G., Casadevall, A., & Fang, F. C. (2013). Why has the number of scientific retractions increased? PLoS One, 8(7), e68397. doi: 10.1371/journal.pone.0068397.

[54] Stern, A. M., Casadevall, A., Steen, R. G., & Fang, F. C. (2014). Financial costs and personal consequences of research misconduct resulting in retracted publications. eLife, 3, e02956. doi: 10.7554/eLife.02956.

[55] Stewart, E. A., Simons, R. L., & Conger R. D. (2000). The effects of delinquency and legal sanctions on parenting behaviors. Families, Crime, and Criminal Justice, 2, 257-279.

[56] Stroebe, W., Postmes, T., & Spears, R. (2012). Scientific misconduct and the myth of self-correction in science. Perspectives on Psychological Science, 7, 670-688. Doi: 10.1177/1745691612460687.

[57] Vogel, G. (2011). Psychologist accused of fraud on ‘astonishing scale’. Science, 334, 579.

[58] Wiedermann, C. J. (2018). Inaction over retractions of identified fraudulent publications: Ongoing weakness in the system of scientific self-correction. Accountability in Research, 25(4), 239-253.

[59] Willcox, B. L. (1992). Fraud in scientific research: The prosecutor’s approach. Accountability in Research, 2(2), 139-151.

[60] Wilson, D. B. (n.d.) Practical meta-analysis effect size calculator [online calculator]. https://campbellcollaboration.org/effect-size-calculator/research-resources/research-for-resources/effect-size-calculator.html.