The Peer Review Process
Peer review is a method of critical appraisal prior to publication of a manuscript, and has existed for over 200 years (Fiedorowicz et al., 2022). It allows for identification of flawed studies, interaction with content experts, improvement of manuscripts, and assesses suitability of a manuscript for a particular journal to ensure its findings are of interest to readers. However, the responsibility for ongoing support of the peer review process oftentimes falls upon individual scholars with extremely limited time. These time constraints are even worse for clinical academics who are juggling both a heavy clinical and academic load. Despite the necessity for peer review, it is oftentimes not well remunerated or recognized and relies entirely on an honour system amongst academics. This can lead to difficulty in finding suitable reviewers, significant delays in publication time, and delays of important scientific findings reaching the general public.
“Peer review is a method of critical appraisal prior to publication of a manuscript, and allows for identification of flawed studies, interaction with content experts, improvement of manuscripts, and assesses suitability of a manuscript for a particular journal to ensure its findings are of interest to readers.”
What is AI?
Artificial intelligence (AI) is a very broad term, however it deals with all aspects of mimicking cognitive functions for real-world problem solving and building systems that think like humans (Holzinger et al., 2019). As such, it is often referred to as machine intelligence in contrast to human intelligence. In recent times, AI has garnered significant interest due to its practical successes in machine learning (ML). ML is a field of AI which aims to develop software to automatically learn from previous information to continually improve to make accurate predictions based on new incoming data. As such, numerous AI tools are being released weekly, many of which are geared toward optimizing efficiency and academic writing.
Relevance of AI to Research
AI has countless applications to the research workflow, from formulating the research question to summarizing findings. As such, I recently published a methodological review titled, “How to optimize the systematic review process using AI tools” in JCCP Advances and was featured in a Papers Podcast on ACAMH Podcasts to further discuss (Fabiano et al., 2024; podcasts, 2024). AI has considerable potential to improve the efficiency of research synthesis. For instance, on average it takes 67 weeks to complete a systematic review; by incorporating AI, this timeline can be reduced to an impressive 2 weeks (Borah et al., 2017; Clark et al., 2020). However, using AI tools may introduce various risks that can impinge on the accuracy, reliability, and credibility of the research, which underscores the importance of human oversight and verification (Fabiano et al, 2024).
Can AI be Implemented into the Peer Review Process?
As there are clear benefits to incorporating AI into the systematic review process, could there be a place for AI to assist with peer review (Bauchner & Rivara, 2024)? A large issue with peer review starts from being unable to locate the relevant content experts who are willing to spend the time to conduct a high quality peer review. This task is particularly onerous when one considers that journals typically require 2 or more independent reviewers per manuscript. Here, AI may serve the role of complimenting human reviewers and potentially reducing the number of human reviewers required. As discussed in my methodological review (Fabiano et al., 2024), AI tools have the potential to automate many of the routine and time-consuming processes, and even perform these tasks more accurately than humans at times. As virtually all high-quality journals require authors to adhere to various reporting guidelines (such as those listed on EQUATOR), AI may be particularly useful at assessing adherence with these reporting guidelines, prior to having a human reviewer assess the manuscript (EQUATOR Network | Enhancing the QUAlity and Transparency Of Health Research, n.d.). This can aid editors in triaging and improving quality of manuscripts, prior to sending them out for external peer review. It is well known that at the external peer review process, numerous biases exist, typically with peer review results favouring prestigious institutions and authors with higher proficiency in English, independent of overall manuscript quality. As such, AI may have utility in providing unbiased reviews which accompany the human reviews of a manuscript.
There has been some preliminary work done which investigated the feasibility of implementing AI into the peer review process. Particularly, a study was just published in NEJM AI titled, “Can Large Language Models Provide Useful Feedback on Research Papers? A Large-Scale Empirical Analysis”, which created an automated pipeline using GPT-4 to provide comments on scientific papers (Liang et al., 2024). Here there was an overlap of 31-39% for the points raised by AI and human reviewers. Further, 57% of users found the AI feedback to be helpful/very helpful and 82% found it to be more useful than the feedback from human reviewers. These findings highlight the utility of AI in the peer review process, however further research is required.
As with the application of AI to the systematic review process, there are various risks and limitations to the use of AI for peer review to consider. Firstly, AI has the potential to generate false information that seems plausible but is not supported by evidence. This underscores the importance of human involvement in the peer review process for oversight and verification. Second, confidentiality is a concern as AI would add the reviewed manuscript to its dataset, effectively placing the work into the public domain. Lastly, many AI tools are only accessible behind paywalls, which may propagate inequality and unfairness in science as not all journals may have equal access to such tools.
“AI has countless applications to the research workflow, from formulating the research question to summarizing findings.”
Conclusions
In conclusion, AI has considerable potential to assist with the peer review process. AI may assist with reducing the number of human reviewers required, assessing adherence to reporting guidelines, and providing less biased reviews compared to their human counterparts. However, it is important to acknowledge the various risks and limitations to the use of AI for the peer review process including generation of false information, confidentiality and accessibility. Research in this area is preliminary, yet promising, and I personally believe that it is inevitable that all journals will adopt some form of AI to expedite their peer review process.
Conflicts of interest
Nicholas Fabiano was the lead author of “How to optimize the systematic review process using AI tools” which was published in JCCP Advances.
References
Bauchner, H., & Rivara, F. P. (2024). Use of artificial intelligence and the future of peer review. Health Affairs Scholar, 2(5), qxae058. https://doi.org/10.1093/haschl/qxae058
Borah, R., Brown, A. W., Capers, P. L., & Kaiser, K. A. (2017). Analysis of the time and workers needed to conduct systematic reviews of medical interventions using data from the PROSPERO registry. BMJ Open, 7(2), e012545. https://doi.org/10.1136/bmjopen-2016-012545
Clark, J., Glasziou, P., Del Mar, C., Bannach-Brown, A., Stehlik, P., & Scott, A. M. (2020). A full systematic review was completed in 2 weeks using automation tools: A case study. Journal of Clinical Epidemiology, 121, 81–90. https://doi.org/10.1016/j.jclinepi.2020.01.008
EQUATOR Network | Enhancing the QUAlity and Transparency Of Health Research. (n.d.). Retrieved September 17, 2024, from https://www.equator-network.org/
Fabiano, N., Gupta, A., Bhambra, N., Luu, B., Wong, S., Maaz, M., Fiedorowicz, J. G., Smith, A. L., & Solmi, M. (2024). How to optimize the systematic review process using AI tools. JCPP Advances, 4(2), e12234. https://doi.org/10.1002/jcv2.12234
Fiedorowicz, J. G., Kleinstäuber, M., Lemogne, C., Löwe, B., Ola, B., Sutin, A., Wong, S., Fabiano, N., Tilburg, M. V., & Mikocka-Walus, A. (2022). Peer review as a measurable responsibility of those who publish: The peer review debt index. Journal of Psychosomatic Research, 161, 110997. https://doi.org/10.1016/j.jpsychores.2022.110997
Holzinger, A., Langs, G., Denk, H., Zatloukal, K., & Müller, H. (2019). Causability and explainability of artificial intelligence in medicine. WIREs Data Mining and Knowledge Discovery, 9(4), e1312. https://doi.org/10.1002/widm.1312
Liang, W., Zhang, Y., Cao, H., Wang, B., Ding, D. Y., Yang, X., Vodrahalli, K., He, S., Smith, D. S., Yin, Y., McFarland, D. A., & Zou, J. (2024). Can Large Language Models Provide Useful Feedback on Research Papers? A Large-Scale Empirical Analysis. NEJM AI, 1(8), AIoa2400196. https://doi.org/10.1056/AIoa2400196
podcasts, A. (2024, June 24). How to Optimize the Systematic Review Process using AI Tools. ACAMH. https://www.acamh.org/podcasts/how-to-optimize-the-systematic-review-process-using-ai-tools/