Injecting youth into peer-review to increase its sustainability : a case study of ecology journals

The ‘tragedy of the reviewer commons’, where the referee pool is being drained by an influx of manuscript submissions, is becoming an increasing problem for journals. To mitigate this growing concern, we suggest that there is a need to inject youth into the peer-review process. Graduate students, post-docs, and junior researchers are an important, yet often underutilized resource, for the peer-review process. A survey of leading ecology journals revealed that editors are generally receptive to increasing the involvement of junior referees in the peer-review process. Through 45 responses to the survey, no journal had specific policies regarding junior referees or maintained databases specifically identifying junior reviewers, and only 34% of respondents actively solicited junior referees. Despite this level of utilization, editors generally feel that junior referees have expertise in specific subject areas, are keen to review, and are willing to make time to provide a high-quality review, particularly when aided by their supervisors. Editors were significantly more likely to select junior referees with higher levels of education and experience (e.g., 90% of respondents were highly likely to select a post-doctoral researcher to conduct a review), although editors were willing to consider graduate students as well. Integrating and expanding on these survey results, we argue that injecting youth into peerreview benefits the process and provides invaluable experience and insight to junior scholars, all while helping to mitigate the problem of the tragedy of the reviewer commons.


Introduction
Perhaps the biggest headache that any journal editor endures stems from securing appropriate referees to provide peer-review for manuscripts.Potential referees are often overburdened with other commitments and struggle to find the time or incentive to undertake a review (Aarssen and Lortie 2009).A given manuscript may be submitted to half a dozen potential referees before one finally agrees to review the document.This process is tedious and cumbersome for both the editorial board and authors alike, as seeking referees may take weeks or even months, causing a substantial delay in arriving at an editorial decision.With a growing number of submissions, overburdened journals reject a higher proportion of manuscripts resulting in authors repeatedly submitting their work to multiple journals, in turn requiring more reviews and putting a further strain on the referee pool (Hochberg et al. 2009).A repeatedly-rejected manuscript may run through several reviewers at multiple journals prior to finally being accepted for publication.This strain results in what has been appropriately termed 'the tragedy of the reviewer commons', where the behaviour of individual authors leads to the detriment of the broader group of scientific researchers by straining the referee pool (Hochberg et al. 2009) and threatens the sustainability of the peerreview process in the long-term.This problem appears difficult to rectify at first glance, but we contend that graduate students, post-docs, and young professionals are underutilized as referees for ecological research and other fields, yet they may contribute substantially to sustaining peer-review.We contacted editors of the leading journals in the field of ecology to gain insight into their journals' policies as well as the editors' own opinions on junior referees and peer-review.Integrating and expanding on these survey results, we argue that injecting youth into peer-review benefits the process and provides invaluable experience and insight to junior scholars, all while helping to mitigate the problem of the overburdened referee pool.

Survey methodology
ISI Web of Science was used to sort the top journals in the topic 'ecology' by their 2008 impact factor.A database of editors from each journal was established and editors were individually contacted by E-mail.We targeted the editor-in-chief or co-editors as needed.The body of the E-mail contained a brief summary of the objective of the survey and the intended use of the results.All respondents were notified that their input would only be presented in aggregate and their name along with the name of their journal would not be reported.The survey questions were embedded in the body of the E-mail.The survey contained three components: (i) questions regarding journal policies on junior referees, (ii) open-ended questions regarding the editors' own opinions on the benefits and risks of junior referees and peer-review, and (iii) quantitative opinionbased questions on junior referee selection and expectations, using a standard Likert scale (Box 1).Editors were given three weeks to reply.In total, 85 editors were contacted and 45 replies were received (52.9% response rate).Likert responses were coded as 1 through 5 and statistically analyzed using Pearson's χ 2 tests at α = 0.05 using JMP 8.0 (SAS Institute Inc., Cary, North Carolina).
Box 1.A survey questionnaire that was distributed to editors of leading ecology journals (i) Journal policy questions: 1) Does your journal have an official policy related to using junior referees?2) Do you actively solicit junior referees?3) Does your journal or publisher maintain a database identifying junior referees?
(ii) Editors' opinions on the benefits and risks of using junior referees: 1) Please list your top three concerns of utilizing junior referees: 2) Please list your top three benefits of utilizing junior referees: (iii) Editors' opinions on junior referee selection and expectations: 1) If a potential referee declines to conduct a review but suggests a junior referee in their place (e.g., a graduate student they supervise), would you be 'highly likely', 'likely', 'neutral', 'unlikely', or 'highly unlikely' to ask the junior referee to conduct a review if they were: a) a senior undergraduate student b) a M.Sc.student c) a Ph.D. student d) a post-doc e) a bachelor-level young professional f) a graduate-level young professional 2) From your experience, do you believe that relative to senior referees, junior referees would be 'highly likely', 'likely', 'neutral', 'unlikely', or 'highly unlikely' to: a) decline to conduct a review b) provide a high quality review c) provide a timely review d) provide a fair review e) accept an article f) reject an article 3) Please feel free to include additional comments relating to junior referees

Journal policies on junior referees
The editors' responses revealed that none of the journals surveyed had an official policy on junior referees and none actively maintained a database that specifically identified and/or ranked junior referees.Many journals tracked whether or not reviewers in their database had a Ph.D. based on optionally submitted affiliation and salutation data; however, this information can only be obtained through optional submissions from the reviewers.However, 34% of respondents answered that they solicited junior referees, most of which were approached through their supervisors.Those editors that did not actively solicit junior referees generally still indicated that they were willing to consider approaching junior referees, particularly if they had already published their own articles on a topic related to the manuscript in need of review.

Editors' opinions on the benefits and risks of using junior referees
When given the opportunity to list the benefits and risks of using junior referees in peer-review, editors' comments revealed that, in general, they are receptive to increasing the involvement of junior referees in the peer-review process, however a number of reservations were also expressed (Table 1).In terms of benefits, editors generally felt that, while young researchers may lack years of experience relative to senior researchers, they are willing to make time to review, have expertise in specific subject areas, and are generally capable of providing thorough and detailed reviews.Editors suggested that junior referees may have fewer ties to other researchers and by association less bias when reviewing for colleagues or leaders in the field.Several editors felt that junior referees may have fewer time commitments and less "reviewer fatigue" than their more experienced colleagues, so their willingness to accept an invitation to conduct a review is likely higher, and they are willing to provide comments in a timely manner (i.e., before the deadline).A number of editors expressed concerns about the junior referee's knowledge of the history of the field and familiarity with older literature.Several editors commented that junior referees can be either overly critical or not critical enough when they are inexperienced with the process.Concerns over lack of technical experience, including experimental design and statistical approaches, were also raised.A concern that was expressed by several editors was the difficulty of identifying junior referees, particularly those with a developing publication record.Interestingly, editors cited benefits to the students themselves as one of their top responses, including training and gaining valuable experience in the publishing process, and enabling junior researchers to contribute to their field by serving a crucial role in the peer-review process, which in turn builds confidence.Respondents also suggested that serving as a referee forces junior scientists to pay careful attention to detail in the manuscripts they review, which often yields a better understanding of the technical and scientific aspects of the manuscripts.

Editors' opinions on junior referee selection and expectations
The survey results showed that editors' preferentially select referees with greater experience (i.e., more likely to choose post-doctoral investigators or Ph.D. students relative to senior undergraduate or M.Sc.students [ Table 2]).Editor opinions were less clear on whether or not they would choose undergraduate-level or graduate-level young professionals to serve as referees.
From their experience, editors suggested that junior referees were less likely to decline to conduct a review, were more likely to provide a high quality, timely, and fair review, and tended to be neutral as to whether or not junior referees are more likely to accept or reject a manuscript.

Discussion
The survey revealed that editors were generally receptive to increasing the utilization of junior referees.However, one concern that was expressed by several respondents was that junior referees have a vast range in their experience with thinking critically, which in turn means that reviews can be either overly critical or not critical enough.Peer-review is an altruistic process and it has been proposed that referees should follow a golden rule for reviewing, where we should strive to review for others with the same quality as we would have them review for us (i.e., timely, unbiased, balanced reviews [McPeek et al. 2009]).Many junior referees are particularly alert to this fact, as those who have received reviews of their own manuscripts quickly learn that there is a vast range in the quality of feedback from the peer-review process.Due to this awareness, junior referees may pay more attention to detail and invest the time to conduct thorough reviews.This heightened attention to detail may come at a cost, as junior reviewers are sometimes trained to be overly critical, and may attempt to measure a manuscript to a golden standard of science (Walbot 2009).Survey respondents were split on the issue of whether or not junior referees had a greater tendency to submit either over-, or under-critical reviews.Thus junior referees, and perhaps all referees in general, run the gamut from being overly critical to struggling to think critically.Only through practice, Table 1.A summary of some key advantages and disadvantages suggested by editors of leading ecology journals on the topic of using junior referees in the peer-review process, as well as a list of ideas for mitigating potential disadvantages.

Willingness to review
Generally keen and have more time to dedicate to review.
Conducting reviews for which they are underqualified.
Should consult with editors and supervisors to ensure they are qualified to review the manuscript.

Expertise
Generally knowledgeable in their particular subject areas.
Less knowledge on broader subject areas and the history of their topic.
May have less experience than senior researchers.
Junior referees should consult with colleagues and conduct necessary background research in order to adequately review manuscripts.

Quality of reviews
With more time to conduct reviews, junior referees can deliver quality, detailed reviews.
May lack confidence to think critically.
Work with supervisors, and perhaps small groups of other junior referees, to ensure the quality of the review.

Supervisor input
Supervisors can help junior referees throughout the process and ensure they provide high quality reviews.
Junior referees may rely heavily on their supervisors for the first few reviews they conduct.
Supervisors should make time to help junior referees with their early reviews.After all, mentorship and training is essential to the scientific process and starting early may set a young researcher towards becoming a better writer and researcher themselves (Bourne 2005).experience, and help from their peers and supervisors, can junior referees find a way to follow the golden rule of reviewing.The experience of serving as a referee can help to develop important critical-thinking skills that not only improve their peer-reviews but also help to improve the quality of their own writing (Bourne 2005).
One potential method of alleviating these concerns is by opting for a transparent review process, where reviewer responses are exposed with the article (e.g., Ideas in Ecology and Evolution; Aarssen 2008).
A number of respondents commented on the level of experience of junior referees.While they lack years of experience, graduate students working on specific research projects are generally well-versed in their subject areas and are up-to-date on the latest papers in their field.In instances where they are unfamiliar with a particular topic (e.g., a statistical method), several editors felt that they may be more likely to seek out more information on it.The graduate student experience helps to ensure this level of knowledge through comprehensive exams, thesis writing, manuscript writing, and conference presentations.These activities keep junior scientists aware of the latest developments in their fields, and, their fresh perspectives may provide new insight for more senior authors in their subject areas.However, proper training in peer-review can only come from those that are more experienced in the process.For example, one editor responded that each year the graduate program in his department holds a course and training workshop to train graduate students on how to be effective reviewers.This could even be taken a step further by introducing training or even certification programs that are governed by professional societies.Currently, this type of engagement is relatively rare, yet would go a long way in training junior reviewers, particularly if they receive additional support from their supervisors.Supervisors can be involved in each component of the review process, such as suggesting their students/employees to editors as potential referees, mentoring them during the review processes, and helping the junior referee proof-read and edit their final reviews.Junior referees should confer with their supervisors and/or colleagues to decide which papers are appropriate to review.Supervisors can then provide input throughout the review process.If supervisors are willing to be engaged in this process (e.g., as evidenced by the authorship on this article; Box 2), they will be setting their students/employees on a track to be better writers and scientists themselves (Bourne 2005).
Junior researchers may be keener to review in order to help build their curriculum vitae, which may be recognized favourably by potential graduate student supervisors, employers, granting agencies, awards committees, or tenure review committees, or may review manuscripts to learn about new and cutting edge research and methodologies in fields related to their Box 2. The authors' positions and peer-review experience.
• Donaldson is a Ph.D. student at the University of British Columbia (UBC) who has 4 years of reviewing experience and has reviewed 12+ manuscripts.
• Hasler is a Ph.D. student at Carleton University (CU) who has 4 years of reviewing experience and has reviewed 12+ manuscripts.
• Clark is an early career research associate at UBC who has 5 years of reviewing experience and has reviewed 30+ manuscripts.
• Hanson is a researcher with the U.S. Fish and Wildlife Service at the Abernathy Fish Technology Center who has 5 years of reviewing experience and has reviewed 30+ manuscripts.
• Hinch is a research supervisor and professor at UBC who has 15 years of reviewing experience and has reviewed 75+ manuscripts.
• Cooke is a research supervisor and professor at CU who has 11 years of reviewing experience and has reviewed 150+ manuscripts.
Cooke is the former supervisor of Donaldson and Hanson and the current supervisor of Hasler.Hinch is the current supervisor of Donaldson and employer of Clark.own to foster ideas that enhance their current and future studies.Some organizations that provide certification for their members (e.g., American Fisheries Society, British Columbia Association of Professional Biologists) consider review experience as a demonstration of professional involvement needed to obtain or maintain certification.With a growing number of young scientists interested in acting as referees, the obvious challenge facing editors is to find ways to reach out to appropriate potential reviewers, a concern echoed by a number of survey respondents.If a senior referee is contacted by an editor but declines to review an article, it should be their responsibility to suggest appropriate alternate reviewers from the field, including junior referees they may be aware of.As evidenced by the survey results, many editors already explicitly ask senior referees to recommend junior referees to conduct reviews.Some families of journals establish keyword searchable databases of student referees where students can request they be added to the database (e.g., American Fisheries Society).However, keywords may not be the most appropriate way to select referees.Publishers could provide online systems where users can create unique profiles which include details of their curriculum vitae that would enable editors to view more specific details of potential reviewers to ensure that they select the right person for the job.Journals may wish to invest time in creating a database of the timing and quality of reviews from both junior and senior referees to aid in future referee selection, a point suggested by one of the survey respondents.Information such as the number of reviews that a referee accepts or rejects in a year, the timeliness of completion of said reviews, and the quality of the final review may provide editors with a powerful tool to thoughtfully decide to whom to send a manuscript.For example, if prior to sending out a manuscript for review, an editor could search such a database and eliminate potential referees for any one of a number of reasons (e.g., junior reviewers who have conducted numerous reviews lately and are more likely to decline, those who typically do not review any papers, those that tend to be tardy with reviews, etc.), the time-consuming search for reviewers could be drastically shortened.
Finally, in order for junior referees to agree to review a paper and be engaged in the process, it is of course the author's responsibility to make their manuscripts 'referee-friendly' and be mindful that their work may indeed be reviewed by junior researchers.Essentially, a writer must assume that their work will eventually be read by individuals with a variety of levels of technical expertise and experience.Therefore, writing with jargon-free prose that provides clarity will help the junior referee and other readers (Sand-Jensen et al. 2007, Walsh et al. 2009).Junior referees will be more engaged and better able to provide a quality review if the article is written clearly, using accessible language.

Conclusions
With the tragedy of the reviewer commons becoming increasingly problematic in ecology, we need to continue to look to junior researchers to serve as peerreviewers.Including junior researchers in the peerreview process early and often will both increase the available pool of reviewers as well as provide them with much needed career experience that can improve the quality of both their reviews and manuscripts for years to come.In order to make this possible, a series of small changes must be made to recruit appropriate junior referees and ensure that the quality of the review process remains at the highest possible level.We are hopeful that this article stimulates discussion on this topic and provides a new direction for the peer-review process.Peer-review is central to science.The scientific community must adapt to the increasing number of submitted manuscripts, and continue to identify new directions and opportunities to ensure the future sustainability of the peer-review process.

Response to referee
Hochberg (2010) presents an evolutionary analogy of peer-review, where the process itself is subject to selection pressures that ultimately result in adaptive responses to a dynamic environment.The internet can be considered the environment that is rapidly shifting the way that peer-review operates.As the environment changes, so must the peer-review process, yet the results of our survey show that peer-review itself is lagging behind these rapid changes, and junior referees need to be better integrated into the process.While implementing changes to address this concern is challenging, two salient directions, both of which rely on internet technologies, for better integrating junior referees into the peer-review process emerged from both our paper and Hochberg's (2010) commentary.We pro-posed the idea of developing a database that tracks junior researchers to enable journals to readily select the best available reviewers for a given manuscript.We also suggested that training and mentoring is essential to the process of learning to be a strong referee.If injecting youth into peer-review can truly help mitigate the tragedy of the reviewer commons, at the very least these two changes need to be implemented.
As Hochberg (2010) states, most journals already have established searchable databases to track their referees, some of which track details about education level and reviewing experience.However, no journals that we surveyed specifically tracked junior referees.In order for junior referees to be included in such a database, they must first be sought out, which can be difficult, particularly if they are new to publishing and are not yet well known to their field.Academic societies can be useful here to help establish databases of their student members who might be interested in serving as a referee.Further, students themselves could contact editors and express interest in reviewing manuscripts.In the latter case, a ranking scheme could be considered to track reviewing experience and expertise.Essentially, this change only requires expanding on databases that most journals already maintain.
Despite the many benefits that emerged from the survey, concerns over the quality of reviews provided by junior referees were expressed by some of the survey respondents and were also echoed by Hochberg (2010).This concern likely stems, at least in part, from the fact that there is no formal training associated with peerreview for referees at any stage in their career, despite the importance that quality reviews have for the scientific process.As alluded to in our article, receiving training from supervisors, mentors, and colleagues is important to the learning process and will go a long way towards solving these concerns.However, formal training programs (which are currently quite rare) would be an excellent way to promote standardized methods for learning how to conduct fair, balanced, and effective reviews.This is another activity that could be offered at the society level and could be linked with a certification program to provide further incentive for review and help build the curriculum vitae of young professionals.Recently, the journal Marine and Freshwater Research initiated a pilot program to mentor junior referees by first recruiting interested students and young professionals, and then providing them with assistance and detailed feedback on their reviews.These examples represent relatively small changes that can be made which may result in large gains towards mitigating the tragedy of the reviewer commons.
As the peer-review process continues to rapidly evolve, small changes need to be made to better integrate junior referees.We suggest that improving databases that track junior referees and establishing training and certification programs for peer-review are important considerations to help the long-term sustainability of the peer-review process.
Hochberg, M.E.2010.Youth and the tragedy of the reviewer commons.

Table 2 .
Editors' responses to junior referee survey.
If a potential referee declines to conduct a review but suggests a junior referee in their place (e.g., a graduate student they supervise), would you be 'highly likely', 'likely', 'neutral',  'unlikely', or 'highly unlikely'to ask the junior referee to conduct a review if they were: