Ideas for judging merit in manuscripts and authors

The integrity and progress of any academic field depends on reliable measures for judging the merit of its publications and their authors. In ecology, as in other disciplines, analysis and debate concerning the merit evaluation process itself, including both its criteria and consequences, have become subjects of a growing body of literature (e.g. Lortie et al. 2007, Statzner and Resh 2010). Troubling sources of bias and controversy over the judgment of merit occur, impacting at least three levels of the dissemination process: journal merit, reflected most commonly by 'impact factor'; manuscript/paper merit, reflected in the peer-review and editor-decision processes for submitted manuscripts, and by citation metrics of published papers; and author merit, reflected in criteria associated with research grant competitions, and with job-application/promotion processes, which draw largely on outcomes connected with the first two levels. The consequences of these merit rankings and perceptions of 'success' are not trivial, scaling up with potentially broad implications — e.g. authors with higher merit scores secure greater funding and hence capacity for continued research, wider propagation of their ideas and discoveries, and thus greater potential impact on socio-culture/economic institutions and policy. Importantly, those who rise to the top may often do so via a variety of subjective approaches, contingencies, and partisanships — not just good science. The topic of merit judgment was also a conspicuous theme in several commentaries featured in our third (2010) volume of IEE. Supp and White (2010) highlight the effect of review articles in elevating a journal's impact factor, and call for implementation of separate metrics that distinguish this effect, thus facilitating more accurate between-journal comparisons of merit. Donaldson et al. (2010), and Hochberg (2010) revisit the 'tragedy of the reviewer commons' issue, with particular recommendation for alleviation through better use of qualified referees represented among graduate students, post-docs and junior researchers. Finally, Wardle (2010) presents results from analyses showing that the recently launched 'Faculty of 1000' model generally fails as a reliable measure for judging the merit of published articles. Serious challenges remain regarding the integrity and value of the conventional peer-review system, and its approach to assigning merit and vetting science. In our view, these challenges need to be met by the development , testing, and implementation of new tools — by practicing scientists, for scientists. As ecologists using journals, we must engage critically in the process and criteria for evaluating merit at all levels — or we …

The integrity and progress of any academic field depends on reliable measures for judging the merit of its publications and their authors.In ecology, as in other disciplines, analysis and debate concerning the merit evaluation process itself, including both its criteria and consequences, have become subjects of a growing body of literature (e.g.Lortie et al. 2007, Statzner andResh 2010).Troubling sources of bias and controversy over the judgment of merit occur, impacting at least three levels of the dissemination process: journal merit, reflected most commonly by 'impact factor'; manuscript/paper merit, reflected in the peer-review and editor-decision processes for submitted manuscripts, and by citation metrics of published papers; and author merit, reflected in criteria associated with research grant competitions, and with job-application/promotion processes, which draw largely on outcomes connected with the first two levels.The consequences of these merit rankings and perceptions of 'success' are not trivial, scaling up with potentially broad implicationse.g.authors with higher merit scores secure greater funding and hence capacity for continued research, wider propagation of their ideas and discoveries, and thus greater potential impact on socio-culture/economic institutions and policy.Importantly, those who rise to the top may often do so via a variety of subjective approaches, contingencies, and partisanships -not just good science.
The topic of merit judgment was also a conspicuous theme in several commentaries featured in our third (2010) volume of IEE.Supp and White (2010) highlight the effect of review articles in elevating a journal's impact factor, and call for implementation of separate metrics that distinguish this effect, thus facilitating more accurate between-journal comparisons of merit.Donaldson et al. (2010), and Hochberg (2010) revisit the 'tragedy of the reviewer commons' issue, with particular recommendation for alleviation through better use of qualified referees represented among graduate students, post-docs and junior researchers.Finally, Wardle (2010) presents results from analyses showing that the recently launched 'Faculty of 1000' model generally fails as a reliable measure for judging the merit of published articles.
Serious challenges remain regarding the integrity and value of the conventional peer-review system, and its approach to assigning merit and vetting science.In our view, these challenges need to be met by the development, testing, and implementation of new tools -by practicing scientists, for scientists.As ecologists using journals, we must engage critically in the process and criteria for evaluating merit at all levels -or we end up with metrics and perceptions of our work that do not adequately reflect the science or the authors.In essence, we need to monitor and challenge the current system to ensure that there are not translation errors in the assessment of the work we strive so fervently to fund, conduct, and publish.
At the very minimum, one of the most challenging and rapidly growing problems with peer-review is the erosion of referee incentive.Several solutions have been proposed but remain mostly untested (Fox and Petchy 2010).The pressure to publish is immense, and for publicly-funded research, there is an expectation from granting agencies-and an ethical responsibility to society-to disseminate the results of this research.Accordingly, publishing one's own work will naturally and normally take precedence over reviewing that of others.The system has survived with gratuitous peerreviewing, but the traditional motivation for this is now suffering from a serious runaway decline, resulting from an eroding confidence in the assumption that one's own work will in turn be reviewed fairly and rigorously, in a timely fashion, and free of prejudice from bias and elitism.
Ecologists do not diligently book-keep this potential reciprocity, but certainly expect the average experience to reflect an equitable and reasonably efficient handling of one's work.The key assumption here is that the current model-run by a mix of fellow academics and publishers-should bring the strengths of both groups to bear on the problem, and not the weaknesses.Many authors are repeatedly frustrated and disillusioned by having manuscripts rejected from journals because these journals are scrambling to be more exclusive than their competitors-driven by an obsession with chasing higher impact factors-and these same authors, therefore, are unlikely to be interested in providing free reviewing service for these journals.In some cases, author exasperation is likely to morph into aggressive and cynical reviewing habits, or even a complete loss of faith in the peer-review system -a system that, in our view, is presently on the verge of crisis.We propose a solution for this problem in the first part of the editorial, followed by a second proposal for addressing the even more intangible problem of assessing author merit.

Author-directed peer-review (ADPR) -a new model for judging manuscript merit
In 2011, IEE will experiment with a new model for peer-review that relies on authors to do most of the work associated with what has always been the most frustrating-and now increasingly unmanageable-task for editors: getting the reviewing done.IEE will continue to seek referees and arrange reviews for submitted manuscripts in the traditional manner, if authors wish.But authors will now also have the option of submitting manuscripts for which they have already obtained reviews, with substantial savings in IEE author fees (and several other general advantages, discussed below).The key elements of the ADPR model are as follows: (1) The author solicits his/her own referees and negotiates a fee if referees require financial reimbursement in order to allocate the necessary time to provide a high quality review.If so, the referee might choose to invoice for a pre-payment/retainer, or after review for services rendered.For most journals-including IEEit will be important to seek referees who hold a PhD, or who are registered in, and expected soon to complete, a PhD program (e.g.Donaldson et al. 2010).
(2) The author asks the referee to complete and return a review form (see Appendix) with: (i) a standard review; and, if publication is recommended, (ii) a signed statement granting the author permission to acknowledge, within the paper, the referee's endorsement for publication (to be verified by the receiving journal; see below) in the event that the paper is published; and (iii) the referee's recommendation for which journal(s) the paper would be particularly suitable for submission.The referee also has the option of indicating a list of journals (if any) for which the above endorsement does not apply, and whether the endorsement is conditional upon the referee being able to post/publish onlinealongside the author's paper (in the event that it is published)-a commentary on the paper.In a separate cover letter, the author could include specific requests, e.g. to have the paper reviewed with particular target journals in mind, or to provide special editing for English grammar/style, etc.If it happens that a referee is able to provide an important contribution to the paper, then negotiation may follow for co-authorship -thus requiring that the new manuscript be submitted to a replacement referee for independent review.
(3) The author receives the above items from each referee, revises the manuscript, as necessary, and asks the same referees to evaluate the revision (if needed, and with further reimbursement if necessary) -or if not interested, then additional referees would be solicited by the author.
(4) Once the author has collected at least two favourable independent reviews-with all three items in (2) above -the author submits the manuscript to the journal(s) of his/her choice, together with the signed ADPR forms (scanned or faxed) indicating the referees' consent to be named within the published paper as endorsing referees.
(5) After standard screening for suitability (e.g.confirming that the topic/content of the submitted manuscript is suitable for the aims and scope of the journal), the editor of the receiving journal sends the paper to the referees identified by the submitting author -requesting their confirmation that the submitted version is indeed the same version that they reviewed and endorsed earlier.If required, the editor may also request copies of these earlier reviews.The editor might also elect to have an advisory/subject editor evaluate the paper regarding any special journal requirements, e.g.assessment of certain statistical/data analyses reported in the submitted manuscript, in the event that the referees may not have been qualified for this evaluation.For IEE, an advisory editor would confirm that the paper meets the criteria of the 'review pipeline' for manuscript acceptance: http://post.queensu.ca/~aarssenl/iee/iee-ReviewPipelineForManuscriptEvaluation.pdf.
(6) If the journal accepts the paper, then the referees' names are disclosed in the published article as a public endorsement that the paper was indeed peer-reviewed and recommended for publication.If the journal rejects the manuscript, the author can then immediately submit the paper, together with the same reviews, to a different journal (or series of journals) as necessary, until publication is found.Importantly, in this process, it is unnecessary to have the paper reviewed again and again by a different series of referees every time the author resubmits to a different journal.Accordingly, there is no reason-in theory-that the manuscript could not be submitted simultaneously to more than one journal.
We anticipate that the ADPR model would provide several advantages over the conventional peer-review model: (i) More rapid publication.There would be no manuscripts sitting for weeks on the desks of busy/neglectful editors, waiting to be sent out for review, stalled by the difficulty of finding referees, or waiting to be processed for author response to referee comments -all of which would have already been addressed and completed prior to submission.(ii) Reduced journal charges to authors.Author fees, especially for open access electronic journals (like IEE), could be reduced because of reduced journal correspondence/processing/handling time.The IEE author fee for submissions using the ADPR model-effective January 2011-would be $100, instead of the standard $300 per published paper (payable at time of acceptance).(iii) Relief for the 'tragedy of the reviewer commons'.There would be no need for multiple reviewing by multiple referees when 'ratcheting' down the 'impactfactor ladder' -as many authors are inclined to do.This would mean less wasteful/redundant reviewing (hence more rapid publication) -and importantly, a profoundly reduced workload for the academic community in general (see also vi).[Presently, editors often know that a paper has been previously reviewed elsewhere, but they have no access to the reviews].(iv) Simultaneous submission to multiple journals.Because the journal is not involved in arranging peer review, there should be no reason why the author could not submit the paper to several journals at the same time, thus not only finding acceptance more rapidly (contributing again to more rapid publication), but also allowing authors to maximize their chances of publishing in the most desirable possible journal.(v) Greater referee incentive.Although the ADPR model can work in theory without any requirement that referees be paid, paying referees-combined with published referee acknowledgement, plus opportunity for referees to post/publish their commentary on the reviewed paper (in the event that it is published)-provides the important principal advantage of referee incentive.(vi) Higher quality manuscripts (less reviewing-time spent on poorly-prepared papers).By having to pay referees, authors should be especially careful in producing a high quality manuscript before sending it out for first review (to avoid unnecessary payment for multiple reviews of multiple revisions).This would contribute, with the above factors, to promoting greater overall efficiency of the peer-review system, and more rapid progress of science.(vii) Higher quality reviews.Paid service combined with published referee acknowledgement, and opportunity for referees to post/publish their commentaries on reviewed papers, would not only minimize referee bias and promote greater referee accountability, but would also engage referees more directly in the mission for discovery that the manuscript represents.Any worry that the ADPR model might be inferior to the conventional process for manuscript merit judgment is unfounded when recognizing the currently limited record of success for reviewing panels of alleged experts (Wardle 2010).
The above advantages, we suggest, outweigh any potential drawbacks of the ADPR model.Some critics might claim that referees would be hard to find, since most would be unwilling to review if they could not remain anonymous.But referees are already hard to find, and getting harder; and while certainly some referees might insist on anonymity, we predict that most would not, and importantly, the best would not.Many good referees are routinely busy and overworked, and so would be attracted, we predict, to the ADPR model through the incentives of reimbursement, and opportunity for published commentary.Instead of being reluctant and 'too busy' to review, therefore, the ADPR model should encourage academics to compete with each other as 'referees for hire'.Importantly, this tool would also certainly expand the referee pool, which is currently limited by the connections, familiarity, or imagination of overworked editors under the current model.
Some might worry that the ADPR model would encourage and allow authors to essentially 'buy' favourable reviews.But this concern is likely to be minimized for at least three reasons.First, there are self-regulating features of ADPR that serve to guard against such bias.Because referee names are disclosed within the published paper, their reputations will be 'on the line'.Most referees, therefore, are likely to be honest, fair and rigorous in their reviews; plus, if their reputations suffer, referees will lose out on potential income from future reviewing.For the same reasons, referees are also unlikely to be flippant or careless about supporting a paper because it is authored by a friend (with or without payment).Referees will not want their names associated as endorsements for inferior papersat least not referees that will be regarded as having integrity with journals.Authors will also not be inclined to request reviews from close colleagues in order to avoid the perception of cronyism; many editors and readers of published papers tend to know (or can easily discover) the identity of an author's previous collaborators and close associates.
Second, as a quality check, concerned editors, can request that referees forward their earlier reviews that were sent to the authors, as confirmation that the manuscript was indeed rigorously reviewed.Editors can also elect to invite the referees to submit and post/publish their reviews/commentaries alongside the reviewed paper, in the event that the latter is accepted for publication (which has always been the policy at IEE).
Finally, editors and publishers retain control over whatever appears in their journals; accordingly, editors that suspect anything untoward about author-referee relationships or arrangements can simply reject the submission.Such concerns/suspicions may be expected occasionally, and even routinely, perhaps, by some editors/journals -but not routinely, we suggest, by most.We predict that most editors will find great advantage in being able to forgo the frustrating task of finding good referees, many of whom are generally too busy to review at all, especially with little or no incentive to do so.
With the ADPR model, editors will not need to settle for poor-quality reviews (e.g. with biased assessments and draconian recommendations for rejection) submitted by reluctant, over-worked, and unrewarded referees hiding behind anonymity.Importantly therefore, the ADPR model serves to promote science for what it is: as a mission for discovery -rather than as a mission for chasing impact-factor, facilitated by traditional editor 'gate-keeping' elitism (Aarssen and Lortie 2009).Recall (again) Wardle's (2010) analyses illustrating that the conventional decision-process for judging (predicting) manuscript merit (presumably using the same 'standards' represented in the editor 'gate-keeping' decision process) has limited success in predicting realized merit for published articles.Clearly, it is time to explore a new model.
We also anticipate, with the ADPR model, that referees will have an opportunity to develop reputations for high-quality reviewing service-like paid columnists/critics in popular media-thus providing supplemental income for post-docs and adjuncts, and indeed, graduate students (Donaldson et al. 2010, Hochberg 2010), or supplemental income in retirement, or during a reduced academic appointment.The best and most reputable referees will be in highest demand for their service; authors will seek to have these referees' endorsements in hand when submitting to a journal, as strong evidence in support of the paper's merit.The quality/impact of an article therefore may be judged as much or more by who the acknowledged referees are (combined with the article's citation metrics), than by the impact-factor of the publishing journal.
Note also that the costs for paying one's own referees can be covered by income earned from one's own reviewing service.Importantly also in this regard, the ADPR model allows journals to reduce their operating costs, and hence their author fees -leaving more author funds available therefore to pay for one's own manuscript reviews; plus, the remaining (reduced) cost for author fees can also be addressed through income earned from paid reviewing.In other words, one can earn enough income as a referee to cover both the costs of paying referees for one's own manuscripts, plus the costs of author fees for publishing one's own papers in open-access journals.This is analogous to how real estate markets work: if financially constrained, you cannot buy a new house without first selling your current house, which in turn requires that the potential buyer (of your house) must first sell his/her house.With the ADPR model, however, there are no analogous real estate agents or lawyers to pay, and hence much of the money needed to support the open-access peer-review publication process is put back into the hands of researchers, rather than into the bank accounts of profitdriven publishers.
Costs for paying referees could also be included as a legitimate line item for budgets in grant applications, just as it is routine to include budget entries for other professional services, e.g.statistical data analyses, molecular genetic lab analyses, soil chemistry lab analyses, English-editing service for non-English speaking researchers, and journal page charges/author fees.
We hope that other journals will also consider experimenting with the ADPR model as an option for manuscript submission, and that a growing body of online data will accumulate, thus allowing future evaluation of the merit of ADPR and potential for improvement.ADPR might conceivably evolve into an online registry for 'holding' ADPR-reviewed papers.Submitting authors could then simply refer several editors to the password-protected URL address for the paper, together with its reviews.An ADPR site could even be designed so that editors could subscribe to peruse this resource, unsolicited, to 'shop' for both the best available papers and the best referees.The site might also evolve to administer referee charges and author payments.Open-sourcing the review process and increasing transparency can only make it betteras with other recent revolutions associated with digital/ online approaches.One might imagine, for example, a future with 'wiki-peer-reviews', or a site similar to Metacritic, but for scholarly articles and not just for the popular media.The broad palette of free software now available could also be retooled to increase efficiency in peer-reviewing, and to increase the online availability (for authors, referees, editors, and publishers) of vast amounts of data on the processes, limitations, and successes of peer-reviewing.

Publication Impact Efficiency (PIE): a new metric for judging author merit
We also suggest that there is a need for additional indicators of author merit beyond the common practice of counting how many papers one has published in high impact-factor journals.We propose a novel metric based not on absolute publication impact (citations), but on the efficiency of that impact -in terms of the grant funding level available to generate the publication record.In other words, Publication Impact Efficiency (PIE) is the total number of citations for an author's work over a specified period, relative to the total grant funding level received over the same time period.
We explored this in a preliminary analysis using data available from the Natural Sciences and Engineering Research Council of Canada (NSERC), reporting recent individual research ('discovery') grant amounts awarded to applicants (http://www.nserc-crsng.gc.ca/Index_eng.asp).We collected data for the ten-year period between April 1999-April 2009 for all successful applicants to the 'Ecology and Evolution' panel whose record of NSERC funding spanned this time period.For these same individuals (N=310), we recorded the total number of citations (as of July 2010) reported by ISI Web of Science (http://apps.isiknowledge.com) for all papers published in refereed journals between 1999-2009, inclusive.
For every ten-fold increase in total NSERC funding level, there is a 10-fold increase in total citations for one's published papers, i.e. there is a slope of 1.0 for LOG-transformed data (Figure 1b).Importantly, this indicates that the NSERC system is, on one level, achieving its aim to award merit (grant funding) proportionately where merit is due.A slope departing significantly from 1.0 would represent an egregious failure of the system -with a slope <1.0 indicating a general trend of diminishing returns with increased grant funding, and a slope >1.0 raising doubts about the 'investment' value of awarding small grant sizes.
There is, however, enormous variation in the data; 73% of the between-individual variation in total citations is unaccounted for by variation in funding level.There are several possible reasons for this, including: between-individual time-lag variation (e.g.variation in publication record between 1999-2009 could be affected by variation in grant funding levels available prior to 1999); between-individual variation in amounts of research funding from other, non-NSERC sources; and between-individual variation in costs (needed budget sizes) associated with different types of research.Further analyses would be required to assess the importance of these factors; however, note that none of these sources of variation is likely to affect the slope of the relationship.One is left wondering, nevertheless, about the extent to which some of the individuals below the regression line in Figure 1b might be over-funded relative to their publication impact (and perhaps dramatically so for the six low outliers, which were triple-checked for data accuracy).
We anticipate that the PIE index has considerable potential value for granting agencies in arriving at decisions for allocating grant funds between competing applicants.It could also be used together with other metrics, such as the H-index (Hirsch 2005), in assessment of job candidates, and tenure and promotion applications.For example, in Figure 1, individual A has had grant funding that is about 10 times larger, but with only about 3 times more total citations for published work compared with individual B. Hence, PIE for B (0.00463) is more than twice as large as for A (0.00175); i.e. all else being equal, individual B could be judged as more efficient in generating research impact (citations) per dollar of grant support.
Science does not occur in a vacuum, nor does our capacity to conduct or publish research.A metric that incorporates the above indicators of merit might promote a more level 'playing field' in comparative analyses.We hope that others will be interested in exploring potential applications of the PIE index, or some evolving version of it, and we would welcome feedback concerning this.

Figure 1 .
Figure 1.Publication Impact Efficiency for 310 researchers in ecology and evolution, reflected in the relationship between total citations for published work versus the total NSERC grant funding available to support the author's research between 1999-2009.Grant funding data (Canadian dollars) were obtained from the Natural Sciences and Engineering Research Council of Canada (http://www.nserc-crsng.gc.ca/Index_eng.asp) for the ten-year period between April 1999-April 2009, for all successful applicants to the 'Ecology and Evolution' panel whose record of NSERC funding spanned this time period.The total number of citations (as of July 2010) for the same individuals was obtained from ISI Web of Science (http://apps.isiknowledge.com) for all papers published by the authors in referred journals between 1999-2009, inclusive.Raw data are shown in (a) and LOG-transformed data are shown in (b), including the least squares linear regression line and associated r 2 , slope and P-values.(See main text for description of individuals A and B).