Are editors of ecological journals good oracles ? A reply to Schimel et al . ( 2014 ) about the malpractice of editorial rejections

In an earlier work, we found that 66% of manuscripts that suffered editorial rejections were finally accepted in journals of similar ranking to which they were originally submitted. We thus concluded that editors appear to be “poor oracles” with regards to being able evaluate the quality of a manuscript without the help of external reviewers. This article was recently criticized by the team of editors of the Ecological Society of America . In this work, we clarify some misunderstandings and offer new evidence supporting our view that external reviews should be the rule in the process of publishing scientific literature. Specifically, here we argue that (a) the claim that editorial rejections are based on manuscripts not adjusting to the journal’s scope rather than on academic quality is unconvincing; (b) if academic quality is being assessed to decide the fate of a submitted paper, this attribute must be evaluated including several external opinions and not only the superficial reading of one person, (c) our survey design was appropriate and, despite the small sample size, the conclusion that editors are poor oracles seems to be fairly reliable and, (d) the practice of sending the majority of submitted papers to external review should not cause the collapse of most popular journals. We insist that for the sake of science, editors need the opinion of external experts and should not act as oracles

An increasing number of ecological journals skip the traditional and productive process of peer review rejecting papers based on the opinion of only one person: the subject editor.This practice is becoming more common despite criticisms related to its subjectivity and inappropriateness (Farji-Brener 2007, Bornmann and Hans-Dieter 2010, Cooke and Lapointe 2012, Arnqvist 2013).In a recent article (Farji-Brener and Kitzberger 2014), we questioned a commonly given rationale of editorial rejections: that editors reject papers that "definitively" would have received negative reviews if they were sent to reviewers (Strong 2007).We tested whether editors are good "oracles" by monitoring the final destiny of a large number of papers that were first rejected without revisions by an editor and re-submitted without changes to a different journal.We found that 66% of manuscripts that suffered editorial rejections were finally accepted in journals of similar ranking (i.e, in the same quartile of SCImago Journal & Country Rank) to which they were originally submitted.We thus concluded that editors appear to be "poor oracles" with regard to being able to "easily identify the most exciting, interesting, cutting-edge science manuscripts" (sensu Strong 2007).Our work, originally published in the Bulletin of the Ecological Society of America, generated a rapid response by the team of ESA editors, who disagree with our perspective, arguments, and conclusions (Schimel et al. 2014).Their main arguments were four: (1) editorial rejections are based not on the "academic quality" of manuscripts, as we suggested, but rather on whether the submitted work fits the journal goals and scope; (2) our conclusion that editors are poor oracles was based on a small, biased sample; (3) editorial rejections are based on the opinion of more than one editor, which reduces the potential biases of one person assessing the manuscript; and (4) editorial rejections are necessary because leading journals receive many more submissions than they can handle.We thank Dave Schimel and his 160 co-authors (hereafter, the Army of Editors, AOE) for keeping alive the subject of the importance of external reviews for publishing scientific literature.Unfortunately, the Bulletin policy allows "only one contribution and one response."Therefore, we submitted this paper to clarify some misunderstandings, discuss some arguments of the AOE and offer new evidence supporting our view that external reviews should be the rule in the process of publishing scientific literature.
We first want to clarify that our original note was not a particular critique of ESA journals; it is a critique of the whole system of avoiding external reviews used by an increasing number of ecological journals.Second, the fact that our note has remained for months as one of the most downloaded articles in the Bulletin suggests that the role of editors in science is today a matter of debate and/or concern among many ecologists.Third, our criticisms are not the complaints of jilted authors.In our academic life we have suffered editorial rejections as well as decided the fate of submitted manuscripts.We critiqued the absence of external reviews merely as scientists concerned about the policies that guide publication of scientific literature.Finally, we agree that journals must establish quality standards and acceptance rules.But exactly for that reason, we propose that external reviews are the most fruitful and objective way to decide rejections.

Testing the "erroneous scope hypothesis" as the genuine reason for editorial rejections
We disagree with the AOE that a misfit with the journal scope is often the main reason behind editorial rejections.The example provided by the AOE to illustrate this argument is inappropriate.As the AOE correctly assert, a paper that fits well in Proceedings of the Royal Society of London may not necessarily fit well in Ecology.However, the opposite is entirely possible, because the first journal publishes topics of general biology, including ecology.In other words, the scope of Ecology could be considered to be within the scope of Proceedings of the Royal Society of London.This phenomenon of nested scopes is common in scientific journals.Indeed, we documented several cases of editorial rejections in leading journals with specific ecology scopes that were finally accepted in top journals of broader biological interests (Farji-Brener and Kitzberger 2014, Table 1).More importantly, we will show that the similarity of scopes among ecology journals makes it almost impossible to base editorial rejections on them.
The AOE claims that editors base their rejections on "explicit, published criteria regarding the scope of the journal" (Schimel 2014: 343).Moreover, the AOE asserts that "each journal has its own unique scope and goal" (Schimel et al. 2014).This may be true for the family of ESA journals, but it is not a general rule for ecological journals.To test the robustness of this argument, we selected seven journals belonging to different quartiles of the SCImago Journal & Country for the subject area of Ecology.We transcribed from the websites the goals and scopes of the selected ecology journals maintaining the journal name anonymous.In a separate column we listed, in random order, the journal names (Table 1).If the assertion of the AOE is true, it should be easy to match the scopes with the corresponding journal.We believe that this task is close to impossible.It is very clear that there are not "unique scopes, goals or published criteria" among these journals.Moreover, there is a broad thematic similarity among different journals' scopes.The only slight differences involved subjective topics such as "important ecological phenomena," "original and innovative ecological aspects," and "the most novel research in ecology."Due to the arbitrary nature of these concepts (Arnqvist 2013, Lortie 2013), more than one opinion and in-depth reading is necessary to establish whether a manuscript is among the most "important, novel, or innovative" works.The same pattern emerges when trying to match scopes with their corresponding journals in areas such as plant ecology, animal behavior, tropical ecology and other sub-disciplines.In sum, the overwhelming similarity between journals makes scope on its own a poor criterion on which to base editorial rejections.
We speculate that journals seek to differ in the quality of the published manuscripts and thus reject those works that do not fit their intended "quality", and not only the scope.This explanation is more plausible than assuming that the 40% of the manuscripts submitted to-and editorially rejected by-ESA journals do not fit the journal scope (Schimel et al. 2014).Moreover, that the "academic quality" is the main cause of editorial rejections (compared with the proposed "improper scope journal" justification) is directly supported by reading the information for authors of some popular journals (e. g., Science explicitly states "submissions are evaluated by the staff editors for potential significance, Table 1."Journal Scope Game".Match the scope on the left with the corresponding journal.Scopes from journals were transcribed as they appear in their web site; trivial information that revealed the journal name was omitted.quality, and interest" [bold added for emphasis] http://www.sciencemag.org/site/feature/contribinfo/faq/#types_faq), or indirectly by the words of the editor-inchief of Ecology, who claimed that "our editors easily identify a large portion of the most exciting, interesting, cutting-edge science manuscripts" (Strong 2007).If those words are not proxies of quality, then, what are they proxies of?There is nothing wrong with journals accepting manuscripts according to pre-established quality rules.Works may differ in their theoretical framework, originality, match between objectives and methods, sample effort, analyses, interpretation and other estimators of academic quality that can be properly evaluated.The main issue is whether one person can accurately evaluate a manuscript with a superficial reading and without the help of external opinions.

Bias or no bias? Strengths and weaknesses of our survey and conclusions
As explained above, we tested whether editors were good oracles-that is, good at making correct decisions about the quality of a manuscript without the opinion of external reviewers.We found that 66% of the 65 editorially rejected papers that were resubmitted without A.This journal publishes articles that report on the basic elements of ecological research.Emphasis is placed on concise, clear articles documenting important ecological phenomena.This journal publishes a broad array of research that includes a rapidly expanding envelope of subject matter, techniques, approaches, and concepts: paleoecology through present-day phenomena; evolutionary, population, physiological, community, and ecosystem ecology, as well as biogeochemistry; inclusive of descriptive, comparative, experimental, mathematical, statistical, and interdisciplinary approaches B. This journal publishes original research papers, reviews, technical reports, notes and comments, and data papers covering all aspects of ecology and ecological sciences.
C. This journal publishes original and innovative research on all aspects of ecology, defined as organism-environment interactions at various spatiotemporal scales, so including macro-ecology and evolutionary ecology.Emphasis is on theoretical and empirical work aimed at generalization and synthesis across taxa, systems and ecological disciplines.Papers can contribute to new developments in ecology by reporting novel theory or critical empirical results, and "synthesis" can include developing new theory, tests of general hypotheses, or bringing together established or emerging areas of ecology.Confirming or extending the established literature, by for example showing a result that are novel for a new taxon, or purely applied research, is given low priority D. This journal publishes original ecology works focusing on patterns and processes at various temporal and spatial scales and at different levels of biological organization.This journal welcomes work in evolutionary and behavioral ecology, eco-physiology, population and community ecology, landscape and ecosystem ecology, numerical ecology, dendro-ecology and, paleoecology.
E. This journal publishes original research articles in ecology.We encourage studies in all areas of ecology, including ecosystem ecology, community ecology, population ecology, conservation ecology and evolutionary ecology.There is no bias with respect to taxon, biome or geographic area.Both theoretical and empirical papers are welcome, but combinations are particularly sought.Priority is given to papers based on explicitly stated hypotheses.
F. This journal publishes the most novel research in ecology.Manuscripts relating to the ecology of all taxa, in any biome and geographic area will be considered, and priority will be given to those papers exploring or testing clearly stated hypotheses.The journal publishes concise papers that merit urgent publication by virtue of their originality, general interest and their contribution to new developments in ecology.We discourage purely descriptive papers and those merely confirming or extending results of previous work.

Oecologia
G. This journal publishes innovative ecological research of international interest.We seek reviews, advances in methodology, and original contributions, emphasizing population ecology, plant-microbe-animal interactions, ecosystem ecology, community ecology, global change ecology, conservation ecology, behavioral ecology and physiological Ecology.In general, studies that are purely descriptive, mathematical, documentary, and/or natural history will not be considered.
changes to other journals were published in journals of comparable academic quality after peer review in the second journal (Farji-Brener and Kitzberger 2014).We thus concluded that editors appeared to be poor oracles.The AOE suggested that our sample was biased and we therefore overestimated the fraction of editorial-rejected manuscripts that could be successfully published elsewhere without revision.The AOE argued that we should include the fate of papers that were substantially revised, or even abandoned after editorial rejection.We did not include those papers because they provide no information to assess whether editors are good oracles.
First, the fate and potential academic quality of abandoned papers is impossible to evaluate.Second, the acceptance of re-submitted, revised manuscripts includes a confounding effect (the paper improvement) that impedes an adequate test of the oracular ability of editors.In other words, if a paper is improved by the authors, resubmitted, and finally accepted in another journal of comparable quality, we will never know whether this acceptance means that the first editor made the wrong decision, or that the paper actually needed the improvements for publication.Also, since editorial rejections rarely provide detailed suggestions for improvements, a common practice is to submit the paper unchanged to another journal.We consider that following the fate of editorial rejected papers that were resubmitted without changes to another journal is therefore a meaningful test to assess the predictive ability of editors.
On the other hand, we agree with the AOE that our sample is small.From the hundreds of emails that we sent, we received 65 answers.Nevertheless, it is plausible that the pattern observed in our small sample will persist when increasing the sample size.Considering sample size, the percent of editorially-rejected papers that were finally accepted in journals of comparable quality ranges between 54 and 78% (66% ± 12, confideence interval of 95%).Also, it is unlikely that increasing the sample size will change the observed pattern because our responses included authors in a wide range of citizenships and academic positions.In sum, we argue that our survey design was appropriate and, despite the small sample size, the conclusion that editors are poor oracles seems to be fairly reliable.To confirm or discard our conclusions, journals could trace the final destiny of their vast number of editorially rejected manuscripts to test the predictive capacity of their editorial board members.

Myths and reality about editorial rejections
The AOE argues that across the ESA family of journals, no submission is rejected based only upon the opinion of one person.Rather, the AOE sustains that editorial rejections in ESA journals occur only after review by two or more subject-matter editors, including at least one expert in the methods and systems.These multiple assessments minimize the effect of potential bias of basing the decision on the opinion of just one person (Schimel et al. 2014).We are glad that a team of editors performs editorial rejections in ESA, and we encourage other journals to follow similar practices.Unfortunately, our personal experience and those of the majority of our colleagues suggest that this noble procedure is not the rule in ecological journals (see also Cooke and Lapointe 2012).We want to illustrate this point with two selected anecdotes that describe why we doubt that editors often read in depth the entire manuscript before rejecting it, or that they consult the decision with other editors.1) MRC, a PhD with published papers in journals such as PlosOne, Ecology and Biology Letters, just finished an online submission of a new manuscript to a leading journal (not Science or Nature).With the satisfaction of having done a good job, MRC went to the bathroom, came back after 10 minutes, and astonished, read a new incoming e-mail announcing that the recently submitted paper was rejected without review.It sounds very unlikely that in exactly 10 minutes, the editor critically reviewed the manuscript, consulted the decision with colleagues, and wrote a rejection email back.In the second anecdote, WE, also a PhD with papers published in journals such as Evolution, PNAS and Nature, sent a new manuscript to another leading journal.Several weeks afterwards, and in the absence of news, WE wrote to the editorial office asking for the status of his manuscript.The editorial office answered they have no records of the paper, and assumed that the submitted manuscript got lost in a recent change of editors.The new editor apologized and encouraged WE to re-send the manuscript, which was finally published in that journal.Weeks after the paper was accepted, in the spam email folder, WE found a rejection letter from the former editor who originally received the manuscript.Evidently, that editor never shared her/his decision with colleagues or with the editorial team.It is also a striking example of how the same paper was editorially rejected by one but not by the other editor of the same journal.These anecdotes illustrate the unclear reasons and subjectivity of editorial rejections, making more ironic the new substitute term for "editorial rejection" suggested by the AOE: "reject following editorial review."Of course, these two examples are academic tales rather than robust quantitative data.The readers will judge whether superficial readings and subjectivity are the rule or the exception in the practice of editorial rejections.

The handling problem: many papers, few reviewers and the (low) relevance of a quick editorial rejection
The final criticism by the AOE is that leading journals receive more submissions than they can handle, mainly because of limitations in the availability of external reviewers.This assertion is based on the untested assumption that the published scientific literature increases at a higher rate than the scientific community.However, it is possible that a large portion of the increased submitted literature is generated by young researchers such as recent PhD students and postdocs-people who are perfectly capable of reviewing papers within their area of expertise (Donaldson et al. 2010).The "golden rule" of accepting to review two manuscripts per each paper submitted as first author may ensure the continuity of the peer-review system.Including the peer-review effort in the evaluation of the academic outputs of faculty (Veríssimo and Roberts 2013) and the application of other incentive policies (Lortie 2011, Hauser andFehr 2007) may also help getting reviewers.Of course, there may be justified cases of editorial rejections based on the opinion of a board of reviewing editors.But these cases should be a small percentage of the submitted manuscripts; high rates of editorial rejection should not be a reason for pride, or a proxy of journal´s quality.Conversely, academic journals should be proud of statements like "we sent the vast majority of our received manuscripts to external review for proper evaluation of their quality".Overall, the availability of potential reviewers should not be a limiting factor.Handling a vast number of papers could be hard, but not an impossible task for an editorial office.
The AOE also stated that many authors express appreciation for the rapid response of editorial rejection.We do not consider speed as an appreciated academic value.A quick response is meaningless if the decision is subjective and lacking constructive criticism that helps improve the manuscript (Wardle 2012).As discussed earlier, avoiding reviews is academically unfruitful for everybody, including journals, editors, referees, and of course, authors (Farji-Brener 2007).

"Rejecting" remarks
In sum, we argue that (a) the claim that editorial rejections are based on manuscripts not fitting within the journal's scope rather than on academic quality is unconvincing; (b) if academic quality is being assessed to decide the fate of a submitted paper, this attribute must be evaluated including several external opinions and not only the superficial reading of one person.Accordingly, our survey suggests that editors often fail in the task of correctly evaluating the suitability of a manuscript without the critical help of external reviewers; (c) the practice of sending the majority of submitted papers to external review should not cause the collapse of most popular journals.On the one hand, it is doubtful that the amount of available referees is a limiting factor.On the other hand, a system based on external reviews should be able to self-regulate: authors will send their better (not their worse) works to the most leading journals, avoiding unnecessary rejections.As discussed earlier, editors are key in the process of publishing scientific literature, enabling publication of manuscripts regarded to be of merit (Cooke and Lapointe 2012).We insist that for the sake of science, editors need the opinion of independent, external experts.In that sense, we agree with the AOE that editors are not oracles.Consequently, they certainly should not act as such.