Rank-order the following properties of the JOURNAL peer-review process in philosophy from BEST to WORST (1=best feature of the peer-review process, 26=worst feature of the peer-review process):
| 1. Good referee reports (sensible, helpful, charitable) (Condorcet winner: wins contests with all other choices) |
| 2. The process as a whole improving the quality of one’s work loses to Good referee reports (sensible, helpful, charitable) by 8–4 |
| 3. Journals that have established deadlines for reviews, improving turnaround times loses to Good referee reports (sensible, helpful, charitable) by 8–5, loses to The process as a whole improving the quality of one’s work by 8–5 |
| 4. Good editors who are sensitive to plight of authors and referees loses to Good referee reports (sensible, helpful, charitable) by 10–2, loses to Journals that have established deadlines for reviews, improving turnaround times by 10–3 |
| 5. Anonymized review giving authors a fair chance regardless of background or prestige loses to Good referee reports (sensible, helpful, charitable) by 8–2, loses to Good editors who are sensitive to plight of authors and referees by 7–2 |
| 6. Journals sharing referee feedback/verdicts with all reviewers, not just the author loses to Good referee reports (sensible, helpful, charitable) by 12–0, loses to Anonymized review giving authors a fair chance regardless of background or prestige by 5–4 |
| 7. The process improving one’s awareness of other works in paper’s area loses to Good referee reports (sensible, helpful, charitable) by 8–2, loses to Journals sharing referee feedback/verdicts with all reviewers, not just the author by 5–4 |
| 8. Bad behavior of authors (in R&R and editorial process) loses to Good referee reports (sensible, helpful, charitable) by 10–0, loses to The process improving one’s awareness of other works in paper’s area by 7–2 |
| 9. Revise-and-resubmits swamping authors & reviewers, needlessly delaying publication loses to Good referee reports (sensible, helpful, charitable) by 11–0, loses to Bad behavior of authors (in R&R and editorial process) by 5–4 |
| 10. Too many false positives (bad papers being published) loses to Good referee reports (sensible, helpful, charitable) by 10–0, loses to Bad behavior of authors (in R&R and editorial process) by 5–4 |
| 11. Too many journals only using doubly-anonymized review (permitting editing to know author’s identity) loses to Good referee reports (sensible, helpful, charitable) by 11–0, loses to Too many false positives (bad papers being published) by 6–2 |
| 12. Top journals are too biased in favor of highly abstract ‘core’ areas (metaphysics, epistemology, meta-ethics) loses to Good referee reports (sensible, helpful, charitable) by 12–0, loses to Too many journals only using doubly-anonymized review (permitting editing to know author’s identity) by 6–4 |
| 13. Desk-rejection is overused (rejecting papers that should go out to reviewers) loses to Good referee reports (sensible, helpful, charitable) by 11–0, loses to Top journals are too biased in favor of highly abstract ‘core’ areas (metaphysics, epistemology, meta-ethics) by 6–4 |
| 14. Desk-rejection is underused (letting too many bad papers go out to reviewers) loses to Good referee reports (sensible, helpful, charitable) by 11–0, loses to Desk-rejection is overused (rejecting papers that should go out to reviewers) by 7–2 |
| 15. Too many half-baked papers are placed under review loses to Good referee reports (sensible, helpful, charitable) by 12–0, loses to Desk-rejection is underused (letting too many bad papers go out to reviewers) by 6–4 |
| 16. Journals (editors and reviewers) are overly conservative philosophically loses to Good referee reports (sensible, helpful, charitable) by 11–0, loses to Desk-rejection is underused (letting too many bad papers go out to reviewers) by 5–4 |
| 17. Violations of anonymized review (referees Google reviewing or otherwise knowing paper’s author) loses to Good referee reports (sensible, helpful, charitable) by 9–0, loses to Journals (editors and reviewers) are overly conservative philosophically by 5–3 |
| 18. Editors and reviewers are overworked loses to Good referee reports (sensible, helpful, charitable) by 11–0, loses to Violations of anonymized review (referees Google reviewing or otherwise knowing paper’s author) by 5–3 |
| 19. Journals are overly conservative procedurally (deeming one negative review sufficient for rejection) loses to Good referee reports (sensible, helpful, charitable) by 12–0, loses to Editors and reviewers are overworked by 6–4 |
| 20. Papers getting ‘lost’ while under review (i.e. never placed under review) loses to Good referee reports (sensible, helpful, charitable) by 9–0, loses to Journals are overly conservative procedurally (deeming one negative review sufficient for rejection) by 5–3 |
| 21. The process is overly opaque (both to authors and reviewers) loses to Good referee reports (sensible, helpful, charitable) by 12–0, loses to Papers getting ‘lost’ while under review (i.e. never placed under review) by 5–3 |
| 22. Poor journal communication with authors (failure to explain why reviews are delayed, etc.) loses to Good referee reports (sensible, helpful, charitable) by 11–0, loses to The process is overly opaque (both to authors and reviewers) by 7–3 |
| 23. Too many false negatives (good papers being repeatedly rejected) loses to Good referee reports (sensible, helpful, charitable) by 11–0, loses to Poor journal communication with authors (failure to explain why reviews are delayed, etc.) by 6–4 |
| 24. Perfunctory reviews (e.g. several sentences that provide no argument for reviewer’s recommendation) loses to Good referee reports (sensible, helpful, charitable) by 11–0, loses to Too many false negatives (good papers being repeatedly rejected) by 7–3 |
| 25. Overly long turnaround times (due to delays/difficulties in finding reviewers + completion of reviews) loses to Good referee reports (sensible, helpful, charitable) by 12–0, loses to Too many false negatives (good papers being repeatedly rejected) by 6–4 |
| 26. Bad reviewers (incompetent, biased, aggressive, etc.) loses to Good referee reports (sensible, helpful, charitable) by 12–0, loses to Overly long turnaround times (due to delays/difficulties in finding reviewers + completion of reviews) by 8–3 |
For simplicity, some details of the poll result are not shown.
| 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 25 | 26 | ||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 1. Good referee reports (sensible, helpful, charitable) | - | 8 | 8 | 10 | 8 | 12 | 8 | 10 | 11 | 10 | 11 | 12 | 11 | 11 | 12 | 11 | 9 | 11 | 12 | 9 | 12 | 11 | 11 | 11 | 12 | 12 | |
| 2. The process as a whole improving the quality of one’s work | 4 | - | 8 | 8 | 8 | 10 | 8 | 10 | 10 | 10 | 11 | 11 | 10 | 11 | 12 | 10 | 9 | 11 | 12 | 9 | 12 | 11 | 11 | 11 | 12 | 12 | |
| 3. Journals that have established deadlines for reviews, improving turnaround times | 5 | 5 | - | 10 | 9 | 11 | 7 | 10 | 11 | 10 | 12 | 12 | 10 | 12 | 13 | 11 | 10 | 12 | 13 | 10 | 13 | 12 | 12 | 12 | 13 | 13 | |
| 4. Good editors who are sensitive to plight of authors and referees | 2 | 4 | 3 | - | 7 | 7 | 4 | 9 | 9 | 9 | 10 | 10 | 10 | 10 | 10 | 9 | 8 | 9 | 11 | 8 | 11 | 10 | 10 | 10 | 11 | 11 | |
| 5. Anonymized review giving authors a fair chance regardless of background or prestige | 2 | 2 | 2 | 2 | - | 5 | 4 | 7 | 7 | 7 | 9 | 8 | 7 | 8 | 9 | 7 | 8 | 9 | 8 | 8 | 9 | 8 | 8 | 8 | 9 | 9 | |
| 6. Journals sharing referee feedback/verdicts with all reviewers, not just the author | 0 | 2 | 2 | 4 | 4 | - | 5 | 9 | 8 | 9 | 9 | 10 | 7 | 10 | 10 | 9 | 8 | 9 | 10 | 8 | 11 | 10 | 10 | 10 | 11 | 11 | |
| 7. The process improving one’s awareness of other works in paper’s area | 2 | 2 | 4 | 5 | 4 | 4 | - | 7 | 8 | 8 | 8 | 8 | 8 | 8 | 9 | 8 | 8 | 8 | 9 | 8 | 9 | 9 | 9 | 9 | 9 | 9 | |
| 8. Bad behavior of authors (in R&R and editorial process) | 0 | 0 | 1 | 0 | 1 | 0 | 2 | - | 5 | 5 | 5 | 6 | 5 | 8 | 8 | 7 | 7 | 7 | 8 | 7 | 7 | 9 | 8 | 9 | 7 | 9 | |
| 9. Revise-and-resubmits swamping authors & reviewers, needlessly delaying publication | 0 | 1 | 1 | 1 | 1 | 2 | 1 | 4 | - | 4 | 7 | 6 | 7 | 7 | 7 | 9 | 7 | 7 | 8 | 6 | 9 | 10 | 8 | 10 | 8 | 9 | |
| 10. Too many false positives (bad papers being published) | 0 | 0 | 1 | 0 | 1 | 0 | 1 | 4 | 5 | - | 6 | 4 | 5 | 7 | 9 | 8 | 7 | 7 | 8 | 5 | 6 | 7 | 7 | 9 | 8 | 9 | |
| 11. Too many journals only using doubly-anonymized review (permitting editing to know author’s identity) | 0 | 0 | 0 | 0 | 0 | 1 | 0 | 3 | 2 | 2 | - | 6 | 5 | 4 | 5 | 6 | 5 | 6 | 6 | 6 | 8 | 8 | 6 | 8 | 6 | 8 | |
| 12. Top journals are too biased in favor of highly abstract ‘core’ areas (metaphysics, epistemology, meta-ethics) | 0 | 1 | 1 | 1 | 1 | 1 | 1 | 3 | 4 | 5 | 4 | - | 6 | 6 | 7 | 6 | 6 | 6 | 8 | 5 | 6 | 8 | 7 | 9 | 9 | 10 | |
| 13. Desk-rejection is overused (rejecting papers that should go out to reviewers) | 0 | 1 | 2 | 0 | 1 | 3 | 1 | 4 | 3 | 4 | 4 | 4 | - | 7 | 6 | 7 | 5 | 5 | 9 | 6 | 8 | 8 | 7 | 9 | 7 | 9 | |
| 14. Desk-rejection is underused (letting too many bad papers go out to reviewers) | 0 | 0 | 0 | 0 | 1 | 0 | 1 | 1 | 2 | 2 | 5 | 4 | 2 | - | 6 | 5 | 5 | 7 | 7 | 5 | 6 | 5 | 8 | 8 | 7 | 9 | |
| 15. Too many half-baked papers are placed under review | 0 | 0 | 0 | 1 | 0 | 1 | 0 | 1 | 3 | 0 | 5 | 4 | 4 | 4 | - | 5 | 4 | 7 | 9 | 4 | 6 | 6 | 7 | 9 | 8 | 11 | |
| 16. Journals (editors and reviewers) are overly conservative philosophically | 0 | 1 | 1 | 1 | 1 | 1 | 1 | 2 | 1 | 1 | 3 | 4 | 3 | 4 | 5 | - | 5 | 4 | 6 | 4 | 5 | 6 | 6 | 9 | 5 | 9 | |
| 17. Violations of anonymized review (referees Google reviewing or otherwise knowing paper’s author) | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 1 | 1 | 1 | 3 | 2 | 3 | 3 | 4 | 3 | - | 5 | 4 | 4 | 7 | 5 | 5 | 5 | 4 | 6 | |
| 18. Editors and reviewers are overworked | 0 | 0 | 0 | 1 | 0 | 1 | 0 | 1 | 2 | 1 | 4 | 4 | 4 | 2 | 3 | 5 | 3 | - | 6 | 4 | 6 | 5 | 6 | 7 | 7 | 9 | |
| 19. Journals are overly conservative procedurally (deeming one negative review sufficient for rejection) | 0 | 0 | 0 | 0 | 1 | 1 | 0 | 1 | 2 | 1 | 4 | 3 | 1 | 3 | 2 | 3 | 4 | 4 | - | 5 | 5 | 5 | 5 | 7 | 7 | 9 | |
| 20. Papers getting ‘lost’ while under review (i.e. never placed under review) | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 1 | 2 | 3 | 2 | 3 | 2 | 3 | 4 | 4 | 4 | 4 | 3 | - | 5 | 6 | 6 | 5 | 6 | 6 | |
| 21. The process is overly opaque (both to authors and reviewers) | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 2 | 1 | 3 | 2 | 5 | 2 | 4 | 5 | 5 | 1 | 4 | 6 | 3 | - | 7 | 8 | 7 | 7 | 9 | |
| 22. Poor journal communication with authors (failure to explain why reviews are delayed, etc.) | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 1 | 1 | 2 | 2 | 4 | 4 | 4 | 3 | 4 | 5 | 2 | 3 | - | 6 | 6 | 8 | 9 | |
| 23. Too many false negatives (good papers being repeatedly rejected) | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 1 | 2 | 2 | 3 | 3 | 3 | 1 | 3 | 4 | 3 | 3 | 4 | 2 | 2 | 4 | - | 7 | 6 | 8 | |
| 24. Perfunctory reviews (e.g. several sentences that provide no argument for reviewer’s recommendation) | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 1 | 1 | 1 | 1 | 1 | 1 | 3 | 2 | 3 | 3 | 3 | 4 | 3 | - | 5 | 8 | |
| 25. Overly long turnaround times (due to delays/difficulties in finding reviewers + completion of reviews) | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 2 | 2 | 1 | 4 | 2 | 3 | 3 | 3 | 5 | 4 | 3 | 4 | 2 | 4 | 2 | 4 | 5 | - | 8 | |
| 26. Bad reviewers (incompetent, biased, aggressive, etc.) | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 1 | 0 | 2 | 1 | 1 | 1 | 0 | 1 | 2 | 1 | 2 | 2 | 2 | 1 | 2 | 2 | 3 | - |
Ballot reporting was not enabled for this poll.
Feel like voting on something else? Try one of these public polls: