layout: true --- # Sequence Learning ## How to Write a Peer Review Korbinian Riedhammer --- # Peer Reviews - play a vital role in ensuring quality and consistency in research -
-
- types of peer review - *open:* both authors and reviewers are known - *single-blind:* authors known, reviewers anonymous (most common) - *double-blind:* authors and reviewers anonymous - benefits and drawbacks? --- # Peer Reviews - journals are typically merit-based: - if it's a good paper, it will be published - journal quality typically measured in impact factor - conferences are typically ranked-merit-based: - the _N_ best papers will be accepted - conference quality typically measured in impact factor and acceptance rate - often a mix between linear scales, categories and free-form text - accept, minor changes, major changes, reject - novelty: 1-5 - comments to the author ??? - gender and personality bias - hard to hide identity, esp. in narrow fields (--> data!) --- # The First Read-Through _The first read-through is a skim-read for an initial impression._ - read the title and abstract; what is the paper about, which data and algorithms are used? - skim-read the remainder of the article - What is the main question addressed by the research? Is it relevant and interesting? - How original is the topic? What does it add to the subject area compared with other published material? - Is the paper well written? Is the text clear and easy to read? - Are the conclusions consistent with the evidence and arguments presented? Do they address the main question posed? - If the author is disagreeing significantly with the current academic consensus, do they have a substantial case? If not, what would be required to make their case credible? - If the paper includes tables or figures, what do they add to the paper? Do they aid understanding or are they superfluous? --- # Spotting Potential Major Flaws Watch out for potential flaws! - insufficient data or incorrect partitioning (training and test?) - statistically non-significant variations - unclear data tables or figures - contradictory data that either are not self-consistent or disagree with the conclusions - confirmatory data that adds little, if anything, to current understanding - unless strong arguments for such repetition are made --- # Concluding the First Read-Through - while reading, take notes: - are the introduction and related work sections well-written and organized?? - could you follow the explanations and conclusions? - are the data and methods appropriate? - is the paper technically sound? - does the data and evidence support the conclusions? - are references adequately - write down two paragraphs: - what are key strengths of the article, if any? - what is the main weakness of the article, if any? - would you accept or reject the paper based on these findings? _This part will be covered by our student peer review._ --- # The Second Read-Through If you decide the paper should be accepted, go over the paper again, this time in-depth. Make notes and comments in reference to line, figure or table numbers. Analyze section-by-section: - Introduction: topic, objective/hyopothesis and main contributions? - Related Work: complete and recent? - Data: description, available/accessible, partitioning? - Method: necessary detail to follow argument, technically sound? - Experiments: relevant to objective/hypothesis, replicable/reproducible, valid evaluation schedule, technical details? - Discussion and Conclusion: does the evidence support the conclusions? --- # The Second Read-Through (cont'd) Check for consistency and correctness: - are there any language issues or typos? - are there any unclear formulations that need improvement? - does the math check out? - are the references correctly cited? (authors, title, journal, year, ...) - are all figures and tables correctly referenced and described? - are there any factual errors or invalid arguments? --- # Sequence Learning 2020 Review Form Example paper \#31337 (see Moodle) Let's gauge... - technical merit - key strengths of the paper (paragraph) - main weakness of the paper (paragraph) - clarity of presentation - quality of references - reproducibility