Andrii Bidochko

• Updated: July 3, 2026
• 6 min read

FirstPass: Grounding AI Scientific Judgment in Multi-Round Editorial Outcomes

Direct Answer

The paper introduces FirstPass, a curated dataset of 3,668 multi‑round peer‑review dialogues from Nature Communications and a fine‑tuned language model that can generate realistic reviewer comments, update them across revision cycles, and predict whether a manuscript will require a standard or extended revision. It matters because it moves AI‑driven peer review from superficial style imitation toward genuine editorial judgment that respects the iterative nature of scientific validation.

Background: Why This Problem Is Hard

Academic publishing relies on a dialogue between authors, reviewers, and editors that unfolds over several rounds. This process is inherently iterative, with each review shaping the next revision and the final editorial decision. Existing AI‑based peer‑review tools suffer from three critical gaps:

• Domain narrowness: Most models are trained on computer‑science and machine‑learning venues, leaving them ill‑equipped to handle the language, methodology, and evaluation standards of biology, chemistry, physics, neuroscience, or earth science.
• Static snapshots: Current systems generate a single review and stop, ignoring the back‑and‑forth exchange that refines hypotheses, corrects methodological flaws, and clarifies claims.
• Stylistic focus: Benchmarks typically reward mimicry of reviewer tone or length, not the ability to anticipate editorial outcomes or suggest actionable improvements.

These shortcomings limit the practical utility of AI assistants for authors who need early, trustworthy feedback before submission, and for journals seeking to streamline editorial workloads without sacrificing rigor.

What the Researchers Propose

FirstPass tackles the problem on three fronts:

1. Dataset creation: By leveraging the mandatory transparent peer‑review policy of Nature Communications (effective November 2022), the authors harvested complete review‑revision‑review cycles across five scientific domains, ensuring 100 % content integrity through automated audits.
2. Model fine‑tuning: Using Low‑Rank Adaptation (LoRA) on the open‑source Qwen2.5‑7B‑Instruct model, they trained three tightly coupled tasks—review generation, reviewer updating, and revision‑cycle prediction.
3. Loss masking strategy: A novel “response‑only loss masking” technique forces the model to focus on the reviewer’s textual contribution rather than the surrounding manuscript context, dramatically improving predictive accuracy.

The resulting system behaves like a virtual reviewer who can:

• Draft an initial critique that mirrors expert language and depth.
• Incorporate author revisions and produce an updated review that reflects the new evidence.
• Forecast whether the next editorial round will be a standard revision (minor changes) or an extended revision (major overhaul).

How It Works in Practice

The FirstPass workflow can be broken down into four conceptual stages:

1. Input Ingestion

The author uploads a manuscript draft (PDF or LaTeX source). The system extracts the abstract, methods, results, and figures using a lightweight parsing module.

2. Initial Review Generation

Using the fine‑tuned Qwen2.5‑7B model, FirstPass produces a full‑length review (average ~1,200 words) that highlights methodological strengths, potential flaws, and suggestions for additional experiments or analyses.

3. Revision Loop

After the author revises the manuscript, the updated text is fed back into the model. The “reviewer updating” head re‑evaluates the changes, amends the original comments, and adds new insights where appropriate.

4. Outcome Prediction

The “revision‑cycle prediction” component classifies the next editorial decision as either “Standard” or “Extended.” This prediction is based on a learned mapping between reviewer sentiment, the magnitude of changes, and historical editorial outcomes in the FirstPass dataset.

What distinguishes FirstPass from prior attempts is the tight coupling of generation and prediction within a single fine‑tuned model, reinforced by the response‑only loss masking that forces the model to treat the reviewer’s voice as the primary learning signal.

Evaluation & Results

The authors evaluated FirstPass on two complementary fronts: predictive accuracy for revision outcomes and quality of generated reviews compared to human references.

Predictive Accuracy

• With loss masking, FirstPass achieved 80.5 % accuracy and an F1‑macro of 78.2 % in distinguishing standard from extended revisions.
• This outperformed the zero‑shot Gemini‑3.1‑flash‑lite‑preview by a margin of 10.4 percentage points.
• Statistical testing (McNemar, p < 0.001) confirmed the superiority of FirstPass over all baselines.

Review Generation Quality

• FirstPass produced reviews averaging 1,187 words, substantially closer to the human average of 2,155 words than any baseline.
• ROUGE‑L scores reached 0.154, a statistically significant improvement over Qwen and DeepSeek zero‑shot baselines (p < 0.001).
• Human evaluators rated FirstPass reviews as “more actionable” and “better aligned with domain‑specific expectations” than competing models.

These results demonstrate that FirstPass not only mimics the length and style of expert reviews but also captures the nuanced judgment needed to anticipate editorial decisions across diverse scientific fields.

Why This Matters for AI Systems and Agents

FirstPass offers a concrete blueprint for building AI agents that can participate meaningfully in collaborative, multi‑turn workflows—a capability that extends far beyond single‑shot text generation.

• Pre‑submission coaching: Researchers can run a “dry‑run” review before sending a manuscript to a journal, reducing the likelihood of costly revision cycles.
• Editorial triage: Journals could deploy FirstPass as an early‑screening assistant, flagging submissions that are likely to need extensive revisions and allocating editorial resources more efficiently.
• Agent orchestration: The modular design (generation + updating + prediction) aligns with emerging agent‑frameworks that require clear role separation and communication protocols.
• Cross‑domain robustness: Because the model was trained on biology, chemistry, neuroscience, physics, and earth science, it can be integrated into platforms serving multidisciplinary research teams.

For developers building AI‑enhanced scientific tools, FirstPass illustrates how to combine large‑language‑model fine‑tuning with domain‑specific, high‑integrity datasets to achieve both linguistic fluency and substantive judgment.

Explore how such capabilities can be embedded in broader automation pipelines on the UBOS platform overview or leveraged for targeted marketing insights with AI marketing agents.

What Comes Next

While FirstPass marks a significant step forward, several avenues remain open for research and productization:

• Broader journal coverage: Extending the dataset to include other open‑review journals will improve generalizability and capture discipline‑specific editorial nuances.
• Fine‑grained feedback: Future models could generate structured checklists (e.g., statistical power, data availability) that integrate directly with manuscript preparation tools.
• Human‑in‑the‑loop interfaces: Embedding FirstPass within collaborative platforms (e.g., via the Telegram integration on UBOS) would let authors receive real‑time reviewer suggestions while drafting.
• Multi‑modal extensions: Incorporating figure analysis, code snippets, or raw data evaluation could deepen the model’s ability to critique experimental design.
• Ethical safeguards: Transparent provenance tracking and bias audits are essential to ensure that AI‑generated reviews do not inadvertently reinforce disciplinary blind spots.

Developers interested in rapid prototyping can experiment with the Workflow automation studio to stitch FirstPass‑style agents into custom pipelines, or leverage the UBOS templates for quick start when building internal review assistants.

Ultimately, the vision is an ecosystem where AI agents act as trusted co‑authors—offering early critique, guiding revisions, and forecasting editorial outcomes—thereby accelerating scientific discovery while preserving the rigor of peer review.

Read the full research details in the original FirstPass paper.

Andrii Bidochko

CTO UBOS

Andrii Bidochko is an AI entrepreneur and researcher focused on AI agents, reinforcement learning, and autonomous systems. He writes about the technologies shaping the future of machine intelligence, from frontier models and agent architectures to real-world AI applications.