Your Journal Has Peer Review, but Your Process Is Not Transparent or Rigorous

You have editors reviewing submissions. You have reviewers providing feedback. But when potential authors look at your journal's policies, they cannot find clear information about your peer review process. When DOAJ or Scopus evaluates your journal, they cannot confirm that your review process meets standards for rigor and impartiality.

The problem is not that peer review is not happening. The problem is that it is not documented, not transparent, and possibly not configured optimally in OJS. A journal that conducts peer review but does not document the process, or does not use OJS's built-in review workflow tools, signals to indexing services and serious researchers that the publication lacks editorial rigor.

Double-blind peer review, where both authors and reviewers remain anonymous to each other, is the gold standard in academic publishing. It reduces bias, improves fairness, and is expected by DOAJ, Scopus, and other quality indexes.

This guide walks you through configuring double-blind peer review in OJS 3 from scratch, including reviewer anonymity settings, review forms, email templates, and workflow best practices, written by OJS specialists who have configured peer review for journals across 20+ countries.

If you want an expert to configure your OJS peer review workflow correctly, visit ojsguru.com for a free consultation.

Understanding OJS Peer Review Architecture

OJS organizes the peer review process into distinct stages and roles. Understanding this structure is essential before configuring it.

Review Stages in OJS:

OJS divides the submission workflow into stages:

  • Submission, author submits manuscript
  • Review, peer review stage (this is where we focus)
  • Copyediting, accepted articles are edited for publication
  • Production, edited articles are formatted for publication
  • Published, article appears in the journal

Reviewer Roles:

  • Reviewers, external experts who evaluate submissions
  • Section Editors, oversee reviewers and make editorial recommendations
  • Editors-in-Chief, make final acceptance/rejection decisions

Review Types Supported by OJS:

  • Single-blind review, authors anonymous to reviewers, but reviewers known to authors
  • Double-blind review, both authors and reviewers anonymous
  • Open review, with no anonymity, all names visible
  • Custom, hybrid approaches

Step 1: Verify Your Review Stage Is Enabled

Before configuring peer review, confirm the Review stage exists in your workflow.

Go to Workflow Settings → Submission Stages

You should see five stages listed in order:

  1. Submission
  2. Review
  3. Copyediting
  4. Production
  5. Published

If the Review stage is missing, click Restore Default Stage Order to add it back.

Step 2: Configure Your Journal's Review Type

Go to Workflow Settings → Review → Review Type

Select one of these options:

Double-Blind Review (recommended for most journals)

Authors and reviewers remain anonymous to each other.
Reviewers cannot see author names.
Authors cannot see reviewer names.

This is the most common and most rigorous approach.

Single-Blind Review

Authors see reviewer names.
Reviewers cannot see author names.

Less common but used by some journals where reviewer transparency is valued.

Open Review

Both authors and reviewers see each other's names.
Least common. Used by some journals prioritizing transparency over anonymity.

For DOAJ and Scopus compliance, select Double-Blind Review.

Step 3: Set Up Reviewer Guidelines and Instructions

Reviewers need clear instructions on what you expect from them.

Go to Workflow Settings → Review → Reviewer Guidelines

Write detailed guidelines for your reviewers:

REVIEWER GUIDELINES

Thank you for agreeing to review this manuscript for [Journal Name].

Review Timeline:
Please complete your review within 2 weeks of receiving this invitation.
If you need more time, contact the Section Editor immediately.

What to Evaluate:
1. Originality, Is this original research or a novel contribution?
2. Significance, Does it advance the field meaningfully?
3. Methodology, Are the methods sound and appropriate?
4. Results, Do the results support the conclusions?
5. Clarity, Is the manuscript well-written and organized?
6. Fit, Does this align with the journal's scope?

Your Review Should Include:
- A summary of the main contributions
- Strengths of the manuscript
- Weaknesses and areas for improvement
- Specific suggestions for revision
- A recommendation (Accept, Minor Revisions, Major Revisions, Reject)

Reviewer Confidentiality:
Your identity will remain confidential unless you request attribution.
You will not see the author's identity.
The author will not see your identity.

Questions? Contact the Section Editor at [email].

Save these guidelines. Reviewers will see them when invited to review.

Step 4: Configure Review Forms and Questions

OJS uses structured review forms to guide reviewer feedback. Custom forms ensure consistency and capture the specific information you need.

Go to Workflow Settings → Review → Review Forms

Click Create Review Form

Create a form with these sections:

Section 1, Manuscript Quality

Add questions:

Question 1:

Type: Dropdown
Question: Is this manuscript original?
Options: Yes / No / Partially
Required: Yes

Question 2:

Type: Dropdown
Question: Are the methods appropriate and rigorous?
Options: Yes / No / Needs Improvement
Required: Yes

Question 3:

Type: Dropdown
Question: Do the results support the conclusions?
Options: Yes / No / Partially
Required: Yes

Section 2, Detailed Comments

Question 4:

Type: Large Text Area
Question: Please provide detailed comments on the manuscript's strengths and weaknesses.
Include specific suggestions for improvement.
Required: Yes

Section 3, Overall Recommendation

Question 5:

Type: Dropdown
Question: What is your recommendation?
Options: 
  - Accept
  - Minor Revisions
  - Major Revisions
  - Reject
Required: Yes

After creating the form, go back to Review Settings and set this form as your default review form.

Step 5: Set Up Your Editorial Team and Reviewer List

Before assigning reviews, establish your editorial structure.

Add Section Editors:

Go to Users & Roles → Users

Create editor accounts:

  • Journal Manager (highest level, can edit all settings)
  • Editor-in-Chief (makes final decisions)
  • Section Editors (manage specific sections and reviewers)

Assign each person to their role with appropriate permissions.

Maintain Your Reviewer Database:

Go to Users & Roles → Users

Create accounts for all active reviewers with the role Reviewer.

Record:

  • Full name
  • Email address
  • Expertise areas (journals, topics they can review)
  • Maximum number of simultaneous reviews they will accept

Step 6: Configure Reviewer Assignment Settings

Go to Workflow Settings → Review → Reviewer Selection

Configure:

Limit Reviewer Access:

Recommend limiting reviewers to view only their assigned submissions.
Disable: Allow reviewers to view all submissions in the journal.

Reviewer Recommendations:

Should editors be able to suggest reviewers during submission review?
Yes,  enables editors to recommend reviewers they trust.

Competing Interests:

Require reviewers to declare competing interests.
Yes,  reviewers must disclose conflicts before accepting.

Save these settings.

Step 7: Configure Review Workflow Stages

Go to Workflow Settings → Review → Review Stages

This controls how many review rounds occur and what happens in each.

Standard Configuration, Single Review Round:

Leave the default single round configuration:

Round 1 of 1
Reviewers: 2
Decision: Accept/Reject/Request Revisions

For Journals Requiring Revision, Two-Round Configuration:

Create two rounds:

Round 1 of 2
Reviewers: 2
Reviewers see each other's reviews: No
Decision: Request Major Revisions / Request Minor Revisions / Reject

Round 2 of 2
Reviewers: 1 (rereviews after revisions)
Reviewers see previous reviews: Yes
Decision: Accept / Reject

Step 8: Set Up Review Request Email Templates

Reviewers need clear invitation emails. Configure the email template they receive.

Go to Workflow Settings → Email → Email Templates

Find REVIEW_REQUEST and edit it:

Subject: [Journal Name] Reviewer Invitation

Dear [Reviewer Name],

You are invited to review the manuscript "[Article Title]" 
submitted to [Journal Name].

Topic: [Article Keywords]
Manuscript Abstract: [Abstract]

We are asking you to:
1. Review the manuscript using our review form
2. Provide constructive feedback to help the author improve the work
3. Make a clear recommendation (Accept / Revisions / Reject)
4. Complete your review within 2 weeks

Your review will be completely confidential. The author will not 
know your identity.

Review deadline: [Deadline Date]

To access the manuscript and submit your review, log in at:
[Journal URL]

Questions? Contact: [Editor Email]

Thank you for contributing to [Journal Name].

Best regards,
[Journal Name] Editorial Team

Ensure the email clearly explains the anonymity guarantee.

Step 9: Assign Reviewers to a Submission

When a submission is ready for review:

  1. Go to the submission in the Review stage
  2. Click Select Reviewers
  3. Search for and select your reviewers (typically 2-3)
  4. Send the review request

OJS automatically:

  • Hides author information from reviewers (double-blind setup)
  • Hides reviewer names from the author
  • Sends the review invitation email
  • Tracks review deadlines and status

Step 10: Monitor Review Progress and Chase Overdue Reviews

Go to Reviews in your submission dashboard

You will see:

  • All invited reviewers and their status
  • Acceptance/decline status
  • Review completion status
  • Days remaining until deadline

When a reviewer hasn't completed their review within the deadline:

Click Send Reminder to automatically send a deadline reminder email.

When a reviewer declines:

Click Select Replacement to invite a new reviewer.

Step 11: Managing Revision Rounds

After reviewers submit their feedback:

  1. Go to Editing tab in the submission
  2. View reviewer comments (authors see comments with reviewer names hidden)
  3. Make an editorial decision: Accept / Request Minor Revisions / Request Major Revisions / Reject
  4. Send the decision letter to the author

If revisions are requested:

The submission returns to the Review stage for a second round. The author uploads a revised manuscript with a revision response letter explaining changes made.

For major revisions, send to new reviewers. For minor revisions, the original reviewers can re-review.

Step 12: Configure Revision Request Email Template

Go to Workflow Settings → Email → Email Templates

Find REVISION_REQUEST and customize:

Subject: Revision Requested, [Article Title]

Dear [Author Name],

Thank you for submitting "[Article Title]" to [Journal Name].

The editorial team and peer reviewers have evaluated your manuscript.
We invite you to revise and resubmit addressing the reviewer comments.

Reviewer Feedback:
[Reviewer 1 Comments]
[Reviewer 2 Comments]

Revision Instructions:
1. Revise your manuscript addressing each comment
2. Prepare a revision response letter explaining changes
3. Resubmit through the journal system

Revision deadline: [Date, typically 4-6 weeks]

Questions? Contact: [Editor Email]

We look forward to your revised submission.

Best regards,
[Journal Name]

Common Peer Review Configuration Mistakes

Mistake 1: Reviewer names visible to authors

Authors see who reviewed their submission, defeating anonymity.

Fix:

  • Go to Workflow Settings → Review → Review Type
  • Confirm Double-Blind Review is selected
  • Test by logging in as an author and viewing reviewer feedback

Mistake 2: Review form not assigned as default

Reviewers receive no structured form and provide uneven feedback.

Fix:

  • Create a review form in Review Forms
  • Go to Review Settings and select the form as default
  • Assign it to the Review stage

Mistake 3: No review guidelines

Reviewers do not know expectations, quality is inconsistent.

Fix:

  • Write detailed reviewer guidelines
  • Include timeline, evaluation criteria, and expectations
  • Reference them in invitation emails

Mistake 4: Reviewer invitations are unclear about anonymity

Reviewers are uncertain whether their identity is protected.

Fix:

  • Edit the REVIEW_REQUEST email template
  • Include clear language: "Your identity will be kept completely confidential from the author"
  • Explain the author's name is hidden from reviewers

Mistake 5: No tracking of review deadlines

Reviews stall indefinitely with no follow-up.

Fix:

  • Set clear review deadlines when inviting reviewers
  • Use the Send Reminder function at deadline
  • Track all pending reviews in your submission dashboard

Best Practices for Production Peer Review

Set realistic reviewer deadlines

Aim for 2-3 weeks. Longer deadlines reduce completion rates. Shorter deadlines rush the process.

Invite 2-3 reviewers per submission

If one declines or misses deadline, you have backup.

Provide detailed review forms

Structured forms improve review quality and consistency.

Keep reviewer load reasonable

Track how many active reviews each reviewer has. Do not overload.

Thank reviewers explicitly

Send a thank-you email after they complete reviews. Consider annual recognition.

Hide all identifying information from authors

Remove reviewer names, affiliations, and identifying comments from author-facing feedback.

When to Call in a Professional

OJS peer review configuration is complex with many interdependent settings. Consider professional help when:

  • Your current review process is not documented in OJS
  • You want to transition from informal to structured peer review
  • You are preparing for DOAJ or Scopus application and need to verify review rigor
  • Your review process is unclear or inconsistent across submissions
  • You want an expert to audit your setup and recommend improvements

OJS Guru configures complete peer review workflows, from review forms through editor instructions through reviewer tracking. We ensure your process is transparent, rigorous, and fully leverages OJS's workflow tools.

👉 Get a free consultation at ojsguru.com

Summary

Double-blind peer review is the foundation of journal credibility. Configuring it correctly in OJS involves 12 key steps:

  1. Verify your Review stage is enabled
  2. Configure double-blind review type
  3. Set up reviewer guidelines
  4. Create review forms and questions
  5. Establish editorial team and reviewer database
  6. Configure reviewer assignment settings
  7. Set up review workflow stages
  8. Customize review request email template
  9. Assign reviewers to submissions
  10. Monitor review progress
  11. Configure revision request email template
  12. Follow best practices for ongoing management

Done correctly, your peer review process will be transparent, documented, and fully leveraged by DOAJ and Scopus evaluators, giving your journal the credibility it deserves.

If you want an expert to configure your OJS peer review workflow, contact OJS Guru at ojsguru.com. We set up peer review for journals regularly and we'll ensure yours meets the highest standards.

OJS Guru is a professional Open Journal Systems service provider specializing in OJS installation, customization, migration, and technical support for research journal publishers in 20+ countries. Visit ojsguru.com to request a free consultation.