ATS Checker Testing Methodology: Sample Set and Scoring Protocol

This methodology defines how GradVix should test ATS resume checkers using controlled resumes, job descriptions, repeat runs and documented reviewer decisions before publishing any comparative benchmark.

1. Benchmark objectives

The benchmark should answer narrow, testable questions rather than attempt to prove which product reproduces every employer ATS.

2. Controlled resume sample set

Version 1.0 should use at least twelve synthetic or fully anonymised resume profiles. Each profile must have a written ground-truth record stating what is accurate, deliberately missing, weak, strong or unsupported.

3. Resume document variants

Each base profile should be exported into controlled variants. The factual content must remain constant unless the variant is specifically testing an evidence change.

4. Job-description controls

Each role should use at least two job descriptions:

• JD-A: a realistic, suitable vacancy aligned with the candidate’s level.
• JD-B: a related but materially different vacancy that should create a lower role-match result.

The complete text must be saved with source URL, access date, role title, location, experience range, mandatory conditions and preferred conditions. Remove employer identity only when publication requires anonymisation; do not rewrite the requirements during testing.

5. Standard test procedure

6. Benchmark metrics

7. Human-review protocol

At least two reviewers should independently assess each report against the ground-truth file. Reviewers should not know the intended brand ranking while coding findings.

1. Mark each tool finding as correct, partly correct, incorrect or unverifiable.
2. Mark each known seeded issue as detected, partly detected or missed.
3. Record whether the recommendation can be followed without inventing information.
4. Resolve disagreements through a documented adjudication note.
5. Preserve both the original reviewer decisions and the final resolved decision.

When reviewer judgement remains uncertain, the published result must show the uncertainty rather than forcing a definitive score.

8. Privacy and data controls

• Use synthetic resumes wherever possible.
• For real resumes, obtain informed permission and remove names, contact details, identity numbers, addresses and confidential employer information.
• Do not upload Aadhaar, PAN, passport, bank, medical or authentication information.
• Record each service’s visible retention, deletion and provider-processing disclosures.
• Delete benchmark accounts or test data after the retention requirement is complete where the product permits it.

9. Publication and reporting rules

Every published benchmark should include:

• testing dates and methodology version;
• tools and account tiers tested;
• sample size and role coverage;
• document and job-description variants;
• raw category results or a sufficiently detailed appendix;
• reviewer method and disagreement handling;
• known limitations, failures and unavailable features;
• commercial relationships or affiliate interests;
• separate product-quality scores and resume ATS scores; and
• a correction log when a product or result changes.

10. Required limitations

• The benchmark cannot access private employer configurations.
• AI-backed tools may change without a visible version number.
• Scores may vary by account tier, file type, traffic conditions or product update.
• Reviewer judgement can introduce subjectivity despite a ground-truth record.
• A test set cannot represent every Indian role, language, industry or experience level.
• Strong benchmark performance does not guarantee an interview or hiring result.

Frequently asked questions