OpenRecently verified

SWE-Bench Task Auditor

  • Software Engineering
  • Platform: Mercor
  • Location: United States
  • Level: Intermediate

$70-90/hrAs published by the platform.

Not sure this job fits you?

  • Add your resume and see every open Software Engineering job scored against your background. Free.
  • Your profile is never sent to the platforms. You decide where to apply.
Add your resume

About this job

This role involves evaluating the quality, correctness, and reproducibility of software engineering tasks used to train and test the AI models of a leading laboratory. You audit tasks at the repository level, baseline patches, test harnesses, and scoring system integrity, providing structured feedback based on an evaluation grid.

What you'll do

  • Evaluate the quality and correctness of software engineering benchmark tasks
  • Audit baseline patches, test runners, and Docker isolation to detect information leakage or reward manipulation
  • Analyze tasks at the repository level and verify the integrity of the scoring system
  • Provide clear and structured written feedback based on an evaluation grid

Requirements

PythonTypeScriptJavaC++Docker
  • 3+ years of experience

Pay

$70-90/hr

Pay as published by the platform. It is not a guarantee of income or hours.

Location

Open to residents of: United States.

How to apply

You apply on work.mercor.com: create a profile, complete a skills assessment, then get matched to projects that fit your background.

All Mercor jobs and how the platform works

Source

Official posting: https://work.mercor.com/jobs/list_AAABoEpzuF806Dv0eAZAV5rw/swe-bench-task-auditor

Last checked on November 20, 2026.

Similar jobs

  • Software Engineering
  • Platform: Alignerr

This role involves writing, debugging, and evaluating AI-generated code to help train and improve AI coding assistants. Software engineers will review code quality, identify bugs and security issues, provide detailed feedback, and design test scenarios to stress-test AI capabilities across various programming languages and frameworks.

  • Software Engineering
  • Platform: Alignerr
  • Level: Intermediate

This remote contract role asks senior full-stack engineers to assess AI-generated code across frontend and backend, judging correctness, quality, and security. It also involves building internal tooling for data annotation and reviewing system designs. It suits experienced engineers who can give precise written feedback that helps AI models write better code.

  • Software Engineering
  • Platform: micro1
  • Location: 58 countries

This role involves contributing to an AI training project as an open source software engineer. You will explore unfamiliar repositories, implement and debug front-end and back-end components, and evaluate patches generated by AI. The ideal candidate is an experienced developer with significant contributions to established open source projects.

Posted September 16, 2026

OpenRecently verified$100-150/hr

Referral link to micro1's job list: search for this role there. Open this exact job

  • Software Engineering
  • Platform: Mercor
  • Location: United States
  • Level: Expert

This role invites experienced software engineers to evaluate and design migration strategies for complex codebases, particularly in mission-critical environments. You will analyze existing systems to identify safe modernization approaches and verify the quality of solutions generated by AI, while preserving functionality and security.

Posted August 22, 2026

OpenRecently verified$200/hr

Referral link: we may earn a fee. Apply without it

  • Software Engineering
  • Platform: Vetto

This job involves checking whether code produced by AI models correctly solves a given software engineering prompt, and whether the accompanying tests really verify that behavior. It is aimed at experienced software engineers who can judge problem definitions and test suites with strict technical standards.

Posted January 27, 2026

OpenRecently verifiedPay not disclosed
  • Software Engineering
  • Platform: Mercor
  • Location: United States

You will build and refine evaluations to test the performance of frontier code models by transforming pull requests into engineering tasks. You will use a proprietary evaluation framework to validate how coding agents solve complex problems, identifying real weaknesses in their implementations. This role is for software engineers with solid fundamentals, capable of navigating substantial codebases and producing rigorous evaluations.

Posted June 2, 2025

OpenRecently verified$100-120/hr

Referral link: we may earn a fee. Apply without it

Keep exploring

Apply on Mercor (opens Mercor in a new tab)

This is a referral link: we may earn a fee if you are selected and meet the platform's conditions.

Apply without the referral link