OpenRecently verified

Applied Engineer, Evaluations Role

  • Software Engineering
  • Platform: Mercor
  • Location: United States

$100-120/hrAs published by the platform.

Not sure this job fits you?

  • Add your resume and see every open Software Engineering job scored against your background. Free.
  • Your profile is never sent to the platforms. You decide where to apply.
Add your resume

About this job

You will build and refine evaluations to test the performance of frontier code models by transforming pull requests into engineering tasks. You will use a proprietary evaluation framework to validate how coding agents solve complex problems, identifying real weaknesses in their implementations. This role is for software engineers with solid fundamentals, capable of navigating substantial codebases and producing rigorous evaluations.

What you'll do

  • Identify repositories with complete and substantial pull requests, then convert them into challenging evaluation tasks.
  • Write task prompts and build evaluations using automated tests, shell commands, and LLM-based scoring criteria.
  • Validate evaluations against reference solutions and deliberately defective implementations to detect gaps and scoring errors.
  • Execute evaluations across multiple models and harnesses, then investigate the causes of failures and refine evaluation criteria.

Requirements

PythonJavaScriptTypeScriptJavaRustC++KotlinPHPRubyGitLLM

    Pay

    $100-120/hr

    Pay as published by the platform. It is not a guarantee of income or hours.

    Location

    Open to residents of: United States.

    How to apply

    You apply on work.mercor.com: create a profile, complete a skills assessment, then get matched to projects that fit your background.

    All Mercor jobs and how the platform works

    Source

    Official posting: https://work.mercor.com/jobs/list_AAABlzI3p6G_LDaGWuNNEa6l/applied-engineer-evaluations-role

    Last checked on November 20, 2026.

    Similar jobs

    • Software Engineering
    • Platform: Alignerr

    This role involves writing, debugging, and evaluating AI-generated code to help train and improve AI coding assistants. Software engineers will review code quality, identify bugs and security issues, provide detailed feedback, and design test scenarios to stress-test AI capabilities across various programming languages and frameworks.

    • Software Engineering
    • Platform: Alignerr
    • Level: Intermediate

    This remote contract role asks senior full-stack engineers to assess AI-generated code across frontend and backend, judging correctness, quality, and security. It also involves building internal tooling for data annotation and reviewing system designs. It suits experienced engineers who can give precise written feedback that helps AI models write better code.

    • Software Engineering
    • Platform: micro1
    • Location: 58 countries

    This role involves contributing to an AI training project as an open source software engineer. You will explore unfamiliar repositories, implement and debug front-end and back-end components, and evaluate patches generated by AI. The ideal candidate is an experienced developer with significant contributions to established open source projects.

    Posted September 16, 2026

    OpenRecently verified$100-150/hr

    Referral link to micro1's job list: search for this role there. Open this exact job

    • Software Engineering
    • Platform: Mercor
    • Location: United States
    • Level: Intermediate

    This role involves evaluating the quality, correctness, and reproducibility of software engineering tasks used to train and test the AI models of a leading laboratory. You audit tasks at the repository level, baseline patches, test harnesses, and scoring system integrity, providing structured feedback based on an evaluation grid.

    Posted August 28, 2026

    OpenRecently verified$70-90/hr

    Referral link: we may earn a fee. Apply without it

    • Software Engineering
    • Platform: Vetto

    This job involves checking whether code produced by AI models correctly solves a given software engineering prompt, and whether the accompanying tests really verify that behavior. It is aimed at experienced software engineers who can judge problem definitions and test suites with strict technical standards.

    Posted January 27, 2026

    OpenRecently verifiedPay not disclosed
    • Software Engineering
    • Platform: DataAnnotation

    You will evaluate AI-generated code by running models on real engineering tasks, analyzing their output against production standards, and stress-testing them to find failures. This role is ideal for experienced software engineers who want to contribute to AI model improvement through rigorous code review and red-teaming.

    Talent poolRecently verified$40-150/hr

    Keep exploring

    Apply on Mercor (opens Mercor in a new tab)

    This is a referral link: we may earn a fee if you are selected and meet the platform's conditions.

    Apply without the referral link