Evaluation Scenario Writer

2 weeks ago

Manila, National Capital Region, Philippines Mindrift Full time

This opportunity is only for candidates currently residing in the specified country. Your location may affect eligibility and rates. Please submit your resume in English and indicate your level of English.

At Mindrift, innovation meets opportunity. We believe in using the power of collective human intelligence to ethically shape the future of AI. 

What we do

The Mindrift platform connects specialists with AI projects from major tech innovators. Our mission is to unlock the potential of Generative AI by tapping into real-world expertise from across the globe.

About the Role

We're looking for someone who can design realistic and structured evaluation scenarios for LLM-based agents. You'll create test cases that simulate human-performed tasks and define gold-standard behavior to compare agent actions against. You'll work to ensure each scenario is clearly defined, well-scored, and easy to execute and reuse. You'll need a sharp analytical mindset, attention to detail, and an interest in how AI agents make decisions.

Although every project is unique, you might typically:

  • Create structured test cases that simulate complex human workflows.
  • Define gold-standard behavior and scoring logic to evaluate agent actions.
  • Analyze agent logs, failure modes, and decision paths.
  • Work with code repositories and test frameworks to validate your scenarios.
  • Iterate on prompts, instructions, and test cases to improve clarity and difficulty.
  • Ensure that scenarios are production-ready, easy to run, and reusable.

How to get started

Simply apply to this post, qualify, and get the chance to contribute to projects aligned with your skills, on your own schedule. From creating training prompts to refining model responses, you'll help shape the future of AI while ensuring technology benefits everyone.

  • Bachelor's and/or Master's Degree in Computer Science, Software Engineering, Data Science / Data Analytics, Artificial Intelligence / Machine Learning, Computational Linguistics / Natural Language Processing (NLP), Information Systems or other related fields.
  • Background in QA, software testing, data analysis, or NLP annotation.
  • Good understanding of test design principles (e.g., reproducibility, coverage, edge cases).
  • Strong written communication skills in English.
  • Comfortable with structured formats like JSON/YAML for scenario description.
  • Can define expected agent behaviors (gold paths) and scoring logic.
  • Basic experience with Python and JS.
  • Curious and open to working with AI-generated content, agent logs, and prompt-based behavior.

Nice to Have

  • Experience in writing manual or automated test cases.
  • Familiarity with LLM capabilities and typical failure modes.
  • Understanding of scoring metrics (precision, recall, coverage, reward functions).

Contribute on your own schedule, from anywhere in the world. This opportunity allows you to:

  • Get paid for your expertise, with rates that can go up to $12/hour depending on your skills, experience, and project needs.
  • Take part in a flexible, remote, freelance project that fits around your primary professional or academic commitments.
  • Participate in an advanced AI project and gain valuable experience to enhance your portfolio.
  • Influence how future AI models understand and communicate in your field of expertise.

  • Food Writer

    1 day ago

    Manila, National Capital Region, Philippines Carnivore Style Full time

    Job Summary:We need a food writer to create engaging and informative content about meat and cooking. You will be responsible for writing articles, recipes, reviews, and stories that showcase our brand, products, and services, as well as the benefits of a carnivore diet and lifestyle. You will also research and interview sources, edit and proofread your...

  • Content Writer

    4 days ago

    Manila, National Capital Region, Philippines Nas Company Full time

    Job Title:Content WriterTeam:Support OperationsLocation:Remote / HybridReports to:Enablement Manager / Head of SupportAbout helps business owners and entrepreneurs build thriving online communities. We're building the tools and support systems that empower people to connect, grow, and build sustainable businesses. As we scale, we're looking for a versatile...

  • Technical Writer

    1 week ago

    Manila, National Capital Region, Philippines PSC Biotech Full time

    Job InformationDate Opened12/02/2025Job TypeFull timeWork Experience4-5 yearsIndustryPharmaCityMakati Commercial CenterState/ProvinceNational Capital Region (Manila)CountryPhilippinesJob DescriptionPSC Software, a division of PSC Biotech, is a leading provider of cutting-edge, cloud-based quality management and regulatory compliance solutions. Our software...

  • Credit Analyst/Loan Writer

    7 days ago

    Manila, National Capital Region, Philippines Hunt St Full time A$2,200 - A$3,000

    ​​Looking for Filipino (Philippines-based) candidatesJob Role: Credit Analyst/Loan WriterCompensation Range: AUD per monthEngagement type: Independent Contractor Agreement Work Schedule: This role is expected to align with the AU business hours (approx. 9 AM - 5 PM, Monday to Friday) for collaboration, but as a contractor, you'll have flexibility in...

  • Credit Analyst/Loan Writer

    4 days ago

    Manila, National Capital Region, Philippines Hunt St Full time

    ​​Looking for Filipino (Philippines-based) candidatesJob Role: Credit Analyst/Loan WriterCompensation Range: AUD per monthEngagement type: Employee of Record Work Schedule: This role is expected to align with the AU business hours (approx. 9 AM - 5 PM, Monday to Friday) for collaboration, but as a contractor, you'll have flexibility in how you...

  • Credit Analyst/Loan Writer

    6 days ago

    Manila, National Capital Region, Philippines Hunt St Full time

    ​​Looking for Filipino (Philippines-based) candidatesJob Role: Credit Analyst/Loan WriterCompensation Range: AUD per monthEngagement type: Independent Contractor Agreement Work Schedule: This role is expected to align with the AU business hours (approx. 9 AM - 5 PM, Monday to Friday) for collaboration, but as a contractor, you'll have flexibility in...

  • Podcast Editor

    6 days ago

    Manila, National Capital Region, Philippines Carnivore Style Full time

    The Role: Carnivore Style – Your #1 Resource on All Things Meat-Related – is looking for a Podcast EditorJoin us – salary ranges from $1,500 to $2,000 per month, plus other perks and benefits. Job Summary:You are skilled and reliable when it comes to editing audio content for podcast episodes. Obviously, creative too.You will be responsible for...

  • Strategic Finance Manager

    1 week ago

    Manila, National Capital Region, Philippines Anchored Solutions MNL, Inc. Full time

    ***Note: This position is open to applicants based in the Philippines only.DUTIES AND RESPONSIBILITIESThis position is responsible for the following:Lead Strategic Finance initiatives in partnership with Growth, Marketing & Partnerships, Enterprise, and Sales to develop business models, evaluate commercialization paths, and drive data-informed...

  • Security - Special Projects Officer - Support,

    2 weeks ago

    Manila, National Capital Region, Philippines SPX Express Full time

    Job DescriptionThis role supports the Sub-lead for Special Projects in reviewing, developing, and documenting security policies, procedures, and operational measures for the SPX Security & Investigation Team. It involves actively participating in operational excellence initiatives, conducting security assessments and red teaming exercises, overseeing...

  • National Maintenance Estimator

    2 weeks ago

    Manila, National Capital Region, Philippines Atlan Stormwater Full time

    The Role:An intermediate role in Atlan Stormwater (AU) to support the National Maintenance Business Development Managers to deliver timely estimates for projects. The role will evaluate different scenarios and proactively seek to add value to the maintenance sales process.Preparing quotations and contracts for national maintenance projects.Create...