AI Quality Engineering Specialist

Overview

We are seeking a proactive leader to architect and uphold high-quality annotation workflows so that AI models are built and evaluated on reliable data without compromising on speed or efficiency.

The Role:

• Lead and mentor a team of Data Quality Analysts, setting clear quality goals, delivering feedback, and fostering a culture of precision and accountability.
• Develop and continually refine robust QA processes, SOPs, and statistical quality metrics (e.g., F1 score, inter-annotator agreement) to protect the integrity of annotation outputs.
• Drive transparency and insight: Create dashboards and reports that reveal quality trends, root causes of errors, and improvement opportunities - communicating these insights to leadership and clients.
• Champion tool innovation and efficiency: Manage annotation and QA platforms (like Labelbox, Dataloop, LabelStudio), and lead the evaluation or implementation of new automation tools to elevate efficiency and maintain quality.

Key Responsibilities:

• Drive the development, refinement, and documentation of quality assurance processes and standard operating procedures to ensure high-quality outputs.
• Establish comprehensive quality metrics (e.g. F1 score, inter-annotator agreement) that align with business objectives and industry standards.
• Continuously review and refine annotation workflows to proactively identify risks and areas to increase efficiency and reduce errors.
• Act as the subject matter expert on annotation quality, providing ongoing feedback, training, and support to annotators and project teams to uphold the highest quality standards.

Required Skills & Qualifications:

• 3+ years of experience in data quality management, data operations, or related roles within AI/ML or data annotation environments.
• Proven track record in designing and executing quality assurance strategies for large-scale, multi-modal data annotation projects.
• Proven track record in a leadership role managing and developing high-performing, remote or distributed teams.
• Deep understanding of data annotation processes, quality assurance methodologies, and statistical quality metrics (e.g., F1 score, inter-annotator agreement).
• Strong data-analysis skills, with the ability to interrogate large datasets, perform statistical analyses, and translate findings into actionable recommendations.
• Excellent communication skills, with experience presenting complex data and quality insights to technical and non-technical stakeholders.
• Proficiency with annotation and QA tools (e.g., Labelbox, Dataloop, LabelStudio).
• High-level of proficiency in common data-analysis tools, such as Excel and Google Sheets.
• Familiarity with programmatic data analysis techniques (e.g. Python, SQL).
• Familiarity with the core concepts of AI/ML pipelines, including data preparation, model training, and evaluation.

Benefits:

• Remote-first flexibility: Work from anywhere you're most productive.
• Every challenge is new: Work on first-of-their-kind problems at the frontier of AI safety.
• Visible impact: Your architectures directly influence how the world's leading AI teams mitigate risk.
• Room to grow: Build the solutions function from scratch and scale into leadership as revenue follows.
• Mission-driven team: Join sellers, researchers, and engineers united by a passion for safer AI.

Other Information:

TaskUs is committed to providing equal access to opportunities. If you need reasonable accommodations in any part of the hiring process, please let us know.