AI Automation QA
Save this job and keep your search organized
Create a free account to save jobs, create alerts and return to this listing from your dashboard.
By continuing, you agree to our Terms & Privacy Policy.
- Definition and execution of testing and quality assurance strategies for AI‑enabled workflows
- Continuous evaluation and monitoring of system behavior in production environments
- Contribution to auditability, risk management, and continuous quality improvement Principal Responsibilities
- Define quality criteria and testing strategies for agent workflows, covering accuracy, latency, safety, compliance, and operational risk
- Build automated evaluation harnesses to assess agent performance, including hallucination rates, tool misuse, policy violations, and task success
- Implement continuous production monitoring to detect anomalies, quality degradation, and emerging safety concerns
- Develop and maintain automated test suites using Playwright for UI testing and custom scripts for API and workflow validation
- Apply LLM evaluation frameworks to assess output quality, regression, and system drift over time
- Produce and maintain dashboards and reports that communicate quality metrics and trends to engineering and stakeholders
- Develop and maintain runbooks for common failure modes and contribute to incident response activities
- Collaborate closely with developers to improve prompts, tool definitions, and workflow designs based on test results
- Ensure testing, logging, and monitoring practices align with data privacy, audit, and regulatory requirements
- Knowledge, Skills & Experience Essential
- Minimum 3 years’ experience in QA, test automation, or DevOps roles (or 2 years with direct experience testing AI or ML‑enabled systems)
- Strong Python skills for test automation, evaluation harnesses, and basic data analysis
- High attention to detail, with a focus on issues that materially impact reliability and user trust
- Comfort working with evolving tools, frameworks, and testing practices
- Collaborative mindset, using evidence‑based insights to influence product and engineering decisions Technical Skills (Required)
- Programming: Python (test automation, evaluation harnesses, data analysis)
- UI Automation: Playwright (end‑to‑end workflow testing)
- AI Evaluation: Deepeval, RAGAS, Evidently.
AI (LLM quality, drift, and regression analysis)
- Workflow Testing: API and agent workflow validation using custom scripts
- Monitoring: Production quality monitoring and anomaly detection Desirable
- Pytest or equivalent testing frameworks
- SQL for querying logs, metrics, or evaluation datasets
- Prometheus, Grafana, or similar monitoring tools
- Familiarity with hallucination detection and AI safety patterns
- CI/CD pipelines and Git‑based workflows *WTW is an Equal Opportunity Employer*
-
Senior Analyst
1 day ago
Pasig City, 00, Philippines NielsenIQ Full-timeJob DescriptionABOUT THIS ROLEThe Consulting Senior Analyst - CBI & Brand (Quantitative) supports the delivery of quantitative research projects in the Philippines, while contributing to broader initiatives across the SEA CBI& Brand Customer Success team.This role plays a key part in executing and delivering high-quality quantitative research, including...
-
Contact Center Quality Assurance Manager
1 day ago
Pasig City, 00, Philippines OneMCI Full-timePOSITION OVERVIEW MCI is one of the fastest-growing tech-enabled businesses services companies in the US, delivering Customer Experience (CX), Business Process Outsourcing (BPO) and technology solutions across multiple industries and countries worldwide. Our team use people, technology and innovation to help organizations improve customer...
-
Pasig City, 00, Philippines OneMCI Full-timePOSITION OVERVIEW MCI is one of the fastest-growing tech-enabled businesses services companies in the US, delivering Customer Experience (CX), Business Process Outsourcing (BPO) and technology solutions across multiple industries and countries worldwide. Our team use people, technology and innovation to help organizations improve customer...
-
Pasig City, 00, Philippines OneMCI Full-timePOSITION OVERVIEW At MCI we are committed to fostering an environment where professionals can build meaningful careers, access continuous learning and development opportunities and contribute to the success of a globally expanding, industry-leading organization. We are hiring an Experienced Client Services Manager to manage client...
-
Experienced Call Center Trainer
1 day ago
Pasig City, 00, Philippines OneMCI Full-timePOSITION OVERVIEW At MCI we are committed to fostering an environment where professionals can build meaningful careers, access continuous learning and development opportunities and contribute to the success of a globally expanding, industry-leading organization. We are looking for an Experienced Call Center Trainer to deliver high-impact...
-
Pasig City, 00, Philippines OneMCI Full-timePOSITION OVERVIEW At MCI we are committed to fostering an environment where professionals can build meaningful careers, access continuous learning and development opportunities and contribute to the success of a globally expanding, industry-leading organization. We are seeking an Experienced Operations Supervisor to lead a team of agents...
-
Contact Center Operations Supervisor
1 day ago
Pasig City, 00, Philippines OneMCI Full-timePOSITION OVERVIEW At MCI we are committed to fostering an environment where professionals can build meaningful careers, access continuous learning and development opportunities and contribute to the success of a globally expanding, industry-leading organization. We are looking for a results-driven Contact Center Operations Supervisor to...
-
Contact Center Client Services Manager
1 day ago
Pasig City, 00, Philippines OneMCI Full-timePOSITION OVERVIEW At MCI we are committed to fostering an environment where professionals can build meaningful careers, access continuous learning and development opportunities and contribute to the success of a globally expanding, industry-leading organization. We are looking for a client-focused and relationship-driven Client Services...
-
Call Center Team Lead
1 day ago
Pasig City, 00, Philippines OneMCI Full-timePOSITION OVERVIEW At MCI we are committed to fostering an environment where professionals can build meaningful careers, access continuous learning and development opportunities and contribute to the success of a globally expanding, industry-leading organization. Are you a highly motivated leader with excellent communication skills? We’re...
-
Customer Service Supervisor I
1 day ago
Pasig City, 00, Philippines OneMCI Full-timePOSITION OVERVIEW At MCI we are committed to fostering an environment where professionals can build meaningful careers, access continuous learning and development opportunities and contribute to the success of a globally expanding, industry-leading organization. Our Employee Development team is hiring experienced Customer Service...