Quality Engineering

About the Role - Evaluation & Validation Engineer

As an

Evaluation & Validation Engineer , you'll evaluate, validate, and improve enterprise AI solutions to ensure they are accurate, reliable, secure, and ready for production. Working within NNIT's AI Center of Excellence, you'll establish evaluation frameworks, quality standards, and validation processes that build confidence in AI systems deployed across global organizations.

You'll collaborate with AI engineers, solution architects, consultants, product owners, and business stakeholders across Europe, Asia, and other regions to evaluate AI solutions throughout their lifecycle—from solution design and development to deployment, monitoring, and continuous improvement.

Success in this role requires analytical thinking, structured problem solving, and a strong understanding of AI evaluation methodologies, software quality engineering, and enterprise delivery.

Roles and Responsibilities: 1. Design AI Evaluation Frameworks Develop evaluation methodologies that measure AI quality, reliability, and business readiness. Design evaluation frameworks for enterprise AI applications. Define quality metrics, benchmarks, and acceptance criteria. Evaluate LLMs, AI agents, and Retrieval-Augmented Generation (RAG) solutions. Develop datasets and test cases for automated AI evaluations. Support Responsible AI and AI governance initiatives.

2. Build Validation & Quality Engineering Build scalable validation processes that improve AI quality and production readiness. Develop automated evaluation and regression testing frameworks. Detect hallucinations, model drift, and quality degradation. Improve AI reliability through continuous benchmarking and testing. Build reusable validation assets, test suites, and engineering standards. Recommend improvements to AI quality and engineering practices.

3. Partner with Stakeholders Collaborate with technical and business teams to ensure AI solutions meet business, operational, and regulatory expectations. Work closely with AI engineers, architects, consultants, and product owners. Translate business expectations into measurable evaluation criteria. Participate in solution reviews and production readiness assessments. Communicate evaluation results, quality findings, and technical risks. Support continuous improvement across the AI Center of Excellence.

What Success Looks Like Success in this role means you: Improve the quality, accuracy, and reliability of enterprise AI solutions. Reduce production issues through proactive evaluation and validation. Establish reusable evaluation frameworks and engineering standards. Increase stakeholder confidence in AI solutions. Support the continued growth and technical maturity of NNIT's AI Center of Excellence.

Qualifications: Technical Skills: At least 4 years of

Quality Engineering, Test Automation, Validation, AI Engineering, or Software Engineering

experience. Experience

designing automated testing or evaluation frameworks . Experience

validating enterprise software or AI applications . Experience working within Agile delivery environments. Experience working within consulting, enterprise IT, or global delivery organizations is highly desirable. OpenAI Evals Promptfoo DeepEval LangSmith Evaluation RAGAS TruLens Prompt Evaluation AI Quality Metrics Benchmark Testing Azure OpenAI Azure AI Foundry LangChain LangGraph AI Agents Retrieval-Augmented Generation (RAG) Python Git CI/CD REST APIs Test Automation Frameworks

Industry Experience Many of our customers operate in highly regulated industries where quality, traceability, compliance, and governance are essential for successful AI adoption. Experience supporting one or more of the following industries is advantageous: Life Sciences Pharmaceuticals Biotechnology Healthcare Medical Devices Banking & Financial Services Insurance Manufacturing Exposure to

Responsible AI ,

Computer System Validation (CSV) ,

GxP ,

FDA ,

EMA ,

HIPAA ,

GDPR ,

ISO 27001 ,

SOC 2 , or similar governance and compliance frameworks is beneficial but not required.

Why Join NNIT's AI Center of Excellence? Joining the AI Center of Excellence gives you the opportunity to build the data foundation that powers enterprise AI and digital transformation for global organizations while expanding your expertise across cloud data platforms, Artificial Intelligence, enterprise consulting, and global delivery. Build Solutions That Matter Work Across Borders Grow Your Expertise Shape the Future

WHO WE ARE

NNIT is a fast-growing IT company with ambitious goals and 1,600 employees. At NNIT, you can make your mark on some of the most challenging projects in the industry, on our journey forward as a company, and on your own career. You’ll work in agile teams with othe