Sr. Engineer, AI Model Evaluation

Join or sign in to find your next job Join to apply for the Sr. Engineer, AI Model Evaluation role at Lenovo We are Lenovo. We do what we say. We own what we do. We WOW our customers. Lenovo is a US$69 billion revenue global technology powerhouse, ranked #196 in the Fortune Global 500, and serving millions of customers every day in 180 markets. Focused on a bold vision to deliver Smarter Technology for All, Lenovo has built on its success as the world’s largest PC company with a full-stack portfolio of AI-enabled, AI-ready, and AI-optimized devices (PCs, workstations, smartphones, tablets), infrastructure (server, storage, edge, high performance computing and software defined infrastructure), software, solutions, and services. Lenovo’s continued investment in world-changing innovation is building a more equitable, trustworthy, and smarter future for everyone, everywhere. Lenovo is listed on the Hong Kong stock exchange under Lenovo Group Limited (HKSE: 992) (ADR: LNVGY). Lenovo’s AI Technology Center (LATC) is the global engine powering our hybrid-AI vision. As Lenovo’s AI Center of Excellence, we’re building a world-class team to define the next era of computing—powered by AI. We’re tackling some of the most exciting challenges in AI: Scaling and deploying foundation models in real-world environments Advancing agentic computing across mobile, edge, and cloud Seamlessly orchestrating intelligent systems to collaborate everywhere Preferred locations for this role are San Jose and Morrisville, but we are able to consider fully remote talent as well for qualified applicants. Summary We are seeking a highly motivated and skilled Sr. AI Model Evaluation Engineer to join our rapidly growing AI team. You will play a critical role in assessing the performance, robustness, and safety of large language models (LLMs), large vision models (LVMs), and large multimodal models (LMMs). This is a challenging yet rewarding opportunity to contribute to cutting-edge research and development in generative AI. You’ll be working with a collaborative team to push the boundaries of what’s possible with AI models and deploy them into innovative products. If you are passionate about making Smarter Technology For All, come help us realize our Hybrid AI vision Responsibilities Design, implement, and evaluate comprehensive evaluation pipelines for large generative AI models, encompassing various metrics and methodologies. Evaluate the performance of publicly available models, and discuss their relative advantages and disadvantages. Establish and maintain benchmarks for evaluating model performance across a range of tasks and datasets. Conduct thorough error analysis to identify patterns in model failures and provide actionable insights for improvement. Design and implement methods to detect and mitigate biases in model outputs, ensuring fairness and equitable performance. Develop and execute robustness tests to assess model resilience against adversarial inputs, noise, and variations in real-world data. Assess model safety, including identifying and mitigating harmful or inappropriate outputs. Experiment with various evaluation techniques, metrics, and datasets to optimize model quality and reliability. Contribute to the development and refinement of evaluation metrics that accurately reflect model performance and desired characteristics. Clearly communicate evaluation results and insights to engineers, researchers, and stakeholders. Identify potential partnerships with third parties. Develop and maintain evaluation tools and infrastructure. Monitor and analyze model performance in production environments, identify degradation, and propose solutions. Stay up-to-date with the latest advancements in large language and multi-modal models, model evaluation techniques, metrics, and related technologies. Contribute to the development of internal tools and infrastructure for model evaluation and monitoring. Required Qualifications Bachelor's or Master's degree in Computer Science, Machine Learning, or a related field. 12+ years of development experience Strong programming skills in Python and experience with deep learning frameworks like PyTorch. Deep understanding of machine learning evaluation principles, including various metrics (e.g., BLEU, ROUGE, perplexity, F1-score) and methodologies. Proven ability to design and conduct rigorous experiments, analyze data, and draw meaningful conclusions. Familiarity with large language models, transformer architectures, and related concepts. Experience with data processing tools and techniques (e.g., Pandas, NumPy). Experience working with Linux systems and/or HPC cluster job scheduling (e.g., Slurm, PBS). Preferred Qualifications Ph.D. in Computer Science, Machine Learning, or a related field. Excellent communication, collaboration, and problem-solving skills. Experience with automated model evaluation frameworks and tools. Experience with techniques for detecting and mitigating bias in AI models. Experience with safety and alignment evaluation methodologies. Experience with A/B testing and online evaluation techniques. The base salary range budgeted for this position in CA, CO, Jersey City - NJ, NV, Ithaca - NY, NYC, WA is $220k - $300k. Individuals may also be considered for bonus and/or commission. Lenovo’s various benefits can be found here: We are an Equal Opportunity Employer and do not discriminate against any employee or applicant for employment because of race, color, sex, age, religion, sexual orientation, gender identity, national origin, status as a veteran, and basis of disability or any federal, state, or local protected class. Seniority level Mid-Senior level Employment type Full-time Job function Engineering and Information Technology Industries IT Services and IT Consulting #J-18808-Ljbffr