Job Description

NOTE:

Only W2.

Only USC, GC.

Title: LLM Quality Assurance

Location: 100% Remote

Duration: 6-12+ months contract

Job Description:

• We are seeking a detail-oriented and highly skilled LLM Quality Assurance to join our team and ensure the reliability, safety, and performance of our AI/ML systems, including Large Language Models (LLMs). The ideal candidate will have a strong background in quality assurance with specialized expertise in testing and evaluating LLMs, prompt engineering, and ethical AI considerations.

Key Responsibilities:

• Develop and execute comprehensive test plans for AI/ML systems, with a focus on LLM evaluation.
• Design and implement robust testing frameworks to assess model accuracy, bias, limitations, and failure modes.
• Utilize advanced prompt engineering techniques to evaluate and refine model outputs.
• Collaborate with cross-functional teams, including AI researchers and engineers, to address identified issues and improve model performance.
• Document test results, bugs, and actionable insights clearly and effectively.
• Stay updated on the latest advancements in AI safety, LLM testing methodologies, and ethical considerations.
• Identify and test edge cases to uncover subtle model behaviors and potential vulnerabilities.

Basic Qualifications:

• Experience: 8+ years in Quality Assurance, with at least 1 year focused on AI/ML systems.
• Expertise: Demonstrated knowledge of LLM evaluation methodologies and prompt engineering.
• Understanding: In-depth awareness of LLM limitations, biases, and potential failure modes.
• Skills: Excellent analytical, problem-solving, and communication abilities.

Technical Skills:

• Proficiency in LLM testing frameworks and evaluation metrics.
• Knowledge of best practices in prompt engineering.
• Familiarity with AI safety protocols and ethical AI principles.
• Experience with version control systems (e.g., Git) and bug tracking tools.
• Basic programming skills for test automation, with a preference for Python.

Key Competencies:

• Creative problem-solving skills to discover and test edge cases.
• Exceptional attention to detail for identifying subtle model behaviors.
• Strong analytical thinking for systematic and comprehensive testing.
• Ability to approach testing from both a user-centric and adversarial perspective.
• Effective collaboration skills to work with cross-functional teams.

Required Education:

• Bachelor's degree or equivalent professional experience in a relevant field.

Job Tags

Similar Jobs