Job Description

Locations

• United States, MA, Boston, Boston, MA, United States
• United States, WA, Bellevue, Bellevue, WA, United States

Salary

Skills Required

• - 2+ years of non-internship design or architecture (design patterns, reliability and scaling) of new and existing systems experienceintermediate
• - Experience programming with at least one software programming languageintermediate

Required Qualifications

• - Bachelor's degree or equivalent (degree in equivalent)
• - 2+ years of non-internship design or architecture (design patterns, reliability and scaling) of new and existing systems experience (experience, 2 years)
• - Experience programming with at least one software programming language (experience)

Preferred Qualifications

• - 3+ years of full software development life cycle, including coding standards, code reviews, source control management, build processes, testing, and operations experience (experience, 3 years)
• - Master's degree in computer science or equivalent (degree in computer science or equivalent)
• - Knowledge of professional software engineering & best practices for full software development life cycle, including coding standards, software architectures, code reviews, source control management, continuous deployments, testing, and operational excellence (experience)
• - Previous experience working with or evaluating AI systems, particularly in the context of code generation (experience)
• - Understanding of machine learning concepts, particularly as they apply to natural language processing and code generation (experience)
• - Data analysis skills, writing script to analyze model performance against human ground truth (experience)
• Our compensation reflects the cost of labor across several US geographic markets. The base pay for this position ranges from $62.16/hr in our lowest geographic market up to $107.50/hr in our highest geographic market. Pay is based on a number of factors including market location and may vary depending on job-related knowledge, skills, and experience. Amazon is a total compensation company. Dependent on the position offered, equity, sign-on payments, and other forms of compensation may be provided as part of a total compensation package, in addition to a full range of medical, financial, and/or other benefits. For more information, please visit https://www.aboutamazon.com/workplace/employee-benefits. This position will remain posted until filled. Applicants should apply via our internal or external career site. (experience)

Responsibilities

• 1. Critically analyze and evaluate code responses generated by our LLMs across various programming languages and paradigms
• 2. Exercise expert judgment to select the most appropriate and efficient code solutions from multiple LLM-generated options
• 3. Make informed decisions on behalf of our customers, ensuring that selected code meets industry standards, best practices, and specific client needs
• 4. Develop and write coding demonstrations to illustrate "what good looks like" in AI-generated code, setting benchmarks for quality and efficiency
• 5. Provide detailed feedback and explanations for your evaluations, helping to refine and improve the LLM's understanding and output
• 6. Collaborate with the AI research team to identify areas for improvement in the LLM's coding capabilities
• 7. Stay abreast of the latest developments in software engineering, coding standards, and AI to ensure our evaluations are accurate

Tags & Categories