元宝-大模型评测产品经理
Tencent
元宝-大模型评测产品经理
Job Description
元宝-大模型评测产品经理
📋 Job Overview
The Product Manager for Yuanbao Large Model Evaluation at Tencent is responsible for designing and implementing automated evaluation systems for large language models, covering general and specialized capabilities like reasoning, writing, speech, and vision-language models. The role involves researching benchmarks, analyzing model performance, optimizing based on user data, and collaborating across teams to drive continuous improvements. This position plays a key role in standardizing evaluation processes and ensuring model reliability in real-world business scenarios.
📍 Location: Beijing, China
🏢 Business Unit: CSIG
📄 Full Description
1.自动评估体系建设：设计并搭建大模型自动评估体系，覆盖通用能力及专项能力（如推理、写作、语音、VLM 等）；构建评测指标体系与自动化评测流程，推动评测体系标准化、模块化、可扩展化；
2.Benchmark 研究与落地：跟踪国内外前沿大模型评测方法与 Benchmark，研究其评测维度与自动化机制；复现与改造高质量评测集，结合业务场景定制评测任务，确保评测的真实性与可复现性；
3.模型效果分析与策略优化：定期对不同版本模型进行系统评测与对比，输出详细分析报告，识别模型优势与薄弱点；针对记忆、写作、语音、多模态（VLM）等专项能力，设计细粒度评测指标与分析策略，支撑模型迭代方向；
4.用户数据分析与体验优化：持续监测与分析真实用户交互数据，挖掘模型表现的 badcase 与典型问题；与算法及产品团队协作，将用户侧问题转化为可量化的评测指标与优化方案；建立用户体验反馈与评测体系联动机制，推动模型效果持续提升与体验闭环优化；
5.跨团队协作与产品规划：与算法、工程、产品团队紧密配合，定义评测需求与指标体系，推动自动评测体系在实际业务与模型研发中的落地，形成评测 → 分析 → 优化的完整闭环。
2.Benchmark 研究与落地：跟踪国内外前沿大模型评测方法与 Benchmark，研究其评测维度与自动化机制；复现与改造高质量评测集，结合业务场景定制评测任务，确保评测的真实性与可复现性；
3.模型效果分析与策略优化：定期对不同版本模型进行系统评测与对比，输出详细分析报告，识别模型优势与薄弱点；针对记忆、写作、语音、多模态（VLM）等专项能力，设计细粒度评测指标与分析策略，支撑模型迭代方向；
4.用户数据分析与体验优化：持续监测与分析真实用户交互数据，挖掘模型表现的 badcase 与典型问题；与算法及产品团队协作，将用户侧问题转化为可量化的评测指标与优化方案；建立用户体验反馈与评测体系联动机制，推动模型效果持续提升与体验闭环优化；
5.跨团队协作与产品规划：与算法、工程、产品团队紧密配合，定义评测需求与指标体系，推动自动评测体系在实际业务与模型研发中的落地，形成评测 → 分析 → 优化的完整闭环。
🎯 Key Responsibilities
- Design and build automated evaluation systems for large models, covering general and specialized capabilities (e.g., reasoning, writing, speech, VLM); develop evaluation metrics and automated workflows to promote standardization, modularity, and scalability.
- Track domestic and international frontier large model evaluation methods and benchmarks, research evaluation dimensions and automation mechanisms; reproduce and adapt high-quality evaluation sets, customize evaluation tasks for business scenarios to ensure authenticity and reproducibility.
- Conduct systematic evaluations and comparisons of different model versions, output detailed analysis reports, identify strengths and weaknesses; design fine-grained evaluation metrics and analysis strategies for specialized capabilities like memory, writing, speech, and multimodal (VLM) to support model iteration directions.
- Continuously monitor and analyze real user interaction data, identify bad cases and typical issues; collaborate with algorithm and product teams to transform user-side problems into quantifiable evaluation metrics and optimization solutions; establish linkage between user experience feedback and evaluation systems to drive continuous model improvement and experience optimization.
- Collaborate closely with algorithm, engineering, and product teams to define evaluation requirements and metrics, promote the implementation of automated evaluation systems in actual business and model R&D, forming a complete closed loop of evaluation → analysis → optimization.
Locations
- Beijing, China
Salary
Estimated Salary Rangemedium confidence
300,000 - 600,000 CNY / yearly
Source: ai estimated
* This is an estimated range based on market data and may vary based on experience and qualifications.
Responsibilities
- Design and build automated evaluation systems for large models, covering general and specialized capabilities (e.g., reasoning, writing, speech, VLM); develop evaluation metrics and automated workflows to promote standardization, modularity, and scalability.
- Track domestic and international frontier large model evaluation methods and benchmarks, research evaluation dimensions and automation mechanisms; reproduce and adapt high-quality evaluation sets, customize evaluation tasks for business scenarios to ensure authenticity and reproducibility.
- Conduct systematic evaluations and comparisons of different model versions, output detailed analysis reports, identify strengths and weaknesses; design fine-grained evaluation metrics and analysis strategies for specialized capabilities like memory, writing, speech, and multimodal (VLM) to support model iteration directions.
- Continuously monitor and analyze real user interaction data, identify bad cases and typical issues; collaborate with algorithm and product teams to transform user-side problems into quantifiable evaluation metrics and optimization solutions; establish linkage between user experience feedback and evaluation systems to drive continuous model improvement and experience optimization.
- Collaborate closely with algorithm, engineering, and product teams to define evaluation requirements and metrics, promote the implementation of automated evaluation systems in actual business and model R&D, forming a complete closed loop of evaluation → analysis → optimization.
Tags & Categories
Answer 10 quick questions to check your fit for 元宝-大模型评测产品经理 @ Tencent.
10 Questions
~2 Minutes
Instant Score
Related Books and Jobs
元宝-大模型评测产品经理
Tencent
元宝-大模型评测产品经理
Job Description
元宝-大模型评测产品经理
📋 Job Overview
The Product Manager for Yuanbao Large Model Evaluation at Tencent is responsible for designing and implementing automated evaluation systems for large language models, covering general and specialized capabilities like reasoning, writing, speech, and vision-language models. The role involves researching benchmarks, analyzing model performance, optimizing based on user data, and collaborating across teams to drive continuous improvements. This position plays a key role in standardizing evaluation processes and ensuring model reliability in real-world business scenarios.
📍 Location: Beijing, China
🏢 Business Unit: CSIG
📄 Full Description
1.自动评估体系建设：设计并搭建大模型自动评估体系，覆盖通用能力及专项能力（如推理、写作、语音、VLM 等）；构建评测指标体系与自动化评测流程，推动评测体系标准化、模块化、可扩展化；
2.Benchmark 研究与落地：跟踪国内外前沿大模型评测方法与 Benchmark，研究其评测维度与自动化机制；复现与改造高质量评测集，结合业务场景定制评测任务，确保评测的真实性与可复现性；
3.模型效果分析与策略优化：定期对不同版本模型进行系统评测与对比，输出详细分析报告，识别模型优势与薄弱点；针对记忆、写作、语音、多模态（VLM）等专项能力，设计细粒度评测指标与分析策略，支撑模型迭代方向；
4.用户数据分析与体验优化：持续监测与分析真实用户交互数据，挖掘模型表现的 badcase 与典型问题；与算法及产品团队协作，将用户侧问题转化为可量化的评测指标与优化方案；建立用户体验反馈与评测体系联动机制，推动模型效果持续提升与体验闭环优化；
5.跨团队协作与产品规划：与算法、工程、产品团队紧密配合，定义评测需求与指标体系，推动自动评测体系在实际业务与模型研发中的落地，形成评测 → 分析 → 优化的完整闭环。
2.Benchmark 研究与落地：跟踪国内外前沿大模型评测方法与 Benchmark，研究其评测维度与自动化机制；复现与改造高质量评测集，结合业务场景定制评测任务，确保评测的真实性与可复现性；
3.模型效果分析与策略优化：定期对不同版本模型进行系统评测与对比，输出详细分析报告，识别模型优势与薄弱点；针对记忆、写作、语音、多模态（VLM）等专项能力，设计细粒度评测指标与分析策略，支撑模型迭代方向；
4.用户数据分析与体验优化：持续监测与分析真实用户交互数据，挖掘模型表现的 badcase 与典型问题；与算法及产品团队协作，将用户侧问题转化为可量化的评测指标与优化方案；建立用户体验反馈与评测体系联动机制，推动模型效果持续提升与体验闭环优化；
5.跨团队协作与产品规划：与算法、工程、产品团队紧密配合，定义评测需求与指标体系，推动自动评测体系在实际业务与模型研发中的落地，形成评测 → 分析 → 优化的完整闭环。
🎯 Key Responsibilities
- Design and build automated evaluation systems for large models, covering general and specialized capabilities (e.g., reasoning, writing, speech, VLM); develop evaluation metrics and automated workflows to promote standardization, modularity, and scalability.
- Track domestic and international frontier large model evaluation methods and benchmarks, research evaluation dimensions and automation mechanisms; reproduce and adapt high-quality evaluation sets, customize evaluation tasks for business scenarios to ensure authenticity and reproducibility.
- Conduct systematic evaluations and comparisons of different model versions, output detailed analysis reports, identify strengths and weaknesses; design fine-grained evaluation metrics and analysis strategies for specialized capabilities like memory, writing, speech, and multimodal (VLM) to support model iteration directions.
- Continuously monitor and analyze real user interaction data, identify bad cases and typical issues; collaborate with algorithm and product teams to transform user-side problems into quantifiable evaluation metrics and optimization solutions; establish linkage between user experience feedback and evaluation systems to drive continuous model improvement and experience optimization.
- Collaborate closely with algorithm, engineering, and product teams to define evaluation requirements and metrics, promote the implementation of automated evaluation systems in actual business and model R&D, forming a complete closed loop of evaluation → analysis → optimization.
Locations
- Beijing, China
Salary
Estimated Salary Rangemedium confidence
300,000 - 600,000 CNY / yearly
Source: ai estimated
* This is an estimated range based on market data and may vary based on experience and qualifications.
Responsibilities
- Design and build automated evaluation systems for large models, covering general and specialized capabilities (e.g., reasoning, writing, speech, VLM); develop evaluation metrics and automated workflows to promote standardization, modularity, and scalability.
- Track domestic and international frontier large model evaluation methods and benchmarks, research evaluation dimensions and automation mechanisms; reproduce and adapt high-quality evaluation sets, customize evaluation tasks for business scenarios to ensure authenticity and reproducibility.
- Conduct systematic evaluations and comparisons of different model versions, output detailed analysis reports, identify strengths and weaknesses; design fine-grained evaluation metrics and analysis strategies for specialized capabilities like memory, writing, speech, and multimodal (VLM) to support model iteration directions.
- Continuously monitor and analyze real user interaction data, identify bad cases and typical issues; collaborate with algorithm and product teams to transform user-side problems into quantifiable evaluation metrics and optimization solutions; establish linkage between user experience feedback and evaluation systems to drive continuous model improvement and experience optimization.
- Collaborate closely with algorithm, engineering, and product teams to define evaluation requirements and metrics, promote the implementation of automated evaluation systems in actual business and model R&D, forming a complete closed loop of evaluation → analysis → optimization.
Tags & Categories
Answer 10 quick questions to check your fit for 元宝-大模型评测产品经理 @ Tencent.
10 Questions
~2 Minutes
Instant Score
Related Books and Jobs