Manager, Generative AI Applied ML Scale AI Full-time - Hybrid - $240K - $380K San Francisco, Ca[...] - San Francisco California

Company: Bronze Technologies, Inc.
Location: San Francisco, California
Posted On: 01/23/2025

Scale's Generative AI Data Engine powers the most advanced LLMs and generative models in the world through RLHF/RLAIF, data generation, model evaluation, safety, and alignment.As the Manager of the Generative AI Applied ML team, you will lead a talented team of research engineers and ML engineers focused on delivering scalable, production-ready solutions to support Scale's GenAI Data Engine, such as rater-assistant models, LLM as a judge, critique modeling, fraud and cheating detection, etc. This role is critical for designing and executing a roadmap that accelerates our frontier model building customers' Generative AI initiatives forward. We are looking for someone with a strong background and hands-on experience in fine-tuning and evaluating LLMs, prioritizing practical, production-oriented problem-solving over academic research. This position requires a deep commitment to building robust, efficient systems that meet the demands of large-scale production, supporting millions of tasks monthly across a hybrid human-machine system, with the aim to scale into billions monthly.This is a unique opportunity to drive impactful, production-focused work on the frontier of AI, collaborating with industry-leading professionals to shape and deliver reliable, high-performance solutions for real-world applications.You will:

• Manage a team of highly effective research engineers and ML engineers. Provide guidance, mentorship, and technical leadership to a team of researchers and engineers working on Generative AI projects.
• Develop and evaluate methods for integrating machine learning into human-in-the-loop labeling systems to ensure high-quality and throughput labels for our customers.
• Implement and improve on state-of-the-art research developed internally and from the community and put them into production to solve problems for our customers and taskers.
• Work with product and research teams to identify opportunities for improvement in our current product line and for enabling upcoming product lines.
• Work with massive datasets to develop both generic models as well as fine-tune models for specific products.
• Build a scalable autorating platform that will improve quality and efficiency of the generative AI data engine.
• Thrive in a high-energy, fast-paced startup environment and are ready to dedicate the time and effort needed to drive impactful results.
• Must be able to commute to the San Francisco Office 2+ times weekly.Ideally you'd have:
  • 7+ years of full time work experience in deep learning, deep reinforcement learning, or natural language processing in a production environment, especially post-training experience with LLMs.
  • Experience managing a 5-20 people team or leading a technical workstream.
  • Strong programming skills in Python, experience in PyTorch or Tensorflow.
  • Experience with MLOps and the automation of model training & evaluation.
  • Experience working with cloud technology stack (eg. AWS or GCP) and developing machine learning models in a cloud environment.
  • Deep appreciation for building high-quality, robust, reusable productionized ML systems.Nice to haves:
    • Publication experience in the field or related topics.
    • Experience with using LLMs as a judge for AutoRating systems.
    • Graduate degree in Computer Science, Machine Learning or Artificial Intelligence specialization.
      #J-18808-Ljbffr More...

          

      ---

      Register an account with us and set up job agents! We'll email you immediately when jobs like this are posted on our site.

      ---