Manager Rlhf Jobs in New York (NOW HIRING)

Key Responsibilities

Technical Leadership & Team Management

• Lead, mentor, and grow a high-performing engineering team, fostering a culture of innovation, accountability, and continuous improvement.
• Establish and enforce engineering best practices, focusing on code quality, reliability, and operational efficiency.
• Set technical direction and ensure alignment with enterprise goals and standards.

Enterprise Platform Architecture

• Architect scalable, secure, and resilient AI systems on Azure (and other cloud environments as needed), working with Large Language Models (LLMs), and collaborating with cross-functional teams to integrate AI into products and processes. A strong technical foundation in areas like Python, AI/ML frameworks, and cloud platforms, combined with project management and leadership skills, are essential for this role.
• Define Power Platform apps from end-to-end, including backend data pipelines.
• Define and enforce architectural standards, including microservices, event-driven design, and API-first principles.

Multi-Tenancy & SaaS Architecture

• Architect multi-tenant B2B environments with tenant-aware data partitions using Cosmos DB and Azure AD B2C/Entra ID to ensure data, security, and cost isolation.
• Build per-tenant vector stores, knowledge bases, and connectors to support customer-specific document ingestion and retrieval.
• Implement telemetry, metering, and consumption-based billing dashboards to track tenant-level usage and optimize resource allocation.

GenAI & Agent Integration

• Design and implement advanced GenAI solutions, including agent-based systems for automation, orchestration, and intelligent workflows.
• Integrate large language models (LLMs), Retrieval-Augmented Generation (RAG), and custom agents with business processes and user-facing applications.
• Lead design using frameworks such as Semantic Kernel, LangChain, AutoGen, or CrewAI to build orchestration among multiple specialized agents.
• Lead domain-specific fine-tuning and adaptation of large language models using Azure OpenAI Service or third-party frameworks (e.g., Tinker AI, LoRA, PEFT, or Delta Tuning).
• Design modular fine-tuning workflows to create tenant-aware or task-specific models that enhance personalization while maintaining shared governance and security.
• Implement automated retraining and evaluation loops to continuously improve response accuracy, tone, and compliance across tenants.
• Design agent toolkits that connect external APIs, databases, and Power Platform components for automated workflows.

Model Lifecycle and LLMOps

•         Oversee model evaluation, tuning, and continuous improvement cycles using reinforcement learning from human feedback (RLHF) or synthetic data augmentation.

•         Establish model-drift detection and retraining triggers to sustain model performance across customer domains.

•         Track usage analytics to measure model performance, accuracy, and user engagement.

•         Govern token usage, compute allocation, and cost optimization strategies using Azure Cost Management.

•         Manage versioning of models, prompts, and embeddings to ensure reproducibility, auditability, and traceability as part of the ongoing GenAI lifecycle operations.

Full Stack Management

• End-to-End Application Development: Design, develop, and maintain robust, scalable, and secure full-stack web applications.
• Frontend Development: Build dynamic and responsive user interfaces using React, TypeScript, and modern state management libraries.
• Backend & API Development: Architect and implement scalable backend services and RESTful APIs using Python (with frameworks like FastAPI or Flask) and caching technologies like Redis.
• Database Management: Design and manage data models and interact with both relational (SQL) and NoSQL databases (e.g., Cosmos DB).
• Power Platform Enablement
• Collaborate with citizen developers and business stakeholders to enable rapid app development using Power Apps, Power Automate, and Power BI.
• Ensure seamless integration between custom Azure services and Power Platform components.

Data Engineering & Processing

• Lead the design and implementation of robust data ingestion, transformation, and processing pipelines using Azure Data Factory, Databricks, Synapse, or equivalent.
• Ensure data quality, governance, and compliance with enterprise and regulatory standards.
• Develop secure pipelines for fine-tuning data preparation, ensuring data labeling, anonymization, and quality scoring to meet Responsible AI standards.
• Leverage Azure ML pipelines or Databricks workflows to automate dataset curation and model retraining schedules.
• Compute & Scalability
• Optimize compute resources using Azure Functions, AKS, and serverless architectures for cost-effective, high-performance workloads.
• Implement auto-scaling, load balancing, and failover strategies for mission-critical applications.
• Security, Compliance & Reliability
• Champion security best practices, including identity management, data protection, and compliance (e.g., SOC2, GDPR).
• Establish monitoring, alerting, and incident response protocols for platform reliability and operational excellence.
• Stakeholder Engagement & Communication
• Act as a technical liaison between engineering, product, business, and executive teams.
• Communicate complex technical concepts clearly to non-technical stakeholders.

Required Qualifications

• Proven experience leading engineering teams in building enterprise-grade platforms on Azure.
• Deep expertise in cloud architecture, distributed systems, and scalable frameworks.
• Advanced proficiency in GenAI, agent-based systems, and LLMs.  Understanding of agent frameworks (e.g., LangChain, Semantic Kernel) and orchestration tools.
• Strong background in data engineering, including ETL, data lakes, and real-time processing.
• Demonstrated ability to design secure, compliant, and reliable cloud solutions.
• Generative AI Expert with: Direct experience integrating LLMs (e.g., Azure OpenAI), developing RAG workflows, and using vector databases (e.g., Azure AI Search).
• Strong proficiency in languages like Python, with experience in AI/ML frameworks such as TensorFlow, PyTorch, or Keras.
• Experience with LLMs, prompt engineering, agentic flows, and foundational AI techniques.
• Experience with cloud-native development (e.g., AWS, Azure) and building scalable microservices.
• Strong fundamentals in software engineering, system architecture, and lifecycle management
• Knowledge of enterprise security, compliance, and governance standards.
• Track record of delivering large-scale, multi-tenant platforms.
• Excellent leadership, communication, and stakeholder management skills.
• Experience working in Agile, Scrum, or Kanban environments.

Preferred Qualifications

• Bachelor’s degree in Computer Science, Engineering, or a related field, or equivalent practical experience.
• Experience building and deploying fine-tuned or adapted LLMs using tools such as Tinker AI, Azure AI Studio, Hugging Face, or MosaicML.
• Knowledge of parameter-efficient tuning methods (LoRA, PEFT, QLoRA) and prompt-based adaptation techniques.
• Understanding of API Gateway & Identity federation (Azure API Management, Entra ID multi-tenant apps).
• Experience with Azure Data Factory, Synapse, Databricks, and Power Platform (Power Apps, Power Automate, Power BI).
• SaaS Experience: Experience building and supporting multi-tenant SaaS applications.
• Azure Ecosystem: Expertise with specific Azure services such as Azure AI Services, Azure App Services, Azure Functions, Azure DevOps, and Azure Kubernetes Service (AKS).
• Infrastructure as Code (IaC): Experience using IaC tools like Bicep or Terraform to manage cloud resources.
• Containerization: Proficiency with Docker and container orchestration tools like Kubernetes.
• Modern Frontend Tooling: Experience with TypeScript and state management libraries (e.g., Redux, Zustand).