Job Summary:
Kake is seeking a Senior Software Engineer to contribute to the development and evaluation of AI training data for AI agents and LLMs. In this role, you will work at the intersection of software engineering and artificial intelligence, helping to build better AI models by leveraging your technical expertise.
Responsibilities:
• Create and review coding tasks based on real-world software engineering scenarios, including debugging, refactoring, code generation, API usage, automated tests, performance, security, and edge cases
• Write high-quality reference solutions that are correct, clear, testable, and aligned with task requirements
• Evaluate AI-generated code and responses using structured rubrics, assessing correctness, clarity, security, performance, maintainability, and instruction-following
• Compare multiple model responses, select the strongest answer, and justify your decision with clear technical reasoning
• Identify bugs, hallucinated APIs, missing edge cases, weak explanations, and poor engineering decisions in AI-generated outputs
• Work with terminal-based development workflows when needed, including running tests, debugging issues, managing dependencies, and navigating repositories
• Follow detailed guidelines consistently and participate in calibration activities to ensure high-quality, reliable evaluations
Qualifications:
Required:
• 5+ years of professional software engineering experience in a backend, fullstack, or systems role
• Strong proficiency in at least one core programming language, ideally Python, JavaScript/TypeScript, Go, Java, C++, or SQL
• Hands-on experience with Terminal-Bench, with the ability to evaluate AI agent performance on terminal-based tasks including compiling code, running tests, managing environments, and completing multi-step software engineering workflows
• Comfortable working with Git, command line/terminal, and common development workflows
• Ability to evaluate code critically - not only whether it works, but whether it is well-designed, secure, and maintainable
• Prior experience in AI data production, RLHF, data annotation, or LLM evaluation projects
• Excellent written and verbal communication skills in English
• Ability to work independently in a remote, asynchronous, fast-paced environment
• High attention to detail and the ability to follow complex, rubric-based guidelines consistently
Preferred:
• Experience with Python-heavy workflows, automated testing frameworks, Docker, Linux, bash, or containerized environments
• Experience with repo-level code reasoning, large codebases, or open-source contributions
• Background in backend systems, data engineering, DevOps, infrastructure, security, or large codebase
Company:
Life is Better with Kake! Kake offers premier teams of software engineers to support some of the world's most prominent and innovative brands. Founded in , the company is headquartered in Austin, TX, US, , with a team of 201-500 employees. The company is currently Growth Stage.