On Call Lead Systems Engineer Jobs (NOW HIRING)

Overview

The Lead Systems Engineer for High Performance Computing (HPC) and Artificial Intelligence (AI) works as part of the Advanced Systems team within Research Computing that supports the hardware and system-level software on the University's centralized high-performance computing and other computing for research systems. The Lead Systems Engineer is responsible for engaging with faculty, researchers, vendors, and other information technology (IT) staff to specify, design, install, and administer computing for research systems while also providing insight into trends and technologies supporting the advancement of AI research. The Lead Systems Engineer is also expected to be in tune to trends in computational research and will be asked to evaluate, pilot, and implement systems that advance Princeton's HPC and AI technologies enhancing Research Computing services. The Lead Systems Engineer serves as an expert for HPC and AI hardware and software and helps researchers troubleshoot system level problems with software, data, and job submission. This position requires one to work closely with colleagues at all levels of technical understanding in the Office of Information Technology (OIT) and University academic departments to provide timely and creative support for research computing. The Lead Systems Engineer is required to work well on teams and independently, and will be asked to lead initiatives within Advanced Systems, requiring only general supervision.

On-call rotation is a mandatory facet of this role, requiring infrequent off-hour and weekend duty.

Responsibilities

Operations:

• Design, maintain, troubleshoot, and refine advanced HPC/AI cluster infrastructure including highperformance interconnects, cluster schedulers, and configuration management across research systems.
• Partner with colleagues in Advanced Data and Storage Management to align designs for scratch filesystems and data management with cluster designs.
• Develop data-transfer pathways and networks to support AIdriven computing workloads.
• Establish and maintain best practices for cluster management and usage to support AI-driven workloads.
• Develop documentation for users and technical staff that can be used by the larger community.
• Develop, enhance, and expand monitoring infrastructure and related protocols for research computing systems.
• Plan and implement scheduled maintenance of operations, including during off hours.
• Perform other tasks as assigned.

Technical Leadership:

• Define and drive the institutional technical strategy for advanced AI and dataintensive HPC.
• Bring creativity, foresight, and mature professional judgment in anticipating and solving novel and complex problems, in determining project objectives and requirements, and in developing standards and governance for all research computing platforms.
• Leveraging expertise in AI technologies, identify, evaluate, and pilot researcher-facing systems that enable the acceleration of research using AI.
• Lead the implementation and expand adoption of modern, automation-driven infrastructure and cluster management practices.
• Promote institutionwide collaboration as the community expert advising and working with faculty,  researchers and vendors on emerging trends and challenges in AIenabled research computing.
• Cultivate a collaborative, knowledgesharing environment by providing technical mentorship to systems specialists and analysts by sharing designs and operational expertise across data systems and HPC/AI infrastructure.
• Contribute to the strategic vision for HPC/AI systems; Advise senior leadership and stakeholders on strategic investments, risks, and opportunities related to research infrastructure.

Troubleshooting and Problem Resolution:

• Monitor HPC clusters, networks, and storage systems for abnormalities, and resolve issues.
• Analyze and solve problems in Linux and HPC/AI computing environments with software, data, and job submissions.
• Use scripting and programming tools to troubleshoot issues.

Qualifications

Essential Qualifications:

• 10+ years of strong experience managing advanced research computing systems.
• Strong expertise with Linux system administration, installation, and troubleshooting.
• Advanced experience writing scripts in languages such as bash, Python and/or Perl.
• Proficient in managing networking in HPC environments. 
• Strong experience managing software in an advanced research computing environment.
• Experience supporting scheduling and managing jobs (SLURM) in large-scale computing environments.
• Strong oral and written communication skills, with the ability to proactively engage peers and communicate effectively across a diverse stakeholder community.
• Strong ability to solve complex and system infrastructure problems, and share expertise with colleagues at all levels.
• Demonstrated ability to collaborate across teams to solve systems and infrastructure challenges, aligning daytoday operational needs with longerterm technical and organizational goals as technologies evolve.
• When provided access to personal, proprietary and/or otherwise confidential data, maintain such data in the strictest confidence and follow procedures to ensure the privacy, security, and proper use of data.
• Education: Bachelors degree in a related field or equivalent experience.

Preferred Qualifications: 

• Experience working in an academic and research settings.
• Experience supporting AIdriven research in open and secure computing environments.
• Familiarity using and administering data-transfer technologies such as Globus that facilitate the transfer of large datasets.
• Experience using and supporting parallel file systems that are commonly used in HPC/AI systems.
• Experience supporting unstructured data in HPC/AI environments. 

Princeton University is an Equal Opportunity Employer and all qualified applicants will receive consideration for employment without regard to age, race, color, religion, sex, sexual orientation, gender identity or expression, national origin, disability status, protected veteran status, or any other characteristic protected by law.

The University considers factors such as (but not limited to) scope and responsibilities of the position, candidate's qualifications, work experience, education/training, key skills, market, collective bargaining agreements as applicable, and organizational considerations when extending an offer. The posted salary range represents the University's good faith and reasonable estimate for a full-time position; salaries for part-time positions are pro-rated accordingly.

If the salary range on the posted position shows an hourly rate, this is the baseline; the actual hourly rate may be higher, depending on the position and factors listed above.

The University also offers a comprehensive benefit program to eligible employees. Please see this link for more information.

Standard Weekly Hours36.25Eligible for OvertimeNoBenefits EligibleYesProbationary Period180 daysEssential Services Personnel (see policy for detail)NoPhysical Capacity Exam RequiredNoValid Driver's License RequiredNo Experience LevelDirector#LI-JJ1Salary Range$135,000 to $150,000Employment Type: FULL_TIME