Job Summary:
Peregrine is a company that helps public safety organizations and governments address societal challenges using their AI-enabled platform. They are seeking a Data Infrastructure Engineer to own the data layer and build systems for ingesting, storing, and serving real-time operational data, enabling critical decision-making for their customers.
Responsibilities:
โข Designing and operating a high-throughput, real-time data integration platform across diverse customer environments
โข Architecting a scalable open table format layer for reliable data storage at petabyte scale
โข Building and optimizing distributed data processing pipelines with Apache Spark and adjacent streaming technologies
โข Driving performance, reliability, and cost efficiency across the full data infrastructure stack
โข Collaborating with platform and product engineering teams to define data contracts, schemas, and integration patterns
โข Establishing best practices, tooling, and patterns that raise the quality bar for data infrastructure across the organization
Qualifications:
Required:
โข 2-5 years of experience operating large-scale data infrastructure systems in production environments
โข Experience with open table formats, particularly Apache Iceberg โ including schema evolution, partitioning strategies, compaction, and time travel
โข Extensive hands-on experience with Apache Spark for batch and streaming data processing at scale
โข Background in real-time data integration and stream processing, leveraging technologies such as Apache Kafka, Apache Flink, or equivalents
โข Experience with data pipeline orchestration using Airflow or similar tools
โข Strong software engineering fundamentals in Python and/or Scala, with a track record of writing production-quality code
โข Experience with AWS or comparable cloud platforms, including S3-based data lake architectures
โข Experience with Kubernetes and containerized deployment of data workloads
โข Degree in Computer Science, Engineering, or a related field, or equivalent practical experience
โข Located in San Francisco and open to working in office
Company:
Context changes everything. Founded in 2018, the company is headquartered in San Francisco, USA, with a team of 201-500 employees. The company is currently Growth Stage.