Role: Platform Engineer- Kafka with Linux
Duration: 6 - 12 months
Location: NYC, NY
Rate: $60/hr.
Job description:
Core Platform Engineer with deep expertise in Apache Kafka and Linux systems engineering.
The ideal candidate will be responsible for designing, deploying, maintaining, and scaling Kafka clusters in mission-critical environments, while also supporting the Linux-based infrastructure that forms the foundation of our real-time data platform.
Responsibilities:
- Design, implement, and maintain highly available Kafka clusters in production environments.
- Perform Kafka upgrades and patching ensuring minimal downtime and seamless transitions.
- Administer security protocols and access controls to protect the platform.
- Monitor cluster health and performance using our established monitoring stack (Prometheus, Grafana)
- Troubleshoot and resolve cluster-related incidents such as broker outages, replication lag.
- Develop and enhance automation scripts to streamline operational tasks and reduce manual intervention.
- Assist with planning and managing cluster resources to support future growth.
- Develop Kafka topics, partitions, ACLs, schema registry, and optimize throughput and reliability.
- Manage a tune Kafka Connect, Kafka Streams, and MirrorMaker as needed for data pipelines.
- Ensure platform reliability and performance on enterprise-grade Linux systems (RHEL, Ubuntu).
- Automate Kafka provisioning, configuration, and monitoring using Infrastructure as Code (IaC) and configuration management tools (e.g., Ansible, Terraform).
- Deep knowledge of Kafka ecosystem components like ZooKeeper, Kafka Connect, Schema Registry, MirrorMaker, KRaft (Kafka without ZooKeeper).
- Build custom Kafka tooling and dashboards for producers and consumers, monitor lag and health metrics.
- Work with engineering teams to onboard new Kafka use cases and ensure security and compliance standards.
- Perform root cause analysis and resolution of platform-level incidents and outages.
- Develop and maintain Linux shell scripts for automation of deployments, monitoring, and maintenance tasks.
Minimum Qualifications:
- BE/B Tech/MCA
- Excellent written and verbal communication skills
- Min 5+ years of experience.
- Experience in Finance Domain
Preferred Qualifications/ Skills:
- Strong hands-on experience with Apache Kafka (2.x or 3.x), including cluster administration and tuning.
- In-depth experience with Linux system internals, kernel tuning, filesystems, and performance troubleshooting.
- Experience in managing Kafka brokers in production environments with millions of messages per second.
- Deep knowledge of Kafka ecosystem components like ZooKeeper, Kafka Connect, Schema Registry, MirrorMaker, KRaft (Kafka without ZooKeeper).
- Scripting and automation skills in Shell, Python, or Go.
- Experience in container orchestration platforms like Docker and Kubernetes.
- Familiarity with SSL/TLS, SASL, RBAC, and encryption in-transit/at-rest for Kafka.
- Expertise in monitoring and alerting for Kafka and Linux systems.
- Strong understanding of distributed systems, message queues, and event-driven architectures