Lead Data DevOps Engineer (GCP)

We are looking for a skilled Lead Data DevOps Engineer with expertise in Google Cloud Platform (GCP) to join our team.

This role suits professionals with hands-on experience in cloud-based data systems, infrastructure automation, and optimizing data workflows. Familiarity with AWS or Azure is beneficial but not mandatory.

Responsibilities

• Design cloud data infrastructure using GCP services such as DataFlow, GCS, BigQuery, Dataproc, and Cloud Composer
• Deploy infrastructure using IaC tools like Terraform to enable provisioning and monitoring
• Work with data engineering teams to develop automated and efficient data workflows using Python
• Set up CI/CD pipelines through tools such as Jenkins, GitLab CI, or GitHub Actions for streamlined deployments
• Optimize performance and improve the reliability of data platforms by collaborating with cross-functional teams
• Maintain cloud-based data tools including Apache Spark, Apache Kafka, and Apache Airflow
• Analyze and resolve issues to ensure scalability and reliability of cloud-based data systems

Requirements

• 5+ years of experience in cloud environments, focusing on GCP services like BigQuery, Cloud Composer, and Dataproc
• Proficiency in Python combined with competency in SQL for data pipeline development
• Knowledge of IaC tools such as Terraform or CloudFormation for infrastructure automation
• Skills in building CI/CD pipelines with tools like Jenkins, GitHub Actions, or GitLab CI
• Background in Linux-based operating systems and shell scripting
• Understanding of network protocols and concepts such as TCP/IP, DNS, and NAT
• Competency in implementing tools like Apache Spark, Apache Airflow, or ELK Stack

Nice to have

• Familiarity with AWS or Azure, including ECS, S3, Data Lake, or Synapse
• Flexibility to use IaC tools such as Ansible
• Background showcasing experience with alternative data workflow automation tools