AWS DevOps Lead/Architect

JOB DESCRIPTION

Job Summary:

UsefulBI is looking for an experienced AWS DevOps Lead/Architect to lead cloud infrastructure, DevOps automation, and platform engineering initiatives for enterprise-grade data platforms. This role involves 70% DevOps / Platform Engineering and 30% Data Engineering support, with a strong focus on AWS cloud infrastructure, CI/CD automation, Kubernetes, and Databricks platform management. The ideal candidate should be comfortable working closely with application teams, data engineers, and client stakeholders.

Key Responsibilities:

• Design, develop, and optimize scalable and resilient data pipelines using Databricks.
• Build and manage cloud infrastructure using Infrastructure as Code (IaC) with Terraform on AWS
• Develop automation scripts and data processing workflows using Python
• Implement and maintain cloud-based data solutions on AWS services such as S3, IAM, Glue, Lambda, EC2, and Redshift
• Ensure high availability, scalability, and performance of data platforms and pipelines
• Collaborate with data analysts, engineers to support end-to-end data lifecycle management
• Implement CI/CD pipelines for automated deployment of infrastructure and data workflows
• Monitor, troubleshoot, and optimize production systems to ensure reliability and performance
• Enforce security best practices and compliance standards across cloud environments

Required Skills & Qualifications:

• Hybrid role requiring 3 days per week onsite in Alameda, CA
• Strong hands-on experience with AWS cloud ecosystem (S3, IAM, Glue, Lambda, EC2, Redshift, etc.)
• Proficiency in Terraform for infrastructure provisioning and automation
• Strong programming experience in Python for data engineering and automation
• Extensive experience with Databricks, including Spark, Delta Lake, and job orchestration
• Solid understanding of data engineering principles, ETL/ELT pipelines, and data modeling
• Experience with CI/CD tools such as GitHub Actions, Jenkins, or equivalent
• Strong knowledge of SQL and relational databases
• 5+ years of experience in DevOps / Cloud Engineering / Platform Engineering.
• Strong hands-on expertise with AWS cloud services and architecture.
• Mandatory experience with Databricks administration, deployment, and optimization.
• Strong experience in Terraform / CloudFormation / AWS CDK.
• Experience with Docker, Kubernetes, and EKS.
• Hands-on experience with CI/CD tools such as Jenkins, GitHub Actions, GitLab CI/CD, or Azure DevOps.
• Strong scripting skills in Python.