About the job
Join the dynamic team at Minderacraft as a Team Lead Data Engineer. We are looking for a motivated and talented individual who possesses extensive experience in AWS cloud services, Databricks, Apache Spark, Python, and SQL. Your role will be pivotal in designing, developing, and optimizing data pipelines and analytical solutions that drive key business initiatives.
Primary Responsibilities:
- Architect and maintain robust data pipelines and transformation workflows utilizing Databricks and Spark.
- Develop and enhance ETL/ELT processes to efficiently manage large datasets.
- Leverage AWS services such as S3, Lambda, IAM, ECS, and CloudWatch to meet data architecture and operational requirements.
- Collaborate with data analysts, scientists, and business stakeholders to gather requirements and convert them into effective technical solutions.
- Ensure the integrity, reliability, and performance of all data systems.
- Advocate for and implement best practices in coding, version control, continuous integration/deployment, and environment management.
- Monitor, troubleshoot, and ensure the high availability of data pipelines.
- Contribute to architectural design decisions, documentation, and process enhancements.
Qualifications:
- Proven experience with AWS Cloud (including S3, Lambda, IAM, EC2 or similar).
- In-depth knowledge of Databricks, Delta Lake, and Unity Catalog.
- Strong proficiency in Apache Spark (preferably PySpark).
- Excellent programming skills in Python.
- Advanced SQL skills, with experience in performance tuning and managing large datasets.
- Adept at thriving in fast-paced, agile environments.
- Strong analytical and problem-solving capabilities, with a proactive mindset to drive improvements.
- Exceptional communication and stakeholder management skills, capable of engaging with diverse teams.
- Familiarity with data governance tools and frameworks (e.g., DataHub, Soda).
- Experience with CI/CD tools like GitHub Actions.
- Availability: Must accommodate meetings and calls in US Pacific Time (PT).
Personal Attributes:
- Self-motivated, accountable, and proactive.
- Strong ownership mentality with the ability to work independently.
- Adept at engaging with both technical and non-technical stakeholders.
- Passionate about building reliable, scalable, and high-quality data systems.
