Qualifications
Key Responsibilities:Design and implement comprehensive data pipelines on AWS utilizing services such as Amazon S3, AWS Glue, AWS Lambda, and Amazon Redshift. Develop data processing and transformation workflows with Databricks, Apache Spark, and SQL to meet analytics and reporting demands. Create and manage orchestration workflows using Apache Airflow to automate data pipeline execution, scheduling, and monitoring. Lead the transition of legacy data systems to modern cloud-based architectures. Establish and uphold CI/CD pipelines for data workflows. Engage with data scientists, analysts, and business stakeholders to comprehend data needs and deliver scalable solutions. Enhance data pipelines for performance, reliability, and cost-efficiency, utilizing AWS best practices and cloud-native technologies. Requirements:Over 10 years of experience in developing and deploying large-scale data processing pipelines in production environments. Hands-on expertise in designing and implementing data pipelines on AWS cloud infrastructure. In-depth knowledge of AWS services including Amazon S3, AWS Glue, AWS Lambda, and Amazon Redshift. Proficient in architecting and developing large-scale data pipelines and data lakehouse architectures using Databricks. Experience in designing batch and real-time streaming solutions leveraging Apache Spark on Databricks. Practical experience with Apache Airflow for orchestrating and scheduling data pipelines. Solid understanding of data modeling, database design principles, and SQL/Spark SQL. Familiarity with version control systems (e.g., Git) and CI/CD pipelines. Strong communication skills and the ability to effectively collaborate with cross-functional teams. Exceptional problem-solving skills and meticulous attention to detail.
About the job
Tiger Analytics is a rapidly expanding advanced analytics consulting firm that specializes in delivering exceptional insights through Data Science, Machine Learning, and Artificial Intelligence. Our team possesses profound expertise, making us a trusted analytics partner for numerous Fortune 500 companies, empowering them to derive substantial business value from their data. Our leadership and contribution to the analytics field have been recognized by prominent market research firms such as Forrester and Gartner. We are on the lookout for outstanding talent to enhance our global analytics consulting team.
As a Lead Data Engineer, you will play a pivotal role in architecting, constructing, and sustaining scalable data pipelines within the AWS cloud ecosystem. You will collaborate with diverse teams to facilitate data analytics, machine learning, and business intelligence projects. The ideal candidate will bring extensive experience with AWS services, Databricks, and Apache Airflow.
About Tiger Analytics
Tiger Analytics is a forward-thinking firm that is at the forefront of advanced analytics consulting. We are known for our innovative approach and ability to help clients leverage data for strategic advantage. Our consultants are recognized experts in their fields, and we take pride in being the preferred analytics partner for Fortune 500 companies.