About the job
Your Role
As a Senior Data Engineer specializing in Machine Learning and AI, you will be pivotal in designing and maintaining robust data pipelines tailored for ML/AI workloads. Your expertise will be crucial in managing large-scale, unstructured, and semi-structured data. You will build feature pipelines and feature stores that promote the reusability and consistency of data for machine learning models.
Key Responsibilities
- Design and maintain data pipelines optimized for handling ML/AI workloads.
- Develop feature pipelines and stores ensuring data consistency for machine learning.
- Collaborate with Data Scientists and ML Engineers to define data requirements for various stages including training, validation, and deployment.
- Guarantee data quality, lineage, and governance to meet the standards required for AI/ML applications.
- Support MLOps practices by integrating data pipelines with model training, monitoring, and deployment workflows.
- Utilize distributed processing frameworks such as Spark, Databricks, or Azure Synapse for scalable ML data processing.
