About the job
As a Data Quality Engineer at jobsforhumanity, you will play a pivotal role in ensuring the accuracy, consistency, and reliability of data within the Microsoft Fabric Data Warehouse and its associated services. Your primary focus will be on testing and validating data pipelines, event-driven processes, and Azure integrations such as Logic Apps and Service Bus. Collaborating closely with data engineers, architects, and business units, you will uphold the integrity of data to support analytics and reporting initiatives.
Key Responsibilities
- Design, develop, and execute rigorous data quality and validation tests for Microsoft Fabric Data Warehouse components, including tables, views, and semantic models.
- Validate ETL/ELT pipelines, including Fabric Dataflows and integrated source systems, ensuring data accuracy and completeness.
- Monitor and test event-driven data processes, encompassing Azure Service Bus messages, queues, and topics.
- Review and validate Azure Logic Apps workflows and integrations that facilitate data movement and transformation between systems.
- Conduct comprehensive end-to-end testing of data ingestion, transformations, and delivery to downstream consumers or BI systems.
- Develop and sustain automated data quality checks, reconciliation scripts, and testing frameworks to enhance efficiency.
- Identify, analyze, and communicate data quality issues, performing root cause analysis and impact assessments.
- Collaborate with Data Engineers, BI teams, and business units to define data quality standards, metrics, and acceptance criteria.
- Monitor key data quality KPIs, implement controls to mitigate defects, and ensure reliability in production pipelines.
- Document test cases, workflows, data quality standards, and testing results clearly for both technical and business stakeholders.
- Familiarity with unit testing, automated data validation, and integration testing tools such as dbt, Great Expectations, tSQLt, pytest, Azure Logic Apps testing, and Service Bus testing scripts.
- Conduct unit testing and validation of Azure Logic Apps workflows and Service Bus message flows.
- Develop and maintain automated data quality checks and integration tests utilizing tools such as dbt, Great Expectations, or custom scripts.
- Assess and validate API endpoints and integration workflows using tools like Postman to ensure accurate data transmission between systems.
- Execute end-to-end testing of event-driven processes and service integrations through Postman and automated scripts.

