Senior Data Engineer - Real-Time Streaming

We are seeking a talented Senior Data Engineer with extensive knowledge of real-time data streaming and distributed data processing to architect, develop, and enhance state-of-the-art data platforms. This pivotal role is essential for advancing event-driven architecture and real-time analytics within critical banking systems, particularly in risk and compliance domains.In this position, you will work synergistically with data architects, platform engineers, and business stakeholders to create low-latency, high-throughput data pipelines that empower sophisticated analytics and informed decision-making.Key ResponsibilitiesDesign, develop, and maintain robust real-time streaming pipelines utilizing Apache Kafka, PySpark, and FlinkConstruct scalable and fault-tolerant event-driven data architecturesHandle high-volume streaming data ensuring low latency and high reliabilityIntegrate diverse data sources into centralized data platforms (Data Lake / Lakehouse)Enhance data pipelines for performance, scalability, and cost-effectivenessUphold data quality, governance, and compliance in line with banking regulationsCollaborate with cross-functional teams to convert business needs into technical solutionsMonitor and debug streaming jobs and production pipelinesRequired Skills & Experience5+ years of experience in Data EngineeringDemonstrated proficiency in:PySpark / Spark StreamingApache Kafka (Producers, Consumers, Kafka Streams)Apache Flink or other real-time processing frameworksProven experience in building real-time / near real-time data pipelinesStrong understanding of distributed systems and event-driven architectureProficiency in Python / Java / ScalaExperience with data lakes, ETL/ELT pipelines, and big data ecosystemsFamiliarity with cloud platforms (AWS / Azure / GCP) is advantageousKnowledge of banking, risk, or compliance data systems is highly preferredPreferred QualificationsExperience in the financial services or banking domainExposure to data governance, regulatory reporting, or compliance systemsUnderstanding of CI/CD pipelines and DevOps practices for data platforms

Join our dynamic team at gsstech-group as a Data Engineer specializing in real-time streaming and event-driven architectures. We seek a talented individual who will take charge of creating scalable data pipelines, enhancing streaming systems, and achieving optimal performance in distributed environments.Key ResponsibilitiesDesign and implement real-time data streaming pipelines utilizing technologies such as Apache Flink, Kafka, and JavaBuild and sustain event-driven architectures for extensive distributed systemsConduct JVM tuning and performance optimization for streaming applicationsDevelop and deploy applications utilizing containerization tools (Docker, Kubernetes)Utilize the Cloudera platform for data engineering and pipeline orchestrationApply robust design patterns while upholding high coding standardsTroubleshoot and resolve challenges within a distributed systems ecosystemCollaborate with DevOps teams to manage CI/CD pipelines (GitHub, Jenkins)Operate on Linux-based systems, including configuration and shell scriptingEnhance data processing through caching mechanisms (e.g., Redis – considered a plus)Required Skills & ExperienceExtensive hands-on experience in Real-Time Streaming (Flink / Kafka / Java)Comprehensive understanding of event-driven architectureExperience with JVM performance tuningProficiency in Docker and KubernetesStrong background in Linux OS and shell scriptingKnowledge of design patterns and scalable system designFamiliarity with CI/CD tools such as GitHub and JenkinsPractical troubleshooting experience in distributed systemsNice to HaveFamiliarity with Redis or other caching systemsExposure to Cloudera Data Platform engineeringPrior experience in the banking or financial domain is advantageous

Join our dynamic team at gsstech-group as a Technology Engineer. We are seeking an exceptional individual with profound knowledge in streaming data platforms and cloud-native technologies. The successful candidate will be instrumental in developing scalable, event-driven data systems, leveraging hands-on experience with Kafka, CI/CD pipelines, and cloud environments.Your Role:Key ResponsibilitiesDesign, implement, and oversee high-throughput Kafka clusters in production settings.Construct and sustain event-driven and streaming data architectures.Develop and manage CI/CD pipelines employing tools such as Jenkins, ArgoCD, and GitOps methodologies.Engage in cloud-native application development across AWS, Azure, or OCI platforms.Automate infrastructure and deployment processes using contemporary cloud automation tools.Collaborate with diverse teams to deliver robust and scalable data solutions.Adhere to best practices in code quality, testing, and deployment automation.Implement effective design patterns for scalable and maintainable systems.Conduct unit testing and integration testing to ensure reliable application delivery.Your Skills & ExperienceExtensive experience in Apache Kafka setup, configuration, and performance tuning within high-volume contexts.Proficient in Git (source control) and CI/CD tools like Jenkins and ArgoCD.Practical experience with public cloud platforms (AWS / Azure / OCI).Demonstrated experience with large-scale streaming datasets.Solid understanding of event-driven architecture and microservices.Familiarity with GitOps practices and DevOps culture.Experience with cloud automation tools.Strong grasp of software design patterns and system design principles.Experience in automated testing (unit & integration testing).Preferred QualificationsExperience with OpenShift / Kubernetes platforms.Knowledge of real-time streaming frameworks and data pipelines.Experience in the banking or financial services domain is a plus but not mandatory.

Job OverviewWe are seeking an experienced RTS Kafka Engineer to spearhead the design, development, and maintenance of cutting-edge real-time streaming platforms. The ideal candidate will possess extensive hands-on expertise with Apache Kafka, cloud-native architectures, and event-driven systems, particularly in high-throughput settings. This position involves constructing dependable streaming pipelines that facilitate real-time data processing while supporting distributed cloud-based systems.Key Responsibilities Design, implement, and oversee Kafka-based real-time streaming solutions in high-throughput environments. Execute Kafka setup, configuration, tuning, and performance enhancements. Create and sustain scalable event-driven and streaming architectures. Build and manage streaming data pipelines for processing large-scale datasets. Implement CI/CD pipelines utilizing tools like Git, Jenkins, ArgoCD, or similar platforms. Collaborate with cloud-native applications deployed on AWS, Azure, or OCI. Automate infrastructure and deployment using advanced cloud automation toolsets. Employ GitOps practices for effective deployment and environment management. Ensure the reliability, scalability, and high availability of systems. Conduct automated unit testing and integration testing for streaming applications. Partner with DevOps, platform engineering, and application teams to deliver comprehensive real-time solutions. Leverage software design patterns and best practices in distributed systems development.Required Skills & Qualifications Proven hands-on experience with Apache Kafka setup, configuration, and administration. Background in high-volume streaming data platforms. Strong grasp of event-driven architecture and distributed systems. Proficient in source control systems (Git). Practical exposure to CI/CD pipelines (Jenkins, ArgoCD, or similar). Familiarity with at least one public cloud platform: AWS Microsoft Azure Oracle Cloud Infrastructure (OCI) Knowledge of cloud automation tools and infrastructure automation. Understanding of GitOps methodology. Expertise in software design patterns and implementation practices. Experience with automated testing (unit & integration testing).Preferred Qualifications Familiarity with Kafka ecosystem tools (Kafka Connect, Schema Registry, ksqlDB).

ValGenesis builds digital validation platforms for life sciences organizations. Its products support pharmaceutical and biotech companies as they move toward digital processes, maintain regulatory compliance, and ensure manufacturing quality throughout the product lifecycle. Thirty of the top fifty global firms in this industry use ValGenesis solutions. More details about the company's work in paperless validation are available at valgenesis.com/about. Role overview The Senior Software Engineer - Data Engineering position is based in Chennai. This role focuses on developing and maintaining data engineering solutions to support ValGenesis platforms. Work will center on building systems that help life sciences clients manage and analyze data for compliance and quality throughout their operations.

Join our dynamic team at minderacraft as a Senior Data Engineer, where your expertise will be pivotal in shaping our data infrastructure. We are seeking a highly skilled individual with a deep understanding of big data technologies, ETL/ELT processes, and data modeling methodologies. Your primary focus will be to design, optimize, and maintain robust data pipelines, ensuring the integrity of our data and supporting our analytics initiatives.

Join Our Team as a Senior Lead Data EngineerAre you passionate about data and technology? At Brillio, we are seeking a highly skilled Senior Lead Data Engineer to guide our data engineering initiatives. You will play a pivotal role in driving data strategy, architecture, and implementation across our projects, ensuring that we leverage data to deliver exceptional solutions for our clients.

Join our dynamic Data Team at Mindera, where as a Senior Data Engineer, you will play a pivotal role in creating the data pipelines and tables that drive our business-critical dashboards, empower self-service analytics, and support advanced machine learning models and real-time data products. Utilizing state-of-the-art tools such as DBT, Spark, and Airflow, you will convert high-volume raw event data into user-friendly, impactful datasets.You will collaborate cross-functionally with Machine Learning Engineers, Data Scientists, and BI Developers to facilitate data-driven decision-making throughout the organization. Our engineers benefit from a culture of autonomy, innovation, and continuous learning, supported by structured career progression paths and access to training resources.As a Senior Data Engineer, your responsibilities will include:Designing and constructing scalable data pipelines, models, and feature stores to support analytics and machine learning workloads.Deploying and managing cloud-native data applications on AWS, leveraging CI/CD pipelines to automate builds, tests, and releases.Ensuring the technical quality, performance, and reliability of production-grade data pipelines through robust observability and engineering best practices.

About ValGenesis ValGenesis stands at the forefront of digital validation solutions for the life sciences sector. Our comprehensive suite of products empowers 30 of the top 50 global pharmaceutical and biotech firms to embrace digital transformation, ensuring total compliance, and achieving excellence in manufacturing intelligence across their product lifecycle.Discover the opportunity to be part of ValGenesis, the leading standard for paperless validation in the Life Sciences industry: valgenesis.com/aboutAbout the Role:As a Senior Software Engineer in Data Engineering, you will play a pivotal role in developing scalable data solutions that support our innovative products.

We are on the lookout for an exceptional Senior Data Engineer to architect, construct, and sustain scalable data pipelines for our enterprise-level data platforms focused on the Risk & Compliance sector. The successful candidate will possess a deep understanding of PySpark, Python, and data engineering best practices, emphasizing data quality, governance, and security.Key ResponsibilitiesDesign, develop, and enhance scalable data pipelines leveraging PySpark and PythonCreate robust ETL/ELT workflows to manage substantial volumes of both structured and unstructured dataCollaborate with data scientists, analysts, and business stakeholders to produce high-quality datasetsGuarantee data integrity, accuracy, and reliability through comprehensive validation frameworks and monitoringImplement data security and access control mechanisms that align with compliance standardsPartner closely with Risk & Compliance teams to fulfill regulatory and reporting obligationsOptimize the performance of data processing jobs and queriesMaintain and upgrade existing data architecture and pipelinesRequired Skills & Experience6+ years of experience in Data EngineeringExtensive hands-on experience with PySpark and PythonSolid background in SQL and Oracle databasesProven experience in constructing and managing large-scale data pipelinesStrong understanding of data warehousing concepts and ETL frameworksExperience with data validation, data quality, and governance frameworksFamiliarity with cloud platforms (AWS/Azure/GCP) is a plusExperience in the banking, financial services, or risk & compliance domain is preferredKey CompetenciesStrong analytical and problem-solving skillsAdept at working in a fast-paced, collaborative environmentExcellent communication and stakeholder management abilitiesMeticulous attention to detail, especially regarding data quality and securityNice to HaveExperience with Big Data ecosystems (Hadoop, Spark)Knowledge of data security and regulatory compliance frameworksPrior experience with enterprise data platforms

About Gen Digital Gen Digital is a global company focused on digital freedom and security. Our brands include Norton, Avast, LifeLock, and MoneyLion, serving nearly 500 million users in over 150 countries. We provide cybersecurity, online privacy, identity protection, and financial wellness products. Our mission centers on helping people manage and secure their digital and financial lives. We value diverse experiences and ideas, and we see AI as a partner for innovation. Gen Digital encourages autonomy, supports career growth, and offers flexible work options, generous time off, competitive pay, and wellness programs. The company culture emphasizes customer satisfaction, open discussion, experimentation, and continuous learning. Team members collaborate in an environment that respects and values differences as strengths. Senior Staff Data Engineer – Role Overview The Senior Staff Data Engineer will serve as a senior technical leader within the organization. This role focuses on designing and implementing large-scale data solutions that support Gen Digital’s cybersecurity platform strategy. The position combines deep technical skill with organizational influence. Key responsibilities include: Designing complex data architectures for enterprise-scale needs Implementing solutions that support a multi-petabyte data infrastructure Mentoring and guiding engineering teams Shaping the technical vision for data systems serving millions of users Location Chennai, India

About Gen Digital Inc. Gen Digital Inc. brings together trusted consumer brands like Norton, Avast, LifeLock, and MoneyLion, serving nearly 500 million users in over 150 countries. The company’s mission centers on digital freedom, cybersecurity, online privacy, identity protection, and financial wellness. Gen’s legacy is built on helping people secure and manage their digital and financial lives. Employees at Gen benefit from flexible work options, comprehensive support, and resources to help them succeed. The company values open communication, experimentation, and continuous learning, and actively welcomes diverse backgrounds and perspectives. Gen offers competitive pay, benefits, and wellness programs to support work-life balance. Role Overview: Senior Data Platform Engineer Location: Chennai, India The Senior Data Platform Engineer joins the Data Platform Operations team, focusing on the daily management and monitoring of Gen’s data platforms and pipelines. This role is key to ensuring smooth operations, ongoing improvements, and reliable maintenance of data infrastructure. What You Will Do Oversee daily operations and health of data platforms and pipelines Drive enhancements for operational efficiency and platform maintenance Work closely with data engineers, analysts, and platform teams Maintain platform stability, observability, and cost-effectiveness Support readiness for new data use cases and business needs What Gen Values Customer focus and a collaborative approach Openness to new ideas and continuous improvement Respect for diverse experiences and backgrounds Supportive teamwork and recognition of individual strengths If the mission and values at Gen resonate, consider exploring a career with the team in Chennai.

About Us:BigID is a pioneering tech startup specializing in cutting-edge solutions for data security, compliance, privacy, and AI data management. We are at the forefront of the data landscape, empowering our customers to mitigate risks, foster business innovation, achieve compliance, build trust, make informed decisions, and maximize the value of their data.We are committed to building a global team united by a passion for innovation and advanced technology. BigID has received numerous accolades, including:Named a Hot Company in Artificial Intelligence and Machine Learning at the Global InfoSec AwardsListed in Citizens JMP Cyber 66 as one of the Hottest Privately Held Cybersecurity CompaniesCRN 100 list recognizes BigID as one of the 20 Coolest Identity Access Management and Data Protection Companies for three consecutive yearsRanked among the DUNS 100 Best Tech Companies to Work forFeatured as a Top 3 Big Data and AI Vendor to Watch in the 2023 BigDATAwire Readers' and Editors' Choice AwardsIncluded in the 2024 Inc. 5000 list for the fourth consecutive year!Shortlisted for the 2024 AI Awards in the Best Use of AI in Cybersecurity categoryAt BigID, our team is the cornerstone of our success. Join our dynamic, people-centric culture where you’ll have the opportunity to collaborate with some of the most talented professionals in the industry who prioritize innovation, diversity, integrity, and teamwork.Who We Are Looking For:We are on the hunt for a Senior Data Platform Engineer to enhance our Data Platform team. The ideal candidate will possess substantial experience in data engineering, particularly with Kafka and Elasticsearch, to design and maintain our robust data platforms. You will collaborate closely with cross-functional teams to ensure the scalability and reliability of our data solutions.Role Overview:As a Senior Data Platform Engineer, you will be instrumental in the design, development, maintenance, troubleshooting, and implementation of our big data architecture. Your proficiency in Elastic, Kafka, and Node.js will play a vital role in ensuring the scalability and performance of our data systems.Key Responsibilities:Develop data processing pipelines utilizing Kafka for real-time data streaming.Enhance and manage search functionalities leveraging Elastic technologies.Work alongside product managers, data analysts, and stakeholders to gather requirements and translate them into technical specifications.Lead code reviews and promote best practices in coding and data handling.

Join our dynamic team at Squircle IT Consulting Services Pvt Ltd as a Data Scientist and Data Engineer. In this exciting role, you will leverage your expertise in data science and engineering to drive innovative solutions and enhance data-driven decision-making across various projects.

Lead Data EngineerAs a Lead Data Engineer at Brillio, you will play a pivotal role in designing, implementing, and managing robust data pipelines and architectures that drive our data-driven decision-making processes. You will lead a team of talented engineers and collaborate with cross-functional teams to deliver innovative data solutions that empower our clients and enhance their operational efficiencies.

Role overview Forbes Advisor seeks a Lead Data Engineer based in Chennai. The position centers on designing and building data infrastructure to support business strategy and deliver insights. Maintaining data integrity and ensuring that information is available to those who need it are key priorities. Main responsibilities Direct the architecture and implementation of data systems Convert raw data into insights that inform business decisions Collaborate with teams across the company to understand and meet their data needs Create solutions that enable better decision-making through data Collaboration This role works closely with cross-functional teams. Gathering input, clarifying requirements, and delivering tailored data solutions are part of daily work to address real business challenges.

Lead Data EngineerJoin our innovative team at Brillio as a Lead Data Engineer, where you will spearhead data engineering initiatives and lead a talented group of engineers. Your expertise will guide our projects, ensuring the development of robust data solutions that drive business decisions. Collaborate with AI and data engineering experts, and leverage cutting-edge technologies to transform data into actionable insights.

Join Infotel India as a seasoned ETL, Python, and Visualization Lead in our vibrant team. In this pivotal role, you'll spearhead the design and execution of robust ETL processes, craft innovative Python solutions, and develop insightful data visualizations to drive strategic business decisions. As a leader, you will work in synergy with stakeholders, data engineers, and analysts to establish efficient data workflows that yield impactful insights. This is a remarkable opportunity for a technical leader to significantly influence our data initiatives and enhance client outcomes.Key Responsibilities Design, develop, and maintain scalable ETL pipelines. Manage and optimize large datasets to guarantee data quality and integrity. Create data processing solutions utilizing Python. Develop interactive dashboards and reports leveraging data visualization tools such as Power BI, Tableau, or equivalents. Mentor and guide junior team members. Collaborate with cross-functional teams to foster data-driven decision-making. Enhance data workflows and boost system performance.

We are seeking a highly proficient Technology Engineer with extensive experience in Redis, Kafka, and cloud-native platforms to become a vital part of the Technology Engineering team at ENBD. The selected candidate will be pivotal in designing, managing, and optimizing high-performance data platforms and streaming architectures within a cloud-native framework.

Role overview BigID seeks a Senior Software Engineer in Chennai to strengthen its data privacy platform. This position centers on building and enhancing software that supports privacy and large-scale data management. What you will do Design, develop, and test software applications that meet high standards for quality and reliability. Collaborate with engineering, product, and QA teams to deliver new features and improvements. Troubleshoot and resolve technical issues as they arise. Maintain and enhance existing systems to ensure ongoing performance and stability. Requirements Strong coding skills with experience building scalable software solutions. Ability to work effectively with others to address complex technical challenges. Background in designing and maintaining software within collaborative teams.