companyDemandMatrix logo

Lead Big Data Engineer

On-site Full-time

Clicking Apply Now takes you to AutoApply where you can tailor your resume and apply.


Unlock Your Potential

Generate Job-Optimized Resume

One Click And Our AI Optimizes Your Resume to Match The Job Description.

Is Your Resume Optimized For This Role?

Find Out If You're Highlighting The Right Skills And Fix What's Missing

Experience Level

Mid to Senior

Qualifications

Essential Qualifications:Minimum of 7 years of hands-on software development experience, specifically in big data and large data pipeline ecosystems. At least 3 years of practical experience in building services and pipelines using Python. Proficient in various data processing systems including streaming, event, and batch processing (e.g., Spark, Hadoop/MapReduce). Familiarity with NoSQL databases such as MongoDB, Elasticsearch, or HBase. Solid understanding of data models, sharding, and data location strategies for distributed data stores in high-throughput environments and their implications for non-structured text data processing. Experience in deploying scalable and highly available systems using AWS or GCP. Desirable Skills:Experience with Docker and Kubernetes. Exposure to CI/CD practices. Knowledge of web crawling and scraping techniques.

About the job

We are on the lookout for a dynamic Big Data Lead who will play a pivotal role in taking our data capabilities to the next level. Your primary focus will be on solving intricate data challenges using cutting-edge big data technologies, dedicating almost 50% of your time to hands-on coding tasks.

Key Responsibilities:

  • Develop and manage large-scale text data processing and event-driven data pipelines, optimizing performance across CPU, network IO, and disk IO.
  • Utilize cloud-native services on platforms such as AWS and GCP to enhance data processing and storage capabilities.

About You:

  • You possess a strong foundation in computer engineering, with expertise in Unix, data structures, and algorithms.
  • You have successfully designed and implemented multiple big data modules and data pipelines capable of handling vast volumes of data.
  • Passionate about technology, you have experience in initiating and executing projects from the ground up.

About DemandMatrix

At DemandMatrix, we are committed to transforming the $100 billion sales and marketing intelligence sector through the innovative use of domain knowledge, machine learning, and artificial intelligence. Trusted by Fortune 100 companies including Microsoft, Google, Adobe, Amazon, and IBM, we excel at identifying their next prospective customers.

Similar jobs

Tailoring 0 resumes

We'll move completed jobs to Ready to Apply automatically.