Software Engineer, Distributed Data Systems

Join Cloudflare as a Principal Systems Engineer, Data, where you will lead innovative projects that enhance our data processing capabilities. You will work collaboratively with cross-functional teams to design, implement, and optimize systems that efficiently handle large-scale data. This role requires a deep understanding of systems engineering principles, strong analytical skills, and a passion for leveraging data to drive decisions. Your contributions will be pivotal in shaping the future of our data infrastructure.

Role overview Pinterest is seeking a Principal Engineer to join the Online Systems team, based in either San Francisco or Palo Alto. This position centers on designing and building scalable systems that serve millions of users. The focus is on enhancing user engagement and optimizing the performance of Pinterest's online platforms. What you will do Architect and develop large-scale applications built for high-traffic environments Collaborate with engineering, product, and design teams to deliver reliable solutions Lead technical decisions that influence the direction of Pinterest's online systems

Join Cloudflare as a Principal Systems Engineer specializing in Internet Protocol Communication (IPC). In this pivotal role, you will lead and innovate within our engineering team, playing a crucial part in enhancing our systems to support seamless communication over the internet. Your expertise will help shape our infrastructure and drive our mission to help build a better internet.

Join Cloudflare as a Principal Systems Engineer and play a pivotal role in shaping the future of our cutting-edge technology solutions. In this hybrid position, you will leverage your extensive expertise to design and implement innovative systems that enhance our service offerings.

Role Overview The City and County of San Francisco's Department of Technology is hiring a Principal System Integration Engineer focused on enterprise systems with Python expertise. This position leads major integration projects that connect and enhance a wide range of internal systems. What You Will Do Design, develop, and implement integration solutions for enterprise systems Lead complex projects that require coordination across multiple platforms and teams Work closely with colleagues from different departments to solve technical challenges Improve system performance and reliability through thoughtful integration work Collaboration and Impact This role works at the intersection of technology and city operations. Projects often require input from diverse stakeholders and have a direct impact on how services run across the organization.

Redpanda Data is building the Agentic Data Plane (ADP), a platform that connects AI agents with enterprise data and systems. The ADP supports real-time, autonomous reasoning and action by agentic applications, powered by Redpanda's multi-modal data streaming engine. Major organizations across industries, including Activision Blizzard, Cisco, Moody's, Texas Instruments, Vodafone, and two of the top five U.S. banks, rely on Redpanda to process hundreds of terabytes of data every day. Backed by investors such as Lightspeed, GV, and Haystack VC, Redpanda operates as a globally distributed, people-first company. Role overview The Principal Software Engineer will architect and develop the Agentic Data Plane, which serves as the control and execution layer for AI agents interacting with enterprise data. This system enables agents to access, analyze, and act on data in real time, while providing human operators with oversight and control for secure operations. The ADP brings together Redpanda's low-latency streaming technology, a distributed query engine for real-time context, a library of over 300 data connectors, and a global policy and observability framework. This framework enforces access controls, records agent actions, and supports replayable audits. What you will do Design and build the core architecture of the Agentic Data Plane, focusing on secure and efficient data interaction for AI agents. Integrate streaming, query, and policy enforcement components to support real-time, autonomous agent operations. Monitor developments in the agentic AI field and translate research into engineering proposals and product strategies. Work closely with Engineering, Product, and Go-To-Market teams, as well as key customers, to shape the direction of the ADP.

Zyphra is a cutting-edge artificial intelligence firm located in the heart of San Francisco, California, dedicated to advancing technology across various modalities.About the Position:We are seeking a Data Engineer - Multimodal Systems to play a pivotal role in the enhancement and expansion of Zyphra's datasets and data pipelines. This position offers a unique opportunity to collaborate with diverse teams and contribute to innovative data solutions. You will engage in the collection of extensive datasets and the development and optimization of high-performance parallel data pipelines.Your Responsibilities Will Include:Executing large-scale data collection across multiple modalities, including text, audio, and image.Designing and implementing highly efficient, parallelized data processing pipelines that integrate various modalities.Conducting rigorous experimental ablations to evaluate the effectiveness of new data enhancements.Candidate Requirements:Proven ability in implementation and prototyping.Capability to transform ideas into experimental frameworks swiftly.Strong collaborative skills, thriving in a dynamic research environment.Eagerness to learn and apply new concepts effectively.Exceptional communication and teamwork skills, capable of contributing to both research and large-scale engineering projects.Preferred Qualifications:Experience in the collection, management, and processing of large datasets.Familiarity with parallel programming frameworks in Python, such as Dask.In-depth understanding of state-of-the-art dataset curation practices.A detail-oriented mindset with a passion for data integrity and verification.Strong foundation in experimental methodologies for conducting thorough ablation studies and hypothesis testing.Knowledge and interest in large-scale, highly parallel data processing systems.Proficiency in PyTorch and Python.Experience with large, complex codebases and the ability to quickly become productive within them.Published research in respected machine learning venues.Postgraduate degree in a relevant field is a plus.

Join the City and County of San Francisco as a Principal Information Systems Engineer specializing in Systems! In this pivotal role, you will lead the development and implementation of advanced information systems, ensuring the highest levels of performance and security. Collaborate with various departments citywide to deliver innovative solutions that enhance operational efficiency and foster public service excellence.

Axon Enterprise develops devices and cloud software aimed at improving safety and justice. Teams value honesty, compassion, and diverse perspectives, working together to make a meaningful impact. The company encourages initiative and personal growth, recognizing the importance of each person’s contribution. Role overview The Principal Data Scientist will join the Office of the Chief Accounting Officer in San Francisco. This position blends technical data science work with strategic collaboration across finance and technology groups. A strong background in finance and accounting is required for success in this role. Main focus Half of the role involves building data analytics tools and pipelines, using AI to speed up large-scale projects such as mergers and acquisitions integration, ERP deployments, data remediation, and SOX compliance. The other half centers on partnering with Controllership, Finance Operations, FP&A, Internal Audit, and Technology to lay the groundwork for AI, automation, and audit readiness. Daily tasks will include using AI-assisted development tools, such as code generation and large language models. Key responsibilities Design and implement data analytics tools, automated validation frameworks, and dashboards to provide Finance Operations and Controllership with real-time insights into data quality across enterprise systems. Use AI to quickly prototype, test, and deploy these solutions. Collaborate with source-system owners in Revenue Operations, Procurement, Human Resources, and Engineering to address data issues at their source and improve upstream data integrity. Maintain data dictionaries, lineage documentation, and quality scorecards for core controllership data assets. Set and enforce standards for master data management (MDM). Work with the Chief Information Officer and IT to develop integration strategies that enable seamless data flow.

At Databricks, we are driven by a passion for empowering data teams to tackle the world’s most challenging problems — from transforming transportation to accelerating medical innovations. We achieve this by creating and maintaining the leading data and AI infrastructure platform, enabling our clients to leverage profound data insights for business enhancement. Founded by engineers with a customer-first mentality, we eagerly embrace every opportunity to tackle complex technical challenges, ranging from the design of next-generation UI/UX for data interactions to scaling our services across millions of virtual machines. Our journey has just begun.As a member of the Runtime team at Databricks, you will be instrumental in developing the next generation of distributed data storage and processing systems. These systems will surpass specialized SQL query engines in relational query performance while offering the programming abstractions necessary to support a variety of workloads, from ETL to data science.Example projects include:Apache Spark™: Contribute to the de facto open-source standard framework for big data.Data Plane Storage: Develop reliable and high-performance services and client libraries for managing vast amounts of data within cloud storage backends like AWS S3 and Azure Blob Store.Delta Lake: Design a storage management system that merges the scalability and cost-effectiveness of data lakes with the performance and reliability of data warehouses, providing features like ACID transactions and time travel.Delta Pipelines: Simplify the orchestration and operation of numerous data pipelines, enabling clients to deploy, test, and upgrade pipelines effortlessly.Performance Engineering: Create the next-generation query optimizer and execution engine that is fast, scalable, and robust.

Join Cloudflare as a Systems Engineer specializing in Data, where you will play a critical role in enhancing our infrastructure and ensuring the reliability of our services. You will collaborate with cross-functional teams to design, implement, and maintain systems that handle vast amounts of data efficiently and securely. Your contributions will be pivotal in optimizing performance and delivering exceptional user experiences.

P-1125SummaryAt Databricks, we are dedicated to empowering data teams to tackle some of the world's most pressing challenges, from detecting security threats to advancing cancer drug research. We achieve this by developing and maintaining a world-class data and AI infrastructure platform, allowing our clients to concentrate on the high-impact issues that are vital to their missions.Our engineering teams create sophisticated products that address real and significant needs. We continuously strive to push the frontiers of data and AI technology while ensuring resilience, security, and scalability that are essential for our customers' success on our platform.Our clients entrust Databricks with their most sensitive data, and the Trust & Safety team is committed to developing the most reliable data analytics and machine learning platform globally. Security Engineering plays a key role in Trust & Safety by safeguarding customer data from malicious threats. We are seeking senior leaders like you to shape the vision and strategy in this crucial area.

Role Overview Nextdata is hiring a Lead Principal Infrastructure Engineer in San Francisco, CA. This position focuses on building the foundation for a decentralized data mesh platform, supporting data ownership and enabling AI, machine learning, and analytics at scale. What You Will Do Develop automation solutions for provisioning and managing the Nextdata OS across multiple cloud platforms. Work closely with the founding engineering team to design and implement a secure, self-service infrastructure for future data product developers. Own the architecture and deployment of the OS, using infrastructure-as-code to ensure high code quality and scalability. Engage directly with customers to understand their requirements and translate feedback into technical improvements. Collaborate with product teams to align infrastructure capabilities with business needs. What You Bring Expertise in large-scale distributed systems and data infrastructure. Experience designing, deploying, and maintaining cloud-based platforms. Strong background in infrastructure-as-code and automation. Ability to work collaboratively with engineering and product teams. Comfort engaging with customers to gather feedback and requirements.

Lila Sciences is seeking a Principal Machine Learning Engineer based in San Francisco, CA. This position centers on designing and building advanced machine learning algorithms to support scientific progress in healthcare and agriculture. Role overview The Principal Machine Learning Engineer will develop and implement new algorithms that strengthen Lila Sciences’ product capabilities and support data-driven decisions. The work will directly affect both internal business outcomes and broader scientific initiatives. Collaboration and impact This role involves close collaboration with engineers and data scientists on projects that address a range of technical and scientific challenges. The solutions created will have a direct influence on the future of healthcare and agriculture by applying advanced analytics and machine learning techniques.

About GustoAt Gusto, our mission is to empower the small business economy. We tackle essential services like payroll, health insurance, 401(k)s, and HR, allowing business owners to concentrate on their passion and their customers. With teams in Denver, San Francisco, and New York, we proudly support over 400,000 small businesses across the nation, while fostering a workplace that reflects and honors the diverse customers we serve. Discover more about our Total Rewards philosophy.About the Role:We are seeking experienced engineers to join our teams focused on enhancing core platform capabilities, optimizing existing systems for scalability, and shaping the future of Gusto's Payroll and Tax systems. As a Principal Software Engineer, you will lead projects from inception to completion, influencing initial feature specifications, optimizing systems towards desired outcomes, executing projects, and maintaining the code that underpins key user experiences. If you are passionate about developing sophisticated software that has a significant impact on the modern economy, we invite you to join us!About the Team:The Payroll team is the core of Gusto’s operations. The Pay Group teams (responsible for payroll and tax) oversee the product, operational, and service aspects that Gusto utilizes to onboard clients, process payroll, calculate taxes and deductions, pay and file with tax authorities, and ensure compliance on behalf of our clients. This domain is intricate and offers opportunities to engage in system design that simplifies processes for both internal and external customers.Sub-Teams:Tax Accuracy: This group is responsible for gathering and maintaining the essential information required to ensure timely and accurate tax payments and filings for our clients, consisting of multiple teams covering various aspects of the process.Ledger: This team is developing a payroll ledger to monitor accrual accounting for payroll-related monetary movements, beginning to support the payroll domain, risk management, payments, and accounting.Engine: This team collaborates closely with the payroll compliance team to facilitate compliant payroll processes, such as minimum wage adjustments and regular pay rates.

About the TeamJoin our Online Data team, where we design and maintain the foundational online database and indexing services for OpenAI’s cutting-edge AI applications. We support the phenomenal growth of ChatGPT, the leading AI application globally, as well as Codex, the fastest-growing development toolset.Our mission is to uphold the reliability, accuracy, and scalability of our extensive online data infrastructure, enabling product and research teams to innovate swiftly without getting entangled in complex multi-region, multi-cloud, exabyte-scale data systems.About the RoleWe are on the lookout for an Engineering Manager to spearhead our Online Data Systems team. This role entails guiding a talented group of engineers dedicated to developing and managing hyperscale data storage and retrieval technologies.You will oversee the execution of highly complex engineering tasks in areas such as distributed query execution, multi-region data federation, self-orchestrating services, performance optimization, and more.This is a unique opportunity to shape cutting-edge technology at an unprecedented scale, where you won’t just be another cog in the machine but will have a significant impact on our trajectory.In this role, you will:Build, lead, and develop high-performing infrastructure engineering teams.Drive the advancement of OpenAI’s proprietary online data technologies, including our core database systems, indexing technologies, and vector search capabilities.Establish delivery metrics based on measurable reliability goals (SLOs, etc.) to ensure superior system performance and resilience.Advocate for the efficient use of agent technology to enhance execution speed.Minimize operational burdens and incident occurrences through improved abstractions and self-healing systems.You might thrive in this role if you:Possess a relentless pursuit of operational excellence, with hands-on experience in building...

Join Cloudflare as a Senior Systems Engineer specializing in our Data Platform. In this pivotal role, you'll be responsible for designing and implementing scalable systems that process vast amounts of data, ensuring performance and reliability across our services. Your expertise will drive innovative solutions that enhance our platform capabilities.

Join Databricks as a Principal Data Scientist and contribute to shaping the future of data analytics and machine learning. In this pivotal role, you will leverage your expertise to drive impactful projects, collaborate with cross-functional teams, and lead innovative research initiatives that enhance our product offerings. As a thought leader, you will utilize your extensive knowledge in data science to influence our strategic direction, ensuring our solutions remain at the forefront of technology.

About GustoAt Gusto, we are dedicated to empowering small businesses to flourish. Our comprehensive solutions encompass payroll, health insurance, 401(k)s, and HR, allowing business owners to concentrate on what they do best: serving their customers and mastering their craft. With teams located in Denver, San Francisco, and New York, we proudly support over 400,000 small businesses nationwide, while striving to cultivate a workplace that reflects and honors our diverse clientele. Discover more about our Total Rewards philosophy.About the Role:We are seeking an experienced and driven Principal Data Scientist with over 10 years of expertise to join our Product team. In this pivotal role, you will utilize your skills in experimentation, statistical inference, and causal analysis to influence strategic decision-making that propels our organizational success. The ideal candidate is a compelling data storyteller with robust statistical and programming capabilities, passionate about leveraging data to support small business growth.In this position, you will collaborate closely with our Product, Engineering, Design, Finance, and other Data teams to become an authority on your domain's data, establish and monitor key metrics to assess business performance, and analyze our Payroll, Benefits, and HR data to extract actionable insights. Additionally, you will adopt AI-assisted methodologies to enhance analysis efficiency, improve rigor, and broaden the dissemination of insights across Gusto.We have multiple senior Data Scientist roles available at this senior level, each concentrating on various critical business areas.Here’s what you’ll do day-to-day:Strategic Leadership: Develop the analytical strategy for a significant product or business segment. Pinpoint high-impact opportunities, set long-term objectives for measurement and experimentation, and synchronize executive stakeholders on data-driven initiatives.Execution Excellence: Drive the implementation of comprehensive analytical roadmaps that influence company strategy. Ensure alignment and uphold consistent analytical rigor and metric integrity across product domains.Scaled Impact: Beyond being an expert in statistical methods and fostering impact...