About the job
Join the dynamic Data Acquisition team at OpenAI, where we spearhead the data collection efforts essential for powering our advanced model training operations. Our team plays a pivotal role in managing web crawling and GPTBot services, collaborating closely with Data Processing, Architecture, and Scaling teams. We are seeking a talented Senior Software Engineer to enhance our Data Acquisition initiatives.
Key Responsibilities:
Lead engineering projects focused on data acquisition, including web crawling, data ingestion, and search optimization.
Collaborate effectively with cross-functional teams to maintain seamless data flow and system performance.
Engage with the legal team to navigate compliance and data privacy regulations.
Design and implement robust distributed systems capable of processing petabytes of data.
Develop algorithms for efficient data indexing and search functionalities.
Build and sustain backend services for data storage, including experience with key-value databases and data synchronization.
Deploy solutions in a Kubernetes Infrastructure-as-Code environment while conducting regular system audits.
Conduct experiments on data to derive insights that drive system enhancements.
Qualifications:
Bachelor's, Master's, or PhD in Computer Science or a related discipline.
A minimum of 6 years of professional experience in software development.
Prior experience with large-scale web crawlers is a significant advantage.
In-depth knowledge of large stateful distributed systems and data processing techniques.
Expertise in Kubernetes and familiarity with Infrastructure-as-Code practices.
A proactive approach to exploring new technologies and methodologies.
Strong ability to juggle multiple tasks and adapt to changing priorities.
Excellent communication skills, both written and verbal.
About OpenAI:
OpenAI is at the forefront of artificial intelligence research and deployment, dedicated to ensuring that the benefits of general-purpose AI are shared by all of humanity. We strive to push the boundaries of innovation while adhering to ethical standards.

