About the job
Spear AI is on the lookout for an experienced Data Engineer to join our team as we develop a pioneering data management and AI platform focused on maritime domain awareness. As a growing defense contractor, we pride ourselves on delivering innovative solutions that bolster national security. Our organizational culture promotes a flat structure, allowing every team member to have a significant impact, collaborate closely with leadership, and engage in meaningful projects across our Hardware, Software, or Services divisions. You will work alongside a talented group of professionals dedicated to excellence and enhancing the capabilities that safeguard our nation.
At Spear AI, we specialize in the creation of sonobuoy sensors that gather edge data from aquatic environments, and we collaborate with the U. S. Navy to process their SONAR data. In this role, you will have the opportunity to contribute to real-world projects that directly influence the capabilities of warfighters and the success of their missions.
Key Responsibilities
As a member of our dynamic team, you will take on a variety of responsibilities, including:
Developing real-time data pipelines utilizing MQTT and Redpanda for effective stream processing.
Creating offline data pipelines with Dagster for batch processing.
Parsing and processing binary message formats from diverse data sources.
Building robust data warehouses using Postgres, Apache Iceberg, Parquet, and S3.
Designing data models that facilitate high-performance querying.
Validating and normalizing various data sources.
Enhancing local development and CI/CD processes using contemporary tools and GitHub Actions.
Your Profile
We are looking for a candidate with a strong foundation in Software Engineering who embodies our core values:
Detail-oriented with a passion for code quality; you ensure your code is well-structured, formatted, and thoroughly tested.
Committed to correctness; you take pride in building reliable libraries and services that users can trust.
Performance-focused; you continually strive to optimize query speeds and overall system efficiency.
Inquisitive and proactive; you enjoy delving into the intricacies of systems, developing prototypes, and conducting experiments to deepen your understanding.

