Principal Data Engineer

About this role

About Us

Abacus Insights is transforming how data works for health plans. Our mission is simple: make healthcare data usable, so the people responsible for care and cost decisions can act faster, with confidence.

We help health plans break down data silos to create a single, trusted data foundation. That foundation powers better decisions—so plans can improve outcomes, reduce waste, and deliver better experiences for members and providers alike. Backed by $100M from top investors, we’re tackling big challenges in an industry that’s ready for change. Our platform enables GenAI use cases by delivering clean, connected, and reliable healthcare data to support automation, prioritization, and decision workflows—and it’s why we are leading the way.

Our innovation begins with people. We are bold, curious, and collaborative—because the best ideas come from working together. We embrace the thoughtful use of AI and automation to drive innovation and efficiency, and we look for individuals who are curious and adaptable—those excited to leverage emerging technologies to enhance how we work—while keeping human insight, connection, and our clients at the center of every decision.

Ready to make an impact? Join us and let’s build the future together.

About the Role

We are seeking an accomplished Principal Data Engineer to join our Tech Ops organization and play a critical role in designing and scaling our enterprise data platform. This is a senior, hands‑on technical leadership role focused on architecting complex data integration solutions, solving high‑impact technical challenges, and setting standards for data engineering excellence across the organization.

In this role, you will work directly with clients, data partners, and internal engineering teams to design, implement, and evolve large‑scale data pipelines in a modern cloud environment. As a recognized technical authority, you will influence architectural decisions, guide best practices, and support mission‑critical healthcare data workflows across the full data lifecycle.

Your day to day

• Architect Enterprise‑Scale Data Solutions: Design, build, and evolve high‑volume batch and real‑time data pipelines using PySpark, SparkSQL, Databricks Workflows, and distributed processing frameworks.

• Own Platform‑Level Integrations: Develop end‑to‑end ingestion and transformation frameworks integrating Databricks, Snowflake, AWS services (such as S3, SQS, Lambda), and external data provider APIs, with a strong focus on data quality, lineage, and schema evolution.

• Lead Technical Design for Clients: Serve as the technical lead for complex client implementations, defining highly available, fault‑tolerant architectures across multi‑account cloud environments.

• Translate Business Needs into Architecture: Convert complex business and regulatory requirements into scalable technical designs, detailed specifications, and reusable engineering patterns.

• Set Engineering Standards: Establish and champion best practices across CI/CD, code quality, testing, orchestration, monitoring, logging, and observability for data platforms.

• Ensure Security & Compliance: Design and implement security‑first data solutions, including RBAC, encryption, PHI handling, auditability, and alignment with HIPAA and SOC 2 requirements.

• Optimize Performance & Cost: Profile and tune compute workloads, cluster configurations, partitioning strategies, indexing, and caching across Databricks and Snowflake environments.

• Provide Technical Mentorship: Mentor senior and junior engineers, conduct design and code reviews, and raise the overall technical bar across teams.

• Produce Technical Artifacts: Create clear documentation, including architecture diagrams, runbooks, and operational standards that support scalable delivery.

What you bring to the team

• Deep Data Engineering Experience: 7+ years of hands‑on experience designing and operating large‑scale, distributed data systems in cloud‑based environments.

• Advanced Programming Skills: Expert‑level proficiency in Python, SQL, and PySpark, including performance‑optimized distributed transformations.

• ETL / ELT Architecture Expertise: Proven experience building and operating production‑grade ETL/ELT pipelines using Databricks, Airflow, or similar orchestration frameworks.

• Cloud Platform Expertise: Strong working knowledge of AWS‑based data services (e.g., S3, SQS, Lambda, IAM) or equivalent cloud technologies.

• Data Platform & Streaming Knowledge: Experience working with dbt, Delta Lake, Kafka, or event‑driven architectures in modern data platforms.

• Warehouse & Modeling Experience: Hands‑on experience with Snowflake or other cloud data warehouses, including schema design and performance optimization.

• Distributed Systems Expertise: Demonstrated ability to design scalable, resilient systems requiring specialized knowledge of distributed computing and cloud‑scale data processing.

• Healthcare Data Exposure: Working experience with healthcare data domains such as claims, eligibility, provider, or clinical datasets.

• Technical Communication: Ability to clearly communicate complex technical concepts to both technical and non‑technical partners.

• Education: Bachelors or Masters degree in Computer Science, Engineering, Data Science, or a related field.

What we would like to see, but not required

• Experience in large‑scale healthcare payer or analytics environments.

• Exposure to machine learning or advanced analytics workflows.

• Familiarity with DevOps or platform engineering practices applied to data systems.

• Experience developing reusable data frameworks or shared platform capabilities.

Compensation: Compensation for this role is based on experience, skills, and location, and includes base salary plus eligibility for performance bonuses and equity grants.

What you’ll get in return

• Unlimited paid time off – recharge when you need it

• Work from anywhere – flexibility to fit your life

• Compreh

Similar jobs

• Principal Data Engineer — Capitaltg · Remote
• Principal Data Engineer — Drivewealth · New York, New York, United States
• Principal Data Engineer — Dragos · Remote
• Principal Data Engineer — Rdccareers · Austin, Texas, United States
• Principal Data Platform Engineer — Doma · Remote
• Principal Data Engineer, People Data — Okta · Bellevue, Washington; Chicago, Illinois; San Francisco, California; Washington, DC
• Principal Data Architect & Engineer — Careportalinc · USA
• Principal Data Architect — Commvault · United States