Sr Data Engineer, Data Analytics & Intelligence, NA

About this role

About Vantage Data Centers

Vantage Data Centers powers, cools, protects and connects the technology of the world’s well-known hyperscalers, cloud providers and large enterprises. Developing and operating across North America, EMEA and Asia Pacific, Vantage has evolved data center design in innovative ways to deliver dramatic gains in reliability, efficiency and sustainability in flexible environments that can scale as quickly as the market demands.

Operational Excellence Data Team Within Operational Excellence, the Data Analytics & Intelligence function enables Operations to move from reactive reporting to proactive, insight-driven execution. The team is responsible for building trusted data foundations, governed KPI frameworks, operational intelligence products, and AI-ready data assets that support performance visibility, decision-making, predictive insights, and scalable operational excellence across North America. This work directly supports Operational Excellence's mission of embedding delivery rigor, process discipline, and data intelligence into how Operations plans, executes, predicts, and continuously improves.

Position Overview This position will be based on-site at our office in Denver, CO. in alignmen t with our flexible work policy. (3 days on site required, 2 days flexible).

Vantage Data Centers is seeking a Sr Data Engineer to help build, operate, and scale the governed data foundation for Operations, North America. This role is designed for an engineer who can independently deliver production-ready pipelines, curated datasets, semantic-model inputs, and AI-ready data products that support reporting, Executive reporting insight preparation, and the emerging AI Insight Solution. As part of the Data Engineering & Business Intelligence team, you will be responsible for delivering reliable data products that support analytics, AI data agents, operational intelligence, reporting, and an emerging AI-enabled platform. You will work closely with IT Global, solution build teams, business SMEs, and data governance partners to ensure data products are secure, reusable, explainable, and aligned with enterprise AI / Fabric direction. Success in this position requires comfort with ambiguity, strong execution discipline, and accountability for building trusted data assets that can be reused across analytics, operational intelligence, and AI-enabled use cases.

Essential Job Functions • Design, build, and maintain reliable, scalable data pipelines using Python and PySpark on the Microsoft Azure data platform. 

• Develop and operate batch and incremental data pipelines leveraging Azure Data Factory for orchestration and Azure Data Lake Storage Gen2 as the primary data store. 

• Build and maintain curated lakehouse / gold-layer datasets and semantic-model inputs that support governed operational insights and AI-enabled consumption. 

• Independently implement SQL- and Spark-based transformations to produce curated datasets that support enterprise reporting, analytics, AI-enabled insight preparation, and downstream consumption. 

• Take ownership of assigned data pipelines and datasets, including monitoring, troubleshooting, performance optimization, documentation, and production support. 

• Work with Azure Synapse, Microsoft Fabric / Lakehouse patterns where applicable, and related Azure analytics services to support analytical workloads and data consumption patterns. 

• Prepare structured operational data for AI-enabled use cases by documenting business rules, source lineage, data reliability constraints, known quality limitations, and data dictionary definitions. 

• Support source visibility, confidence context, and Data Reliability & Trust Indicator integration where applicable so downstream analytics and AI outputs can be understood and trusted. 

• Contribute to ontology, taxonomy, semantic model, and data dictionary alignment needed to connect operational context, KPIs, incidents, work orders, and other enterprise data domains. 

• Collaborate with business analysts, operations SMEs, data stewards, IT Global, and cross-functional stakeholders to translate requirements into practical, working data solutions. 

• Apply established data governance, security, access-control, data classification, and engineering standards to ensure compliant, maintainable, and scalable solutions. 

• Identify, document, and route data-quality issues to accountable owners, helping improve source correction rather than masking defects downstream. 

• Participate in code reviews, technical discussions, sprint planning, and platform improvement initiatives as an active contributor. 

• Proactively identify data quality issues, pipeline risks, platform dependencies, and improvement opportunities, and communicate them clearly in a fast-paced environment. 

Duties • Develop and maintain PySpark notebooks and jobs to ingest, transform, validate, and curate data within the enterprise data platform. 

• Build and modify Azure Data Factory pipelines for batch and incremental data ingestion. 

• Implement Spark-based transformations that write curated datasets to Azure Data Lake Storage Gen2 and/or Fabric Lakehouse patterns using established folder structures, naming conventions, and governance standards. 

• Create and maintain SQL views, tables, lakehouse objects, and semantic-model inputs to support analytics, operational intelligence, and AI-enabled consumption patterns. 

• Prepare datasets for Fabric Data Agent / AI agent use cases by documenting business rules, joins, grain, quality limitations, source lineage, and operational definitions. 

• Respond to pipeline failures, data validation issues, operational alerts, and data-quality escalations with clear root-cause analysis and practical remediation steps. 

• Perform performance tuning of Spark jobs and SQL workloads, including partitioning, filtering, incremental logic, query optimization, and resource-aware design within established architectural p

Similar jobs

• Sr Data Analyst, Data Science Analytics — The Home Depot · STORE SUPPORT CENTER, ATLANTA - 9090
• Sr Data Engineer — Beyondtrust · Johns Creek, GA
• Sr Data Scientist — Gartner · Stamford, CT | Irving, TX
• Sr Data Analyst — The Home Depot · STORE SUPPORT CENTER, ATLANTA - 9090
• Sr Data Analyst — Crossmediainc · New York, New York
• Sr. Data Analytics Engineer — Fabletics · Remote
• Sr Data Analyst, Customer Insights — Terminix · Remote
• Senior Data Engineer, Reporting & Analytics — Drata · Hybrid - San Francisco