Azure Databricks & Synapse Data Engineer - PySpark, Spark SQL, Delta Lake

Azure Databricks Synapse Data Engineer PySpark, Spark SQL, Delta LakeJob Details

Job Title: Azure Databricks Synapse Data Engineer
Location: Indianapolis, Indiana
Work Arrangement: Hybrid
Onsite Requirement: One day per week onsite
Residency Requirement: Indiana residents only; local candidates only
Number of Openings: 1
Employment Type: Contract
Estimated Start Date: November 2, 2026
Estimated End Date: November 30, 2027
Contract Duration: Through November 30, 2027
Potential Extension/Renewal: Not Provided
Submission Deadline: October 28, 2026
Working Hours: Not Provided
Hours Per Week: 40 hours
Candidate Pay Rate: $43.35 per hour
Education: Master's degree, bachelor's degree with relevant experience, or qualifying hands-on data engineering experience, as detailed below
Background Check: Not Provided

Position Overview

The Azure Databricks Synapse Data Engineer designs, builds, and maintains secure, scalable data solutions across Azure Databricks and Azure Synapse Analytics.

This role supports data integration, pipeline development, data modeling, governance, and documentation while collaborating with cross-functional teams to deliver high-quality, reliable data products. Responsibilities include developing batch and streaming pipelines, optimizing data warehouse workloads, enforcing data governance standards, gathering requirements, reviewing solutions, and providing technical support.

The position also supports operational processes, contributes to policy and procedure improvements, and may provide mentorship or supervision based on agency needs.

Education

Candidates must meet one of the following qualification pathways:

• Master's degree in data science, analytics, informatics, computer science, mathematics, engineering, or a related field

• Bachelor's degree in a related field plus 2+ years of relevant experience

• 4+ years of hands-on experience in a similar data engineering role

Relevant certifications and coursework may substitute for formal education or experience requirements.

Required Skills / Experience

The requisition specifies 4 years of experience for the technical qualifications listed below unless otherwise indicated

• Databricks Development: Developing with Databricks notebooks in PySpark, Spark SQL, and R, including transformations, joins, aggregations, and window functions

• Delta Lake: ACID transactions, MERGE upserts, schema enforcement and evolution, time travel, and table maintenance using OPTIMIZE, Z-order, and VACUUM

• Lakehouse Architecture: Medallion (bronze/silver/gold) architecture and incremental ingestion using Auto Loader, Structured Streaming, or Lakeflow/DLT

• Unity Catalog: Workspace catalogs, schemas, permissions, lineage, external locations, and storage credentials

• Performance Optimization: Right-sizing clusters and SQL warehouses and diagnosing partitioning, shuffle, skew, and caching issues using Spark UI

• Workflow Orchestration: Scheduling and managing dependencies with Azure Data Factory (ADF) pipelines and Databricks Workflows/Jobs, including Git folders (Repos) for source control

• Azure Synapse Data Warehousing: Designing MPP warehouse tables, including hash, round-robin, and replicated distribution, columnstore indexes, and partitioning

• ADLS Gen2 Querying: Querying Parquet, Delta, and CSV files using OPENROWSET, external tables, and views

• Dimensional Modeling: Building star-schema facts, dimensions, stored procedures, and views, including SCD Type 1 and Type 2

• Data Loading: Developing with COPY INTO, PolyBase, and CETAS, and building parameterized Synapse/ADF pipelines using copy activities, data flows, and triggers

• Warehouse Workload Management: Workload management, resource classes, statistics, materialized views, result-set caching, and pool scaling or pausing

• Azure Storage and Security: ADLS Gen2 folder/file-format design using Parquet/Delta, Entra ID access control, managed identities, and Key Vault

• CI/CD and Source Control: Git and CI/CD in Azure DevOps or GitHub, including branching, pull requests, and deploying Synapse artifacts and Databricks Asset Bundles

• Monitoring and Troubleshooting: Monitoring pipelines and workloads with Azure Monitor/Log Analytics, setting up alerting, and troubleshooting failed jobs

• Lakehouse Integration: Using Microsoft lakehouse patterns, including interoperability and migration paths between Synapse and Databricks

• Communication: Strong written and verbal communication skills

Preferred Skills / ExperienceHighly Desired

• Databricks Certified Data Engineer Associate certification; Databricks Certified Data Engineer Professional is a plus

• Microsoft Certified: Azure Data Engineer Associate (DP-203), if already held

Desired

• 4 years of experience with Power BI connectivity to Synapse, including Import versus DirectQuery and understanding how data model design affects dashboard performance

Job Responsibilities

• Assist with the data technology strategy, including resolving strategic data integration issues related to data quality, stewardship, real-time and event-based integration, and developing a unified vision for data integration

• Provide direction and oversight for designing and implementing data solutions for integration and synthesis across multiple secure systems

• Assist with defining, implementing, and maintaining data management policies and data engineering design standards

• Provide advice, training, and technical support for various projects

• Participate in requirements gathering, solution reviews, and explaining technical complexities to non-technical personnel

• Collaborate with solution leads, project managers, data architects, data scientists, and data analysts on solution design, architecture, and implementation

• Extract, transform, and load data from a wide variety of data sources using various data engineering tools and methods

• Design, build, and maintain batch and streaming data pipelines in Azure Databricks using PySpark, Spark SQL, and Delta Lake, following a medallion lakehouse architecture

• Develop, load, and tune dimensional data models and SQL workloads in Azure Synapse Analytics, including dedicated and serverless SQL pools, to support reporting and dashboard needs

• Orchestrate and monitor data workflows using Databricks Workflows/Jobs and Synapse or Azure Data Factory pipelines, including scheduling, alerting, and failure recovery

• Apply data governance and security controls, including Unity Catalog permissions, role-based access, row- and column-level security, and data masking, to protect sensitive health data (PHI/PII) in accordance with HIPAA and agency policy

• Document and demonstrate solutions through documentation, flowcharts, layouts, diagrams, charts, code comments, and clear code

• Conduct systems analysis and recommend changes to operational policies and procedures

• Perform other duties, responsibilities, and activities as assigned

Work Schedule Work Arrangement

• Hybrid schedule with onsite work one day per week

• 40 hours per week

• Specific daily working hours are not provided

Residency Location Requirements

• Candidates must currently be located in Indiana

• Local candidates only

• Onsite work is required one day per week in Indianapolis, Indiana

Contract Information

• Contract start date: November 2, 2026

• Expected end date: November 30, 2027

• Candidate pay rate: $43.35 per hour

• Submission deadline: October 28, 2026

Ideal Candidate

The strongest candidate is likely to be an experienced Azure data engineering professional with hands-on expertise in both Azure Databricks and Azure Synapse Analytics. The core of the role is building and optimizing production-grade data pipelines, implementing medallion lakehouse architecture, developing dimensional warehouse models, and managing secure, reliable data workflows.

A particularly competitive candidate would demonstrate strong PySpark, Spark SQL, Delta Lake, Unity Catalog, Synapse SQL, ADLS Gen2, orchestration, and CI/CD capabilities. Experience protecting sensitive health information through appropriate access controls and governance practices is also important to the responsibilities of the position.

Databricks certification and an existing Azure Data Engineer Associate certification are strong differentiators, while Power BI connectivity and dashboard performance knowledge are desirable.

Candidates with only general data science, reporting, or dashboard development experience may be less aligned if they lack the hands-on data engineering, pipeline development, lakehouse, and Synapse warehouse skills emphasized in the requisition.

Important: Indiana residency is mandatory, and candidates must be willing to work onsite one day per week.