JOB DETAIL

Job Summary

The Data Platform Operations Specialist supports the reliable operation and controlled release of data-platform workloads across Databricks, Microsoft Fabric, Azure Data Factory, and related Azure services. The role works with data engineering, analytics, application, QA, security, cloud, Release Management, and business teams to prepare releases, monitor data workloads, troubleshoot failures, validate results, and maintain operational documentation.

This role executes approved operational procedures and contributes to automation, standards, and continuous improvement. It does not provide unrestricted production administration. Production deployments and changes should use approved Azure DevOps pipelines, protected environments, managed identities, service principals, and documented escalation paths. The specialist supports product teams while preserving security, change control, auditability, and clear production ownership.

Job Duties

• Monitor and support Databricks jobs, notebooks, workflows, clusters, libraries, runtime dependencies, and scheduled executions.
• Monitor and support Microsoft Fabric notebooks, pipelines, lakehouses, warehouses, dataflows, semantic models, and workspace activities.
• Monitor and support Azure Data Factory pipelines, datasets, linked services, triggers, integration runtimes, parameters, and data flows.
• Review scheduled runs, failed activities, retries, delays, data freshness, capacity concerns, and downstream impact.
• Perform approved operational procedures for restart, retry, pause, resume, reprocess, backfill, and controlled recovery.
• Support approved application and data-platform releases through Azure DevOps pipelines.
• Confirm release prerequisites, artifact versions, environment settings, approvals, testing evidence, and support coverage.
• Execute documented deployment, validation, and rollback steps under the direction of the Release Manager or designated release owner.
• Support Databricks notebook and job promotion, Fabric workspace and artifact promotion, and ADF pipeline deployment.
• Review deployment logs, pipeline results, approval gates, migration results, smoke tests, and post-release validation.
• Build and maintain straightforward Azure DevOps YAML pipeline tasks and reusable operational steps.
• Use Git, pull requests, branch policies, pipeline variables, artifacts, environments, approvals, and service connections according to team standards.
• Write Python and PowerShell scripts for validation, automation, reporting, deployment support, troubleshooting, and repetitive operational tasks.
• Use REST APIs and JSON to integrate with Azure DevOps, Azure, Databricks, Fabric, ADF, monitoring, and support systems.
• Perform approved checks for data freshness, completeness, row counts, schema compatibility, reconciliation, and expected pipeline results.
• Review Databricks, Fabric, ADF, Azure Monitor, Log Analytics, dashboards, logs, and alerts.
• Perform initial Level 3 operational triage for failed releases, failed data loads, notebook errors, trigger issues, access problems, configuration problems, and data-quality incidents.
• Collect timelines, error messages, deployment versions, affected pipelines, resource details, and business impact.
• Create or update support records and degradation documentation in accordance with the release-management workflow.
• Escalate complex application, Azure infrastructure, security, database, network, identity, or recurring reliability issues to the appropriate specialist.
• Use approved service connections, managed identities, Key Vault references, and least-privilege access patterns.
• Maintain runbooks for common release, monitoring, troubleshooting, retry, rollback, and data-validation procedures.
• Document known dependencies, schedules, support contacts, escalation routes, and recovery expectations.

Qualifications, Knowledge, Skills and Abilities

Education:

• Bachelor’s degree in computer science, Information Systems, Data Engineering, Software Engineering, or a related field preferred.

Overall Experience and Technical Skills:

• Five or more years of experience in data engineering, data-platform operations, cloud support, DevOps, release execution, or production application support required.
• Hands-on experience supporting production or non-production data pipelines across Databricks, Microsoft Fabric, Azure Data Factory, or comparable data-platform services.
• Experience with Azure DevOps or another CI/CD platform, including Git, pull requests, YAML pipelines, artifacts, variables, environments, approvals, and deployment history.
• Working knowledge of Python, PowerShell, YAML, JSON, REST APIs, command-line tooling, and basic automation for validation, reporting, deployment support, and operational tasks.
• Basic SQL skills, including querying, joins, validation, schema awareness, reconciliation, data-quality checks, and safe production-change practices.
• Experience with monitoring, incident triage, technical documentation, operational runbooks, release validation, rollback support, and post-release checks.
• Familiarity with Azure Storage, Key Vault, managed identities, role-based access control, Azure Monitor, Log Analytics, and related cloud data-platform support services.

Preferred Technical Skills and Training:

• Databricks deployment patterns or Databricks Asset Bundles.
• Microsoft Fabric Git integration or deployment pipelines.
• Azure Data Factory ARM, Bicep, or YAML-based deployment patterns.
• Infrastructure as code concepts using Bicep or Terraform.
• Data-quality tools, reconciliation checks, schema-drift detection, or data-observability practices.
• Basic CI/CD testing, release gates, rollback, feature flags, or deployment-slot concepts.

Certifications:

• Microsoft AZ-900 or equivalent Azure fundamentals training.
• Microsoft DP-203, AZ-400, Databricks, Fabric, Python, PowerShell, ITIL, Agile, or SRE training preferred.

Language(s):

• English

Other Knowledge, Skills & Abilities:

• Strong troubleshooting, organization, and attention to detail.
• Ability to follow documented procedures and recognize when a situation exceeds the approved procedure.
• Ability to communicate technical issues, business impact, status, and next steps clearly.
• Ability to work effectively with distributed teams across time zones.
• Ability to handle multiple operational tasks while protecting release quality and documentation standards.
• Ability to work calmly during failed jobs, failed releases, data-quality issues, and production incidents.
• Strong customer-service orientation for internal data, application, and business teams.
• Willingness to learn new data-platform technologies and supported operating standards.
• Ability to build trusted relationships while maintaining required security and change controls.