The Data & Operations AI Specialist serves as the Level 3 technical lead for Artificial Intelligence and Data Platform estate. You will be responsible for the architecture, engineering, and advanced troubleshooting of AI infrastructure, data pipelines, and MLOps lifecycles across a multi-cloud environment (Azure and OCI).
Responsibilities:
AI Infrastructure & Platform Engineering
- Design & Architecture: Maintain the monitoring architecture for AI/ML platforms and configure advanced dashboards in Grafana and Azure Monitor.
- Environment Governance: Manage Azure Machine Learning (AML) workspace configurations, compute targets, and Databricks cluster lifecycles (including runtime versions and platform patching).
- Resource Optimization: Oversee GPU resource allocation, reserved capacity, and cost-performance optimization to align with FinOps goals.
- Security Integration: Ensure all AI services utilize private endpoints, VNET integration, and RBAC controls to protect sensitive citizen data.
Data Pipeline & ETL Management
- Pipeline Engineering: Own the design, optimization, and remediation of Azure Data Factory (ADF) and Synapse pipelines.
- Advanced Troubleshooting: Resolve complex bottlenecks related to authentication failures, data format changes, and ETL performance.
- SOP Leadership: Author step-by-step Standard Operating Procedures (SOPs) for the L1 NOC team to handle routine monitoring and first-line triage.
MLOps & Model Lifecycle
- Automation: Implement CI/CD pipelines for model training, testing, and deployment to AML endpoints.
- Model Reliability: Configure data drift detection thresholds and automated retraining triggers.
- Recovery Operations: Develop self-healing scripts and automated recovery runbooks for critical AI workflows.
Governance & Compliance
- Audit Management: Implement and maintain audit logging for all AI decisions and model outputs, ensuring logs flow to the SIEM/vSOC.
- Regulatory Alignment: Conduct quarterly AI governance reviews to ensure compliance with NESA standards and data privacy guidelines.
Requirements
- AI/ML Platforms: Deep expertise in Azure Machine Learning and Databricks.
- Data Integration: Proficiency in Azure Data Factory and Synapse.
- Infrastructure-as-Code (IaC): Experience with Terraform or ARM Templates for reproducible deployments.
- Observability: Ability to use Dynatrace, Grafana, and Azure Monitor for deep-tier diagnostics.
- Containerization: Knowledge of AKS, Istio Service Mesh, and KEDA.
- ITIL Mastery: Strong understanding of ITIL-aligned Incident, Change, and Problem management.
- Security Mindset: Familiarity with NESA standards and UAE data residency requirements.
- Technical Writing: Ability to draft complex SOPs and Root Cause Analysis (RCA) documents within 48 hours of an incident.
- Certifications: Microsoft Azure Data Scientist Associate or Azure AI Engineer Associate is highly preferred.