AI pipelines are transforming how enterprises handle data, but they’re also prime targets for security risks. In “Designing a metadata-driven ETL framework with Azure ADF,” I showed how metadata can streamline data integration with Azure Data Factory (ADF). With AI now at the forefront, securing these pipelines — data, models and all — is critical.

So, I set out to build a zero-trust architecture for MLOps in Azure, using Microsoft Entra ID, Azure Key Vault and Private Link, all orchestrated with metadata. This article walks through my approach, the challenges I hit and what I learned along the way. Figure 1 shows the architecture, laying out how it keeps AI workloads locked down.

Designing a zero-trust MLOps architecture

Vikram Garg

The zero-trust mindset

Zero-trust means trusting nothing by default — every user, service and data flow has to prove itself. For MLOps, where sensitive data and proprietary models are in play, this is non-negotiable. I built the architecture around three principles:

• Verify everything. Authenticate every access with Microsoft Entra ID.
• Keep permissions tight. Use metadata to assign only what’s needed.
• Assume the worst. Encrypt data, isolate networks and monitor relentlessly.

Metadata-driven security

My original framework used metadata in Azure SQL Database to drive ETL pipelines, and I knew I could extend it to manage security for AI workloads. I added new tables to handle the security setup:

• Access_Policies. Defines who gets what access — say, data scientists running inference or analysts viewing outputs.
• Secret_References. Points to Azure Key Vault for credentials and tokens, keeping sensitive data out of scripts.
• Network_Rules. Sets up Private Link endpoints and firewall rules for services like Databricks.
• Audit_Logs: Tracks every action for compliance and auditing.

Here’s a sample metadata entry for securing a pipeline:

{
  "job_id": 201,
  "security": {
    "access_policy": {
      "role": "DataScientist",
      "permissions": ["execute_inference", "read_model"],
      "entra_id_group": "ds_team"
    },
    "secret_reference": {
      "key_vault": "ml-vault",
      "secret_name": "databricks-token"
    },
    "network_rule": {
      "service": "Databricks",
      "private_endpoint": "pe-databricks-eastus"
    }
  }
}

This lets ADF pull security settings — like a Databricks token or access rules — directly from metadata, keeping things clean and secure.

Securing data and models

AI pipelines handle valuable data and models, so I focused on locking them down:

• Microsoft Entra ID. Authenticates users and services across ADF, Databricks and ADLS Gen2. Service principals cover automated tasks, while users are verified for manual access.
• Key Vault. Stores credentials and API keys, referenced via metadata. For example, a Databricks token is fetched at runtime, never hardcoded.
• Private Link. Routes traffic over Azure’s private network for services like Databricks and SQL Database, with metadata automating endpoint setup.

For hybrid environments, I used self-hosted integration runtimes (IRs) to connect on-premises data, with metadata picking the right IR. This keeps data secure, whether it’s in the cloud or on-site.

Protecting the MLOps lifecycle

MLOps spans data prep, model training, inference and monitoring, and each step needs its own security. Here’s how I handled it:

• Data prep. ADF pipelines pull data from sources like SQL Server, using Entra ID credentials and storing it encrypted in ADLS Gen2. Metadata controls access.
• Training. Databricks clusters are secured with Entra ID and Private Link. Metadata sets cluster configs, like auto-scaling, to keep costs and security in check.
• Inference. Databricks jobs use metadata to access models and data, with Key Vault securing tokens. Outputs are encrypted in Delta tables or Azure SQL.
• Monitoring. Azure Monitor tracks pipeline and model performance, with metadata defining alerts (e.g., model accuracy drops below 85%). Audit logs in Azure SQL ensure traceability.

I learned the hard way that skipping robust monitoring can bite you — catching issues like model drift early is a lifesaver.

A team-friendly security interface

Security shouldn’t be an IT gatekeeper. I built a web interface on Azure App Service where data engineers, data scientists and security admins can manage metadata. Engineers tweak pipeline settings, scientists update model permissions and admins handle access — all self-service. It cuts bottlenecks and keeps everyone aligned.

Navigating the challenges

Building this wasn’t all smooth. Here’s what I ran into:

• Entra ID setup. Mapping roles across services was complex. Metadata-driven RBAC saved me from manual configs, but it took some trial and error.
• Private Link complexity. Setting up endpoints for multiple services got messy. Metadata streamlined it, but I had to test connectivity rigorously.
• Hybrid environments. On-premises data access needed secure IRs. Metadata handled runtime selection, but validation was key to avoid hiccups.
• Performance hits. Encryption and authentication added latency. I optimized Databricks clusters and prioritized urgent jobs via metadata to keep things fast.

What I learned

This project showed me that zero-trust can be practical, not just a buzzword. Metadata makes security scalable — adding a new model or tightening access is just a metadata update. The web interface got teams collaborating, and tools like Private Link and Key Vault made the pipeline rock-solid. My big takeaway? Balance security with performance — overdo it, and you’ll slow things down; underdo it, and you’re exposed.

Security…and scalability

This zero-trust architecture for MLOps in Azure delivers a secure, scalable way to run AI pipelines. It builds on metadata-driven design to make security manageable and team-friendly. I’d love to hear how you’re securing AI workloads. For more, check the Azure Security documentation and Databricks security guide.

This article is published as part of the Foundry Expert Contributor Network.
Want to join?