Question 40 of 45
How do we govern our AI supply chain and manage upstream model dependencies?
Published by AI Governance Institute · Practical Governance for Enterprise AI
A governance framework for managing the risks introduced by upstream AI dependencies — foundation models, third-party datasets, AI-enabled development tools, and compute infrastructure — as components of the organization's AI supply chain.
If you only do 3 things, do this:
- 1.Your AI system's risk profile includes your suppliers' risk profiles. A foundation model provider's safety failure, a training dataset's license violation, or a compromised AI tool in your development pipeline becomes your problem. Map the supply chain before you assess the risk.
- 2.The hardest supply chain risk to manage is the one you do not know about: shadow AI tools used in development, undisclosed training data in vendor models, and AI features embedded in software you did not purchase as an AI product.
- 3.AI supply chain governance is distinct from software supply chain security (SBOM, dependency scanning) but should be integrated with it. The AI-specific layer adds model provenance, training data lineage, and capability change monitoring to the existing software supply chain framework.
The Situation
Who this is for: Security, Procurement, and AI Governance teams at organizations with production AI systems that depend on third-party models, datasets, or AI-enabled tooling
When you need this: When building an AI procurement policy, when responding to a supply chain security incident, or when a regulator asks about the provenance of AI components in production systems
The Decision
What are our material AI supply chain dependencies, what risks do they introduce, and what controls apply to each?
The Steps
- 1Map the AI supply chain: for each production AI system, trace the upstream dependencies (foundation model, training data, fine-tuning data, inference infrastructure, AI tools used in development)
- 2Classify each dependency by risk tier: Critical (failure directly affects production AI output), Significant (failure could affect production within 30 days), Limited (indirect or easily substitutable)
- 3For Critical and Significant dependencies, assess: concentration risk (single provider), integrity controls (can tampering be detected), change management (how are updates communicated), and security posture (what is the provider's vulnerability disclosure process)
- 4Integrate AI supply chain review into your procurement process for new AI capabilities
- 5Define incident response procedures specific to supply chain events: what do you do if a foundation model provider is compromised, if a training dataset is found to contain sensitive data, or if an AI tool in your development pipeline is backdoored
- 6Review and update the supply chain map annually and after any material change to production AI systems
The Artifacts
- —AI supply chain map (system → upstream dependencies → risk tier → controls)
- —Supply chain risk assessment for Critical and Significant dependencies
- —AI procurement checklist with supply chain review requirements
- —Supply chain incident response runbook
The Output
A documented AI supply chain map for every production system, with risk assessments for Critical and Significant dependencies and a defined incident response process for supply chain events.
What is in your AI supply chain
The AI supply chain extends further upstream than most teams initially realize. For a typical production AI system, the supply chain includes: the foundation model (with its training data, safety fine-tuning, and ongoing updates); any fine-tuning or retrieval datasets added by the organization; the inference infrastructure (cloud compute, model serving layer); the AI APIs or SDKs used to invoke the model; and the AI-enabled development tools used to build and test the system — including coding assistants, testing tools, and evaluation frameworks.
Each layer of this chain introduces distinct risk types. Foundation model risks include training data contamination, undisclosed capability changes, and safety regression in model updates. Dataset risks include license violations in training data, personal data ingested without consent, and adversarially injected data in retrieval stores. Infrastructure risks include standard cloud supply chain concerns amplified by the sensitivity of model weights and inference inputs. Development tool risks include the possibility that AI coding assistants have introduced vulnerable or license-encumbered code.
Mapping the supply chain before assessing it is essential. Many organizations discover during the mapping process that they have AI dependencies they were unaware of: vendor software that has silently added AI features, AI tools used by individual developers that touch production code, and foundation model providers with their own upstream dependencies that are not publicly documented.
Concentration risk and substitutability
Concentration risk — the degree to which production AI systems depend on a single provider, model, or dataset — is one of the most consequential supply chain risk factors and one of the least frequently assessed. An organization whose three most business-critical AI systems all depend on the same foundation model provider, with no tested fallback, has a concentration risk that could affect business continuity if that provider experiences an outage, regulatory action, or safety incident.
Assess substitutability for each Critical dependency: if this dependency failed or became unavailable tomorrow, what would the organization do? Could it switch to an alternative provider within 24 hours, 7 days, 30 days, or not at all? For dependencies with low substitutability and high concentration, consider maintaining a secondary option in a tested state, even if it is not in active use.
Document concentration risk explicitly in the supply chain map. Present it to the AI governance committee as a standing risk item. Organizations with high concentration in a single provider should disclose this as a material dependency in governance reporting.
Integrating with software supply chain security
AI supply chain governance should be integrated with, not parallel to, the organization's existing software supply chain security program. The software supply chain security program already covers dependency management, SBOM generation, vulnerability scanning, and third-party code review. AI supply chain governance adds model provenance, training data lineage, and capability change monitoring to this existing framework.
The integration point is procurement: every new AI component should go through the same supply chain review as other software dependencies, with an AI-specific layer added. Model cards and training data documentation should be treated as equivalents to SBOMs. Model weight integrity verification should be integrated into artifact management alongside binary signing and checksum verification.
Where AI supply chain risks are not covered by existing software supply chain processes — particularly around training data provenance and capability change monitoring — build AI-specific processes and document them as extensions to the existing framework, not as separate programs. Consolidation reduces organizational complexity and improves the likelihood that supply chain controls are consistently applied.
Governance Controls
Operational controls that implement the guidance in this playbook.
Related frameworks
Not sure where to start? Answer 3 questions and get a tailored compliance action plan.
What applies to me? →