Analyzing the Risks of Generative AI: MIT Sloan’s Framework for a New Tech Era
Generative AI is a double-edged sword, offering unprecedented opportunities for efficiency and innovation while simultaneously creating a new, complex landscape of business risks. Moving from mere experimentation to full-scale deployment requires organizational leaders to develop a sophisticated understanding of these risks and implement robust mitigation strategies. Recent research from the MIT Center for Information Systems Research (CISR), based on interviews with over 60 data and technology executives, provides a critical framework for identifying, categorizing, and managing these emerging threats.
The MIT CISR research introduces a vital distinction between two fundamental types of generative AI risk: embedded and enacted. This classification is essential because each type requires a distinctly different management approach.
Embedded risks are inherent to the generative AI technology itself. They are “built-in” to the foundation models an organization chooses to adopt. These risks are heavily influenced by the quality of the training data used, the inherent (and often opaque) behavior of the model, and performance changes that might occur from vendor updates. Because embedded risks are often beyond an organization’s direct control, mitigation relies more on vendor transparency and proactive, independent evaluations. For example, if a model is trained on biased data, the embedded risk is that it will generate biased output.
Enacted risks, by contrast, arise from the specific choices an organization makes regarding how it deploys, configures, and utilizes the generative AI. This encompasses everything from how system prompts are designed, to the level of access granted to internal systems, to the safeguards put in place against adversarial attacks. Organizations have a high degree of control over enacted risks, but managing them requires diligent internal governance and strategic oversight.
To illustrate how these risks can manifest, consider a common use case: a hiring manager using an LLM to draft a job description. The MIT researchers identify several key components within this process that can introduce significant problems.
-
Training Data: Foundation models are trained on massive datasets from the web, which may contain outdated information, biased language, or inaccurate details. A model might generate a description reflecting obsolete HR practices or an industry’s pre-existing biases, failing to meet modern norms.
-
Foundation Models: LLMs can be inconsistent, generating varying responses to the same input, and may “hallucinate,” producing incorrect information that sounds plausible. The lack of transparency into how the model reached a conclusion makes error diagnosis difficult.
-
User Prompts: The quality of an AI’s response depends heavily on the input. Without clear instructions, the output may not meet expectations. Furthermore, users could inadvertently create significant data privacy risks by including sensitive company data or PII in their prompts.
-
System Prompts: These “hidden” instructions govern organizational context and enforce guardrails. Poorly engineered system prompts can create vulnerabilities or, conversely, overly rigid rules that result in boilerplate, unappealing descriptions.
As organizations advance to more sophisticated deployments, the risk space expands. Techniques like Retrieval-Augmented Generation (RAG), which ground LLMs with proprietary data, can inherit data quality issues or expose sensitive information that was previously difficult to find. Furthermore, the use of AI agents that can act autonomously introduces risks of reduced visibility, difficulty tracking data flow, and “autonomy creep” where agents perform tasks beyond their original authorization without proper human oversight.
To successfully manage this complex risk landscape, MIT CISR recommends a three-part action plan:
-
Map Exposure: Conduct an exhaustive inventory of all generative AI solutions and tools in use, documenting the underlying foundation model, system prompt design, connected data assets, and necessary human intervention points.
-
Adopt Differentiated Approaches: Manage embedded risks by requiring contractual transparency and notifications from vendors, and enacted risks by establishing internal governance capabilities and technical controls.
-
Assign Ownership and Audit Trails: Establish clear ownership for ongoing risk assessments and create detailed audit trails that log prompts, outputs, and human interventions. Given the high stakes of generative AI, its risk space can no longer remain a black box. Understanding and mitigating these risks is crucial for succeeding in this new era.
SEO and Metadata
-
Yoast Keyword: Generative AI risks
-
Meta Description: Navigating the generative AI risk landscape: MIT Sloan’s framework for managing inherent and enacted risks.
-
Tags: Artificial Intelligence, Generative AI, Business Risk, Data Governance, MIT Sloan, Technology Management, AI Security, Risk Mitigation, AI Strategy
