NIST AI Risk Management Framework Playbook

MAP   

MAP 5.1

Likelihood and magnitude of each identified impact (both potentially beneficial and harmful) based on expected use, past uses of AI systems in similar contexts, public incident reports, feedback from those external to the team that developed or deployed the AI system, or other data are identified and documented.


About


AI actors can evaluate, document and triage the likelihood of AI system impacts identified in Map 5.1 Likelihood estimates may then be assessed and judged for go/no-go decisions about deploying an AI system. If an organization decides to proceed with deploying the system, the likelihood estimate can be used to assign TEVV resources appropriate for the risk level.

Suggested Actions
  • Establish assessment scales for measuring AI systems’ impact. Scales may be qualitative, such as red-amber-green (RAG), or may entail simulations or econometric approaches. Document and apply scales uniformly across the organization’s AI portfolio.
  • Apply TEVV regularly at key stages in the AI lifecycle, connected to system impacts and frequency of system updates.
  • Identify and document likelihood and magnitude of system benefits and negative impacts in relation to trustworthiness characteristics.
Transparency and Documentation


Organizations can document the following:

  • Which population(s) does the AI system impact?
  • What assessments has the entity conducted on trustworthiness characteristics for example data security and privacy impacts associated with the AI system?
  • Can the AI system be tested by independent third parties?

AI Transparency Resources:

  • Datasheets for Datasets. URL
  • GAO-21-519SP: AI Accountability Framework for Federal Agencies & Other Entities. URL
  • AI policies and initiatives, in Artificial Intelligence in Society, OECD, 2019. URL
  • Intel.gov: AI Ethics Framework for Intelligence Community - 2020. URL
  • Assessment List for Trustworthy AI (ALTAI) - The High-Level Expert Group on AI - 2019. LINK, URL
References


Emilio Gómez-González and Emilia Gómez. 2020. Artificial intelligence in medicine and healthcare. Joint Research Centre (European Commission). URL

Artificial Intelligence Incident Database. 2022. URL

Anthony M. Barrett, Dan Hendrycks, Jessica Newman and Brandie Nonnecke. “Actionable Guidance for High-Consequence AI Risk Management: Towards Standards Addressing AI Catastrophic Risks”. ArXiv abs/2206.08966 (2022) URL



Back to Top ↑