AI Testing: NIST’s Dioptra as a Step Forward, and Other NIST Guidance

As part of NIST’s recent mandate to formalize AI Testing set forth in President Joe Biden’s Executive Order on AI, NIST recently released a testbed called Dioptra that can be utilized to conduct evaluations to assess AI developers’ claims about their systems’ performance. Dioptra helps users identify attacks that would reduce model performance, and quantify the failures that potentially result. Dioptra’s capabilities align with the core principles of rigorous AI testing, emphasizing the necessity of validating AI systems against risks such as reliability, security, and fairness.

Continue reading