EPIC Joins Coalition Comment on NIST Guidance on AI Benchmark Evaluation

EPIC joined 17 other civil society organizations to submit a comment on March 31 in response to the National Institute of Standards and Technology’s draft guidance on Practices for Automated Benchmark Evaluations of Language Models. 

The comment, led by the National Fair Housing Alliance, commends NIST and the Center for AI Safety and Innovation’s creation of consistent AI evaluation standards buturged NIST to further strengthen the guidance.  

Our recommendations center on two priorities: incorporating civil rights principles as explicit requirements for this framework and encouraging the development of domain-specific benchmarks with particular attention to areas such as housing, lending, and criminal justice. 

We are primarily concerned that the current draft guidance does not provide sufficient guidance on how the framework could be used to test the civil rights impacts of AI systems, in particular discrimination in treatment, discrimination in outcome (disparate impact), and broader harms to protected classes.  

The coalition comment notes that high-stakes and civil-rights-sensitive fields such as housing, lending, criminal justice, employment, healthcare, child welfare, and education deserve domain-specific benchmarks. “To build AI systems worthy of public trust in high-stakes domains, benchmarks must capture domain-specific concerns,” the comment reads. 

EPIC has long advocated for centering civil rights, including privacy, in the development and use of AI.