Commercial AI Use

Many commercial uses of AI have been shown to disproportionately effect marginalized groups, such as black people, poor people, and people with disabilities. More broadly, the rapid proliferation of these tools has contributed to an exponential increase in data collection and sharing that the limited existing U.S. data protection laws appear unable to meaningfully address. In an effort to address the myriad and continuous impacts of commercial AI use on consumers, EPIC has consistently engaged with federal and state agencies, enforcement bodies, and lawmakers. EPIC uses multiple approaches in holding AI accountable and pushing for better protections, including filing lawsuits, such as the ongoing suit against AI abuses in housing, urging federal and state enforcement bodies to open investigations into harmful or rights-violating AI products, and working with coalitions to target AI harms in specific high-risk areas, such as financial services or AI chatbots acting as therapists or targeting children. EPIC has a long history of engaging on AI harms and privacy risks, from petitioning the Federal Trade Commission to promulgate a rulemaking “concerning the use of artificial intelligence in commerce” generally, to issuing complaints urging investigations into hiring algorithms, housing screening algorithms, and scoring algorithms. Further, EPIC has long advocated for comprehensive data protection legislation, moratoriums on particularly dangerous tools and commonsense AI regulation to protect the public.

Without requirements for transparency, notice, access, nondiscrimination, data minimization and deletion, consumers are at high risk for adverse consequences from unaccountable AI systems.

Sector-Specific Examples of Commercial AI

Employment

AI and algorithmic tools used in hiring include resume scanners, face and voice analysis of interviews, and games for applicants that measure “fit” with a company. While employed, employers regularly surveil their employees whether they are in-person (i.e. surveillance cameras, productivity trackers) or remote (i.e. keystroke trackers, use limitations). EPIC has supported bills that empower workers to know how employment decisions are made and filed a Federal Trade Commission complaint targeting HireVue’s use of opaque algorithms and facial recognition. The complaint alleged that HireVue’s AI tools—which the company claimed could measure the “cognitive ability,” “psychological traits,” “emotional intelligence,” and “social aptitudes” of job candidates—were unproven, invasive, and prone to bias.

Education

Algorithms in education are used both prior to enrollment and while an individual is enrolled. Prior to enrollment and throughout the enrollment process, many schools use AI programs that purport to evaluate applications and help “manage enrollment.” During education, AI tools may push students to certain classes and majors through systems that often use race as a major predictor, and schools may force students to use invasive proctoring and surveillance tools. EPIC has filed comments with the Department of Education noting that AI systems in education require special safeguards and urging the Department to limit funding to AI programs in education unless they meet clear safety and fitness requirements. EPIC previously filed a complaint with the Office of the Attorney General for the District of Columbia alleging that five major providers of online test proctoring services engaged in unfair and deceptive trade practices through excessive collection of students’ biometric and other personal data and routine reliance on opaque, unproven, and potentially biased AI analysis to detect alleged signs of cheating.

Healthcare

EPIC’s recent report on privacy issues in health care contains whole chapters on the risks of AI in healthcare, from chatbots giving flawed medical advice to AI systems used for everything from assessing insurance claims to suggesting and rejecting treatment options. Algorithms and automated decision-making systems have been used to prioritize care for certain functions like kidney replacement, estimate morbidity risk for patients with COVID-19, and inform administration of pain medicine, among other functions.

Housing

In housing, there is significant use of algorithms and automated decision-making systems to screen and surveil tenants, as well as significant use of biometric entry. Many of these AI screening systems are riddled with errors and biases in their reports and conclusions. EPIC is engaged in a lawsuit with RentGrow over this very issue, alleging that use of this flawed AI system constitutes an unfair and deceptive practice.

Public Benefits

Technology companies create algorithmic tools that surveil, collect and analyze data to predict the likelihood a given public benefit recipient may be committing fraud. Based on information obtained through a freedom of information request, EPIC filed a complaint with the FTC against Thomson Reuters’ Fraud Detect AI system, which is used to track and assign “risk score[s]” to public benefit recipients related to whether their public benefits claims are fraudulent. In fact, the system’s fraud predictions were frequently incorrect, resulting in hundreds of thousands of legitimate claims being rejected.

Insurance

AI is used in all facets of insurance: attempting to estimate risk, personalizing insurance rates, claim administration, and more. Insurance companies have used untested and questionable AI systems for insurance fraud prediction and to reject claims.

Criminal Legal Cycle

Algorithms used for pre-trial or at trial risk assessment, predictive policing, and gang databases are all most commonly developed by commercial firms. Additionally, companies like Amazon and Citizen offer products that surveil individuals, crowd source reports of crime, and partner with local law enforcement.