How to Self-Audit Your ATS for Disparate Impact Risk
To self-audit your ATS for disparate impact, pull applicant data for every decision stage, calculate each group's selection rate, compare it with the highest group's rate using the four-fifths rule, test whether gaps are statistically significant, and trace any flagged gap back to the filter or score that caused it. Then document everything.
This guide is for HR, talent acquisition and people analytics leaders who use resume screens, scoring or assessments inside an applicant tracking system and want to find problems before a complaint, an audit or a plaintiff's lawyer does.
Key takeaways
- The employer, not the ATS vendor, is responsible for whether its selection procedures have an adverse impact.
- The four-fifths rule is a screening test. Pair it with a significance test and watch for small samples.
- Audit the overall process for each job and every automated gate inside it: screens, scores, assessments, scheduling and background rules.
- Vendors should be able to explain what each gate scores and share bias audit results. If they can't, write that down.
- Federal enforcement priorities have shifted, but Title VII, state laws and private lawsuits still apply.
Why does the employer carry the risk, not the vendor?
Many HR teams assume that if bias lives inside a vendor's model, the vendor owns the problem. That's not how the rules are written. The EEOC's guidance on employment tests and selection procedures says vendor documentation can help, but "the employer is still responsible for ensuring that its tests are valid" under the Uniform Guidelines on Employee Selection Procedures. The EEOC's long-standing questions and answers on those guidelines say the same thing about procedures run by an agency or consultant: the employer stays accountable.
A scenario we see often: a company turns on a knock-out question and an AI match score in its ATS during a busy hiring season. Nobody looks at outcomes by group because "the vendor handles that." Two years later, a rejected applicant files a charge, and the company can't explain how the score worked or show that anyone ever checked it.
The ground has shifted in the past two years, and it's worth being clear about how. In 2023 the EEOC published technical assistance on assessing adverse impact in software and AI used for selection. That document was taken down from the agency's website in early 2025 while it was reviewed, as Seyfarth reported. In April 2025, Executive Order 14281 directed federal agencies to deprioritize enforcement of disparate-impact liability, naming Title VII's provision.
What did not change: the text of Title VII, the Uniform Guidelines in the Code of Federal Regulations, private lawsuits and state and local laws. New York City's Local Law 144 bars employers from using an automated employment decision tool unless it has had a bias audit within the past year, requires a public summary of results and requires notice to candidates. The city began enforcing it on July 5, 2023. Other states have their own rules; see our overview of state AI employment laws in Colorado, Illinois and NYC. Lower federal enforcement priority is not the same as lower risk.
What does disparate impact mean inside an ATS?
Disparate impact is not intentional discrimination. It's a neutral-looking practice that screens out a protected group at a substantially higher rate and isn't justified by the job. Every automated gate in your ATS is a selection procedure for this purpose.
The standard screening test is the four-fifths rule in the Uniform Guidelines (29 CFR 1607.4(D)). A selection rate for a race, sex or ethnic group that is less than four-fifths (80%) of the rate for the group with the highest rate is generally treated as evidence of adverse impact.
Here's the worked example from the EEOC's Q&A on the guidelines: 80 White applicants, 48 hired, is a 60% selection rate. 40 Black applicants, 12 hired, is a 30% rate. The impact ratio is 30 / 60 = 50%, well below 80%, so adverse impact is indicated.
Two caveats matter. First, smaller differences can still count when they're significant in both statistical and practical terms, especially with large numbers of selections. Second, large differences built on small numbers may not show adverse impact if they aren't statistically significant. The Q&A gives an example where moving a single hire would reverse the result; in that case the agencies generally wouldn't require validity evidence, though a lower rate that persists over time can still form a pattern.
One more practical note. The Q&A says the guidelines compare rates by sex and by race or ethnic group, not by intersectional subgroups such as Black women, though records for subgroups must still be kept. Many analysts look at subgroups anyway as a diagnostic, because a problem can hide inside a combined number.
Where in the ATS funnel does disparate impact show up?
Adverse impact doesn't only happen at the final hire. It can appear at any gate. These are the ones worth checking first:
- Knock-out questions and keyword screens. Filters on specific degrees, school names, job titles, years of continuous experience or employment gaps can screen out groups at different rates, and often aren't needed to do the job. For practical guardrails, see AI resume screening in your ATS.
- Automated scoring and ranking. A model trained on past hiring decisions can learn the patterns in those decisions, including any that tracked demographics. Proxy variables such as ZIP code or graduation year are the usual suspects.
- Assessments and tests. Cognitive tests, situational judgment tests and personality inventories run through the ATS are classic selection procedures. If one shows adverse impact, the guidelines expect validity evidence.
- Interview scheduling. If candidates who can't pick a slot in a narrow window are dropped, the scheduling step is screening people out. Hourly workers, caregivers and people relying on public transit are more likely to be affected.
- Video and voice analysis. Tools that analyze faces or speech deserve extra scrutiny. The 2018 Gender Shades study found commercial gender classification systems misclassified darker-skinned women at error rates up to 34.7%, against a maximum of 0.8% for lighter-skinned men. That study wasn't about hiring tools, but it shows how unevenly face analysis can perform.
- Background check rules. The EEOC's 2012 guidance on arrest and conviction records says an automatic, across-the-board exclusion from all employment is inconsistent with the factors it sets out, and encourages targeted screens and individualized assessment. Auto-adjudication rules in your ATS should follow that logic.
How do you run the self-audit, step by step?
This can be done by an HR analytics lead or a people operations manager with ATS reporting access. You don't need a statistician for the first pass, but you do need careful data handling and good notes. Involve employment counsel early so the work can be structured under privilege if your counsel recommends it.
Step 1: Pull 12 months of applicant data
Export, at minimum: applicant ID, application date, requisition and job, voluntary self-identified race or ethnicity and sex, and a yes/no field for each decision gate showing whether the applicant moved past it. If your ATS can't produce stage-level outcomes, that is a finding in itself.
Group jobs sensibly. The Q&A allows a test used across several jobs to be assessed in the aggregate, but impact in job assignments still has to be assessed job by job.
Note your self-identification response rate. If many applicants declined to answer, say so in your report and treat conclusions with care.
Step 2: Calculate selection rates at each gate
Selection rate = applicants in group X who passed the gate / applicants in group X who reached the gate.
Example: 200 Hispanic applicants reach the keyword screen and 110 pass, so their rate is 55%. Repeat for every group at every gate. Many practitioners treat around 30 applicants per group at a gate as a rough minimum before reading much into a rate; flag smaller groups instead of ignoring them.
Step 3: Apply the four-fifths rule
Find the group with the highest rate at each gate, then calculate impact ratio = group rate / highest group rate. Flag anything below 0.80. Record it in a simple table:
| Gate | Group | Selection rate | Highest rate | Impact ratio | Flag |
|---|---|---|---|---|---|
| Keyword screen | Hispanic | 55% | 68% | 0.81 | No |
| Assessment | Hispanic | 46% | 63% | 0.73 | Yes |
| Overall process | Hispanic | 12% | 19% | 0.63 | Yes |
The figures above are illustrative. Notice how two moderate drops compound into a larger gap overall. Under the guidelines, adverse impact is first determined for the overall process for each job; if there is impact overall, you analyze the components to find the source. Checking every gate anyway is cheap and tells you where to look.
Step 4: Test for statistical significance
The four-fifths rule measures practical significance. A significance test tells you whether the gap is likely to be chance.
- Chi-square test. Build a 2x2 table: group X passed, group X failed, comparison group passed, comparison group failed. In Excel,
=CHISQ.TEST(actual_range, expected_range), with expected counts calculated as row total x column total / grand total. A p-value below 0.05 is the usual threshold. - Fisher's exact test. Use it when any cell has an expected count below 5. Most statistics packages and many free calculators run it.
Record the test, the statistic and the p-value for each flagged result. If a gap is large but not significant because the sample is small, keep collecting data rather than closing the item.
Step 5: Trace flagged gates to the decision criteria
For each gate that is flagged and significant, ask the vendor in writing:
- What attributes are scored or filtered at this gate, and how are they weighted?
- What data was the model trained or calibrated on, and what was its demographic makeup?
- Has the vendor run its own bias audit of this feature? Ask for the method and results.
- Can you export the score components for your applicants so you can see which inputs drive the gap?
If a vendor can't or won't answer, document the refusal. It matters for your next decision and for any later investigation.
Step 6: Document findings and actions
Write a report covering the data source and extraction method, sample sizes and response rates, calculation methods, all impact ratios and significance tests, and your interpretation. For each flagged item, record the corrective action or the reason for continued use, with the date and the approver.
What should you do when you find adverse impact?
You have three broad options:
- Stop using the practice. Remove the filter, score or feature. This is usually the lowest-risk path, though it may mean reconfiguring workflows or revisiting the contract.
- Change it. Drop a knock-out question, reweight a score, or add human review for borderline candidates. Document the change and rerun the analysis after enough new applicants have gone through.
- Validate it. If the procedure genuinely predicts job performance, the guidelines recognize criterion-related, content and construct validity evidence. The EEOC's guidance also points to the question of less discriminatory alternatives: if another procedure would predict performance about as well with less adverse impact, you should consider adopting it. Validation is specialist work, so bring in an industrial-organizational psychologist.
Validation done after a charge is filed is harder to defend than validation done before you use the tool. Hiding names and other identifying details during early review is one change many teams try first.
How often should you audit, and what records should you keep?
The Uniform Guidelines' recordkeeping section (29 CFR 1607.15) calls for adverse impact determinations at least annually for each group that makes up at least 2% of the relevant labor force. If the overall process has impact, keep records showing which components cause it. If impact is eliminated, keep component records for the period it existed and keep collecting them for at least two years afterward. Employers with 100 or fewer employees can use a simplified recordkeeping option.
Annual is the floor. Re-run the analysis after any material change: a new vendor, a new scoring model, a model update pushed by the vendor, or a new knock-out question. Tie the audit to your EEO-1 reporting cycle, since much of the data overlaps.
Separately, EEOC regulations require employers to keep application and hiring records for one year from the record or personnel action, whichever is later, and to keep relevant records until final disposition if a charge is filed (29 CFR 1602.14). State rules can be longer; our guide to multistate HR records retention covers how to handle conflicts.
Five questions to ask your ATS vendor
- Has this feature had an independent bias audit, and can we see the method and results? In New York City it's required for covered tools. Elsewhere, the answer still tells you a lot.
- What data trained the scoring model? A model built on a homogeneous workforce's history can reproduce that history.
- Can we export individual scores and the inputs behind them? If not, you can't audit the gate.
- How will you tell us when a model update changes scoring? Updates can shift outcomes without anyone on your side changing a setting.
- What will you commit to in the contract? Ask for representations on validity, audit support and data access. If the vendor offers none, factor that risk into your decision.
For a wider team process covering these questions, see our EEO compliance playbook for AI recruiting tools.
Self-audit checklist
- Data: 12 months of applicants, stage outcomes and self-ID, with response rates noted.
- Rates: selection rate per group, per gate and overall, per job or job group.
- Ratios: four-fifths comparison with flags.
- Significance: chi-square or Fisher's exact for each flag, with small-sample notes.
- Root cause: vendor answers on inputs, weights, training data and audits.
- Action: stop, change or validate, with dates and owners.
- Records: report, data extract, calculations, vendor correspondence and approvals, stored as compliance records.
- Schedule: next annual audit, plus triggers for out-of-cycle reviews.
FAQ
Is the four-fifths rule a legal safe harbor?
No. It's a rule of thumb from the Uniform Guidelines that agencies use to decide when adverse impact is generally indicated. Smaller differences can still be adverse impact when they're statistically and practically significant, and larger ones may not count when they rest on very small numbers. Treat it as a screen that tells you where to look harder.
Do we still need to audit if the EEOC has deprioritized disparate impact?
Yes, in most cases. Executive Order 14281 changed federal agency enforcement priorities, but it didn't amend Title VII, and it creates no rights for any party. Private plaintiffs can still bring disparate impact claims, and state and local laws such as New York City's Local Law 144 still apply. An audit also simply tells you whether your hiring tools work fairly.
Who should run the audit, HR or legal?
Usually both. HR or people analytics has the data and the ATS knowledge, and can run the calculations. Employment counsel should help scope the work, decide how findings are documented and advise on corrective action. For validation studies, bring in an industrial-organizational psychologist. Independent auditors are required for some tools in some jurisdictions, such as NYC.
What to do next
- Map every automated gate in your ATS, including knock-out questions and scheduling rules, and note who configured each one.
- Run steps 1 to 4 on your highest-volume job family this quarter.
- Send the five vendor questions in writing before your next renewal.
- Put the annual audit, and the triggers for an extra one, on your compliance calendar. If your team builds its own screening tools, also read the compliance gap when HR-built AI agents make decisions.
Sources
- 29 CFR 1607.4, Information on impact (four-fifths rule), Legal Information Institute
- 29 CFR 1607.15, Documentation of impact and validity evidence, Legal Information Institute
- 29 CFR 1602.14, Preservation of records made or kept, Legal Information Institute
- Questions and Answers on the Uniform Guidelines on Employee Selection Procedures, U.S. EEOC
- Employment Tests and Selection Procedures, U.S. EEOC
- Enforcement Guidance on the Consideration of Arrest and Conviction Records (2012), U.S. EEOC
- Executive Order 14281, Restoring Equality of Opportunity and Meritocracy, The White House
- Automated Employment Decision Tools (Local Law 144), NYC Department of Consumer and Worker Protection
- Gender Shades (2018), Proceedings of Machine Learning Research
- EEOC's changing AI policies, Seyfarth Shaw
This article is general information, not legal advice. Talk to employment counsel about your situation.
Last reviewed: October 2026.

Comments
Post a Comment