How banks test GenAI and what top banks demand

The six core asks from Banks

1. Big-bank references first. Citi checks prior clients.
2. Clear internal fit. Solve a live bank use-case.
3. Trusted vendor stack. Copilot and ChatGPT pass.
4. Strong founders. No drama. Clean history.
5. Hard model testing. Accuracy and safety splits.
6. Explainable outputs. Black boxes stay out.

The test gauntlet

1. Scope the use-case tightly.
2. Define success metrics.
3. Build scrubbed datasets.
4. Include edge cases.
5. Run “red team” prompts.
6. Probe jai.lbreak risks.
7. Score hallucination rate.
8. Track bias and drift.
9. Check tokens and cost.
10. Assess latency at peak.
11. Require human oversight.
12. Sign off with RMG.

Pilot-to-production gates

1. Policy approval first.
2. Third-party risk cleared.
3. Security review passed.
4. PII flows documented.
5. Logging and audit ready.
6. Fallback plan in place.
7. Owner named and trained.
8. Rollback tested twice.

Bank examples

SouthState Bank.

1. Two tools are allowed only.
2. Copilot and ChatGPT.
3. AI risk body in place.
4. Costed and monitored.

Citi / Citi Ventures.

1. Seeks proven vendors.
2. Focus on real fit.
3. Multi-year partner path.

Goldman Sachs.

1. GS AI Assistant live.
2. Warns on over-reliance.
3. Human review required.

JPMorgan.

1. Enterprise AI push.
2. Rese, arch, and controls.
3. Focus on hallucination risk.

Bank of America.

1. 50% staff use AI help.
2. Council vets pilots.
3. Explainability required.

What to measure

1. Task quality uplift.
2. Time-to-complete cut.
3. Error rate delta.
4. Agent containment rate.
5. Jailbreak success rate.
6. Toxicity score trend.
7. Cost per 1k tokens.
8. P95 latency in ms.
9. Adoption by role tier.
10. Audit issues count.

Control pattern

1. Use private RAG first.
2. Mask PII at source.
3. Block outbound secrets.
4. Ground with citations.
5. Force reason + source.
6. Disable code exec.
7. Rate-limit prompts.
8. Watermark outputs.
9. Keep HITL on by default.

Model risk linkage

1. Follow SR 11-7 end-to-end.
2. The board sets the risk appetite.
3. RMG defines reviews.
4. Indep. validation runs.
5. Ongoing monitoring stays.

Operating cadence

1. Monthly policy forum.
2. Quarterly model review.
3. Drift alerts in hours.
4. Access recert each Q.
5. KPI pack to ExCo.

What not to do

1. No shadow AI builds.
2. No public data leaks.
3. No unlogged prompts.
4. No unsourced claims.
5. No auto-send to clients.

CRE sales note

1. CRE sales likely to rise.
2. Pricing gaps drive deals.
3. Capital rules add push.

Leadership watch

1. Gruenberg exit closed.
2. FDIC is now under Hill.
3. Tone shifts to lighter regs.
4. Banks expect policy change.

Quick launch playbook

1. Pick one workflow.
2. Draft a risk memo.
3. Select a small cohort.
4. Build eval datasets.
5. Run AB test pilots.
6. Measure and decide.
7. Scale only with controls.

Read my book "Using AI in Banking: Real-world application, Use cases, AI technology, and Solution for Banks"

https://www.amazon.com/Using-AI-Banking-application-technology-ebook/dp/B0DHYHX5CW