From target to countersigned evidence.
Countersign is software that attacks an AI assistant under written authorization, measures how often each attack works, checks the assistant's answers against the institution's own documents, and records everything needed to reproduce and retest. This page describes the planned product; the console prototype runs it on sample data.
Inputs, engine, evidence.
What each suite actually tries.
Pick a suite to see a representative attack, the control it tests and where the result maps. Examples are illustrative and use a fictional institution.
Two tiers, so a finding can be defended.
Judging whether a bot's answer is "unfair or deceptive" is hard to do reliably with a model. So the plan separates what can be proven from what needs a person.
Contradicts your own documents
The institution uploads its fee schedule, Reg E error-resolution timelines, disclosures and product terms. A finding is a contradiction between what the bot said and the source document, with the clause cited. These are verifiable and go straight into the evidence pack.
Needs a person's judgment
Possible UDAAP-risky framing, dispute deflection, collections-language risk. These are labeled indicators, reviewed by a person before they reach the evidence pack, and the judge's precision and recall are published per suite. The plan's kill metric: if precision on design-partner data is below about 85%, Tier 2 stays out of the pack.
Mapped where it applies. Labeled where it does not.
| Framework | How Countersign uses it |
|---|---|
| NIST AI RMF + Generative AI Profile | Primary mapping for measurement and management of AI risks |
| FFIEC IT Examination Handbook | Security testing and third-party oversight sections |
| NYDFS Part 500 | Vulnerability management and third-party provider controls; NYDFS's 2024 AI letter expects vendor AI to be covered (Paul Hastings) |
| OWASP LLM Top 10 (2025), MITRE ATLAS | Attack taxonomy for every security finding |
| ISO/IEC 42001, EU AI Act | Management-system and robustness evidence; EU Annex III high-risk date reported as December 2, 2027 under the provisional omnibus agreement (Hogan Lovells; final text to be confirmed) |
| Reg E, Reg B, UDAAP, GLBA | Conduct suites and policy-as-code checks |
| SR 26-2 / OCC Bulletin 2026-13 | By analogy only The April 2026 guidance excludes generative and agentic AI (Orrick), so it is shown as principles applied by analogy, out of formal scope |
Mappings indicate where testing evidence is relevant. They are not a compliance determination, certification or legal advice.
What gets built, in what order.
| Phase | Timing | Deliverables |
|---|---|---|
| 0 · Prototype | Q1–Q2 2027 | Console on synthetic data; measured cost per run and per true finding; Tier 1 checks on a sample fee schedule |
| 1 · MVP | Q2–Q4 2027 | Black-box web-chat harness on one vendor partner's sandbox; Tier 1 checks; statistical reporting; evidence pack; self-hosted runner; SOC 2 observation from day one |
| 2 · Bank-grade | 2028 | SOC 2 Type II; Tier 2 with published precision and recall; vendor CI integration; third-party finding ingestion; dry-run tool-call proxy |
| 3 · Standards and benchmarks | 2028–2029 | Evidence format usable by AIUC-1 and HITRUST assessors; peer benchmarks from derived metrics; GRC/TPRM integrations; voice |
| 4 · Control surface | 2029–2030 | Continuous AI inventory and posture for regulated firms; adjacent-industry policy packs |
What Countersign is not
It does not certify compliance, give legal advice or sell a runtime firewall. It does not compete with platform red-team tools on attack breadth; it consumes their output.
Data handling by design
Exfiltration tests capture whatever a bot leaks, so the design assumes nonpublic personal information: self-hosted runner, auto-redaction, data-processing agreements, and only derived, non-reversible metrics leave the customer's environment.
Try the console.
Register a sample target, run a campaign, and open the evidence report. Fictional data, real interactions.
Open the console prototype