AISPA Audits Hidden Instructions in 88 Commercial AI Products, Finds About 40% Contain Rules That Disadvantage Users

System prompts determine how a model describes its identity, handles personal data, discloses limitations, and prioritizes the product provider’s interests, but users and external auditors typically cannot see this layer. AISPA (Artificial Intelligence System Prompt Assurance) aims to turn system prompt auditing from scattered exposés into a comparable evaluation framework: researchers classify each instruction according to eight user-related dimensions, then assess whether it is protective, neutral, or potentially contrary to users’ interests.

The study analyzed system prompts for 88 commercial products collected in a public GitHub repository, annotating 3,249 instructions in total. According to the paper, 98.9% of products contain at least one protective instruction, but only 24% cover all eight dimensions. The average number of protective rules varies by more than tenfold across developers. About 40% of products contain at least one problematic instruction, and protective and problematic rules frequently appear in the same prompt. This shows that merely checking whether safety provisions exist is insufficient to determine how a product actually prioritizes competing interests.

The accompanying System Prompt Index makes selected prompts, company-level results, and an interactive auditing tool publicly available, allowing researchers to trace the basis for annotations. For engineering teams, AISPA can be adapted into a static analysis workflow for prompts: before a release, teams can compare areas such as identity transparency, data use, manipulation, and appeals, while incorporating prompt changes into code review and regression testing. It also serves as a reminder to teams using third-party agents or prompt templates that hidden instructions are themselves supply-chain components whose provenance and version history must be recorded.

However, this is not a ranking of the overall safety of the 88 products. The sample comes from a public repository and may be biased toward versions that were leaked, open-sourced, or easy to extract; these may not match the prompts currently deployed in production. Textual rules also do not necessarily reflect a model’s ultimate behavior. Next steps should include product providers verifying versions, researchers publishing inter-annotator agreement, and combining static prompt audits with tests of real conversations, tool calls, and data flows.