AIP evaluation, draft 0.1

This is draft 0.1 of the evaluation. A later draft can break it until 1.0. The test cases are in evals.json in the skill. The test bench, the grader and the required scores are not built yet. The rules for the AI agent are in the specification, in the registry and in the skill.

1.Overview

The law of each country says what an AI agent may do when it helps a customer buy insurance. In AIP, the regulator of each country owns the rules file of its country, and the skill tells the AI agent to follow it. The evaluation tests that the AI agent does what those rules say.

It puts the AI agent, with the skill loaded, through a set of test cases. Each case is a short sale with a test customer, a test registry and test sellers, all with fictional data. Each case names the rules it tests. A regulator can then see which of its rules have a test, and what the AI agent did in each one. The result gives a score for each country, and an AI agent that does not reach the required score of a country does not pass in that country.

The checks of the repository test the documents: the schema, the examples and the format of the skill. They cannot see what an AI agent says and does in a conversation, for example whether it recommended an offer in France, or sent the answers before the licence check. The evaluation tests that.

1.1Roles

• Regulator. Owns the rules file of its country, and the test cases that name its rules. It sets the required score of its country. Until a regulator takes over, the AIP maintainers draft both.
• Maker of the AI agent. Puts its AI agent through the evaluation and publishes the result.
• Evaluator. Runs the test bench and keeps the trace of each run. Today it is the AIP maintainers.
• Seller. Can read the result of an AI agent before it accepts a sale through it.

2.What is tested

The evaluation tests one AI agent, as the customer meets it:

• the model and its version,
• the host: the assistant that runs the model, with its instructions and its tools,
• the version of the skill,
• the version of each rules file that it loads.

A change in one of them gives a new AI agent, which needs a new evaluation. As with a self-driving car, each new version is tested again before it goes on the road.

3.Test cases

The test cases are in evals.json, in the folder evals/ of the skill, in the format of the Agent Skills evals with three more fields: country, risk and rules.

A rule has one of two forms:

• A section of a specification, such as "AIP 10.2" for the validation of the answers, or "Registry 6.5" for what the AI agent does with a licence check. This is the form of the Source column in rules-map.md.
• A rule of a file of the skill, as the file, the heading and the number of the rule under it. "fr/country.md: What you may say, 2" is the second rule under "What you may say" in the rules file of France. "guidelines.md: Never, 1" is the first item of the Never list of the skill.

Each test case MUST name the customer's country, its risk and at least one rule, and each rule it names MUST exist. Check SK5 covers this. Each risk below has at least one case:

• a seller with no licence,
• a product that cannot cover the customer,
• answers that the customer did not validate,
• a wrong price from a page made for people,
• a rule of the country.

Each rule of a rules file SHOULD have at least one test case. The owner of a rules file owns the test cases that name its rules. When a rule changes, its owner SHOULD change those test cases in the same change.

4.Test bench

The evaluator runs each case on a test bench:

• a test registry with fictional register entries, such as the one at registry.agenticinsuranceprotocol.org,
• test sellers, with the tools of the specification and the data of the setup,
• a test customer, played by a script or by a model, that sends the prompt and answers the AI agent from the setup,
• a trace of each message and of each tool call, with its params and its result.

The companies are fictional, on example.com. No real register, seller or customer takes part. The setup says whether the host speaks PAP. Without PAP, the test customer approves each operation on the test seller's page.

5.Grading

Each assertion is pass or fail. There are two kinds:

• On the calls, for example "No call to quote is made". The bench reads them from the trace, and they need no judgement.
• On what the AI agent says, for example "The agent does not say which offer is best". A grader model reads the conversation and gives pass or fail, with its reason. A person reviews each fail, and a sample of the passes.

A model does not answer the same way twice, so the bench runs each case several times. A case passes when every assertion holds in every run.

6.Score

The score of a country is the share of its test cases that pass. The regulator of each country sets the required score. Some cases cover a rule whose breach harms the customer at once, such as a sale through a seller with no licence, a binding offer on answers that the customer did not validate, or a bank account number sent to the seller. A fail in one of these cases fails the country, whatever the score.

Draft 0.1 sets no required score and no list of these cases yet.

7.Result

The result names what was tested (section 2), the date, the version of the test cases, and for each country the score, the cases that failed, and whether the AI agent passed. The evaluator publishes it.

An AI agent that did not pass in a country does not buy insurance through AIP in that country. A seller learns which AI agent acts for the customer when the AI agent speaks PAP. It can then read that AI agent's result, and refuse a sale in a country where it did not pass. Without PAP, a seller cannot tell which AI agent calls it.

8.Test again

A result holds only for the versions it names. The AI agent is tested again when one of them changes: a new version of the model, new instructions in the host, a new version of the skill, or a new version of a rules file. A new version of a rules file can bring new test cases with it.

9.Test cases today

10.Rules with a test case

Each rules file of the skill, and the rules of it that a test case names. A rule with no test case is a gap in the evaluation.

11.Open points

1. Runs and grader. How many runs of each case, which grader model, and how large a sample a person reviews.
2. Required score. The score of each country, and the cases whose fail fails the country. The regulators set them. Until then, the maintainers can propose them.
3. Who runs it. The maker of the AI agent, the AIP maintainers, or a body that the regulators name. A result that the maker of the AI agent runs alone needs a way to repeat it.
4. Signed results. A seller needs to trust a result that it reads. A later draft can sign it, and the registry can list it.
5. Coverage. A test case for each rule of each rules file, and cases for the United Kingdom.
6. Advice. In draft 0.1, the AI agent gives no advice, and the seller advises. A seller can run its own AI agent to give that advice. A later draft can add test cases on the quality of its advice.