SLEEC-PATROL: Diagnosis evaluation

What this study is about

SLEEC-PATROL checks a neural network against SLEEC rules. A SLEEC rule states what a system must do in a given situation: when an event happens and a condition holds, then the system must give a response. Here the system is a neural network that drives a car: the condition is about what the network sees, and the response is an action it must, or must not, choose.

For each rule, the verifier Marabou searches every input the network can receive within the ranges the specification declares. If it finds one for which the network does not do what the rule demands, the rule is violated, and that input is its counter-example. SLEEC-PATROL then shows a diagnosis: what the rule demanded, what the network did instead, and the values behind both. This study asks whether that diagnosis helps you understand the violation.

What you will see

Two environments, one after the other:

• Highway (): a network that drives a car on a three-lane highway, choosing to change lane, keep going, speed up or slow down.
• SDD-OIA (): the deciding part of a driving model that first recognises concepts in a picture (a red light, a pedestrian, an obstacle...) and then turns their probabilities into stop, forward, left or right.

Each environment starts with a page describing it. Then every violated rule has a page of its own, showing the rule and its diagnosis as SLEEC-PATROL shows them. To make each rule easier to understand, its page also gives it in plain words.

What we ask you

On each rule page, read the rule, in plain words and as written, open its diagnosis, and answer:

1. Does the diagnosis help you to understand the violation? Yes or no.
2. If no, what could be clearer? Required when you answer no.

Your answers for an environment are submitted when you finish its last rule. You can go back to change an answer; submitting an environment again replaces its earlier answers.

Reading a rule

Each rule page first gives the rule in plain words, as a reading aid; the rule as written in SLEEC, below it, is what was verified.

A rule is shown as it is written in SLEEC. Words in blue are measures, what the network reads; words in purple are output events, the responses the rule demands; words in brown are constants. For example, SDD-OIA's rule r13 reads when a red light is seen, then stop:

A rule may carry exceptions, written with unless. Before it is verified, such a rule is split into parts that are checked one by one: the main case, and one part for every exception that states a response of its own. A rule written as

r when A then B unless C then D

is checked as

r_main      when A and (not C)   then B
r_unless1   when A and C         then D

Each part of the rule is tinted by the verdict of its own check, and a rule is violated as soon as one of its parts is. The verdicts are:

• VIOLATED the network can break this part of the rule, and the counter-example that breaks it is shown.
• COMPLIANT nothing in the declared ranges breaks this part of the rule.
• UNVERIFIED the verification did not finish (e.g. a timeout), so this part is neither proved nor broken.

Reading a diagnosis

Every violated part carries a Diagnosis button. The diagnosis shows the following; the numbers match the example below.

1. the rule again, with the part it is about coloured;
2. for Highway, a picture of the situation the verifier found: the ego in dark blue, the cars the rule names in yellow, the other cars the network sees in white;
3. the output the rule expected, and the output the network gave instead;
4. the relevant values: the measures this part of the rule names (blue), the response it demanded (purple) and, in red, the action the network took instead;
5. under show the full counter-example, every value the network was given and every score it answered with;
6. under SLEEC rule measure constraints, the values of the constants the rule names.