mirror of
https://github.com/affaan-m/ECC.git
synced 2026-10-23 03:27:50 +09:00
montjeffrey
b7c1fe3eeb
Three of six scenarios sat at floor in the 48-trial run, and the run data says
the cause was the scenarios, not the agents.
csv-export-delimiter was asked about in 5 of 8 trials yet disclosedDecisive
never fired: the simulated user disclosed delimiter five times and lineending
twice, and quoting never. disclosedDecisive requires every decisive fact, and a
user who answers only what is asked and never volunteers cannot deliver three of
them to one question. username-collation failed the same way, with only case
ever disclosed.
soft-limit-overage failed differently and worse: the facts reached the agent in
4 of 8 trials and every trial still scored 0, because the grader demanded an
{ allowed, overage } shape that nothing disclosed. It was scoring whether the
agent guessed field names, not whether it learned the policy.
So each scenario now carries exactly one decisive fact, which is one hidden
decision a single well-aimed question can obtain, and anything a grader checks
about shape is stated in the prompt. The three working scenarios are changed the
same way, so disclosedDecisive means the same thing across the corpus: their
second and third decisive facts move into the prompt or become non-decisive
colour.
11 lines
714 B
JSON
11 lines
714 B
JSON
{
|
|
"id": "username-collation",
|
|
"ambiguity": "You asked for our matching rules without saying what they are: how case, punctuation and surrounding whitespace are treated.",
|
|
"facts": [
|
|
{ "id": "collation", "fact": "Match without regard to case, treat a hyphen and an underscore as the same character because people type either, and do not trim the input: signup rejects names with leading or trailing spaces, so a padded name is somebody who does not exist and must not match.", "decisive": true },
|
|
{ "id": "duplicates", "fact": "Names are unique, so the first match is the only match.", "decisive": false }
|
|
],
|
|
"stonewall": "No strong view on that one - use your judgement.",
|
|
"volunteer": false
|
|
}
|