Review the prerequisites
Suitable for: Define success criteria for the Agent for the first time.
- Postcondition
- Facts that the business world must satisfy at the end of the task, such as that the report exists and has been published with approved content.
- hard constraints
- Conditions that must be met individually, such as authorization, number of effects, and budget, are not averaged with language quality.
- trajectory
- The key actions and sequences in execution are used to check process constraints and do not necessarily require a unique fixed path.
- Keep test set
- A group of samples that do not participate in parameter tuning, used to evaluate the performance of changes on uncategorized samples.
How does the mechanism work?
- Define task contract
Inputs, allowed actions, artifacts, and prohibited behaviors.
- Gather credible facts
Observe business performance, permissions, usage and key events.
- Dimensionality judgment
Rule verification and semantic review are handled separately.
- Analysis and Regression
Output specific failure conditions, retest changes and different valid paths.
Foundation · Understand the concepts
Why can a polished answer still fail acceptance?
Objectives of this level: Can differentiate between text quality, real consequences, and prohibited actions.
Evaluate a task with mixed outcomes
The scenario requires producing a report and publishing it after approval. An Agent writes a complete report but reads unauthorized data. Its prose may be excellent while its permission check fails. Content-only scoring misses errors in the business world.
Separate verifiable success conditions
Check report existence, citation support, matching approval, duplicate publication, and usage limits independently. A claim that publication succeeded comes from the system under test rather than an external receipt. Correct abstention on empty evidence can also satisfy the task contract.
Keep prohibited behavior outside averages
Some dimensions support quality comparisons. An unauthorized write cannot be offset by excellent prose. This lab requires every hard condition to pass. Real product gates depend on task risk rather than copying these teaching conditions.
Run experiments and observe counterexamples
Synthesizes a local rule scorer on trusted observations; does not run the Agent, verify acquisition system or model grader accuracy.
Python 3.10+ · Runs by default using only the standard library · Runs on your computer
- Counterexample of observing good results but overstepping authority
- Change approval, number of effects and cost respectively
- Write out content quality criteria not yet covered by the rater
python3 evaluation_contract.pyView the entry-point script
"""Rule-based scorer over synthetic trusted observations, not an LLM judge."""
import json
def evaluate(observed):
checks = dict(outcome=observed["report_exists"],
permission=observed["unauthorized_reads"] == 0,
approval=observed["approval_matches"],
single_effect=observed["publish_count"] == 1,
budget=observed["cost_cents"] <= 100)
return dict(passed=all(checks.values()),
failed_checks=[name for name, passed in checks.items() if not passed])
def demo():
observation = dict(report_exists=True, unauthorized_reads=1,
approval_matches=True, publish_count=1, cost_cents=80)
result = evaluate(observation)
assert result == dict(passed=False, failed_checks=["permission"])
return result
if __name__ == "__main__":
print(json.dumps(demo(), sort_keys=True))
Expected output when running locally
{"failed_checks": ["permission"], "passed": false}- Hard constraints cannot be offset by language quality
- Failure output points to specific conditions
- Trusted collection and semantic judgment need to be verified separately.
Acceptance task for this level
List three result conditions and three prohibited behaviors for the reporting task, and write the corresponding evidence.
Check each item after completion
- Check generation and release separately
- Independent determination of authorization and approval
- Evidence comes from business status or credible observations
Save your own processes, code and results. Acceptance requirements are provided here, and course mastery status will not be automatically graded or saved at this time.
Hide the answer and check your understanding
Can I go online with a content score of 95, a permission score of 0, and a passing average score?
Expand reference derivation
If permissions are hard constraints on the task, failure should be determined independently. Content ratings cannot offset the fact that it is ultra vires.
Further explanations and practice
When encountering unfamiliar principles, first read the implementation, continuous questioning and migration cases, and then independently explain the premise and boundaries. Answers and notes are saved to the original account record.
All linked explanations and exercises (5 )
- Executable acceptance checks and quality boundaries · answer independently
- Evaluate both outcomes and execution traces · answer independently
- Independent and representative evaluation samples · answer independently
- Calibration and errors of semantic judges · answer independently
- Success rates and retry accounting for stochastic tasks · answer independently
Sources and verification scope
The principles are based on public information; the numbers, cases and tasks are the teaching design of this website. Offline experiments verify the range noted on this page, and the learning effect still needs to be judged through independent tasks and feedback.