Independent and representative evaluation samples

LEARN · PRACTICE · REFLECT

Knowledge learning and personal records

First read along the principles, Q&A and migration cases. When you need to check your understanding, switch to reinforcement exercises or start personal recording.

Answers and personal notes

my answer0 / 20,000

Each modified commit will be kept as an independent history. Your level of mastery is up to you to evaluate yourself against the standards.

Core concept · Independent and representative evaluation samples

Understand the core principles first

Preparatory concepts:Training and testing division, sampling, Mission initial state

The evaluation set is a limited set of observations of the target task distribution. Sample similarity, selection process, and environmental residues determine how far a score can be generalized, and increasing numbers cannot automatically fill coverage gaps.

Selecting demonstrations differs from sampling

Demos often choose smooth inputs. Evaluation also needs ambiguous requests, unanswerable questions, and actual failures. Stratify by risk, difficulty, and frequency, defining tasks, initial states, and acceptance criteria. Report high-risk tails separately rather than letting common easy cases hide them.

Split by the source of shared information

Reworded tickets and questions from one document share information. Putting them in different splits leaks test knowledge into tuning. Group by original task family, document, or time period before splitting. Repeated prompt tuning on held-out questions adapts to that set even without model training.

Version samples with their environment

Question text alone cannot reproduce a result. Save initial database state, tool versions, permissions, and reference evidence. Reset side effects between attempts so prior files cannot help subsequent runs. Redact production failures, establish reproducible conditions, and mark disputed labels. This dataset process is instructional guidance.

Check understanding with a question

How to build a credible Agent review set when there are only 30 good-looking demos?

Stratify real tasks and known failures across common, long-tail, unanswerable, denied, and tool-failure cases. Each fixture specifies input, initial state, permitted actions, acceptance, and evidence. Separate tuning from held-out evaluation. Redact and authorize production samples. Version labels and disputes instead of changing expected answers merely to pass a release.

Implementation and trade-offs

First define what business it represents

List user task types, risks, complexity and actual proportions, and draw samples respectively. High-frequency simple problems reflect the overall experience, while high-risk tasks such as low-frequency unauthorized writes require independent thresholds and cannot be diluted by the overall average. Preserve trigger conditions and environments when collecting failure samples, not just end-user issues.

Each question is an executable task package

Log user input, data snapshots, tool availability status, permissions, expected artifacts, and prohibited actions. Open questions can define multiple valid outcomes or constraint sets, and it is not mandatory to refer to a certain sentence for the answer. Judgment instructions are given to the annotators, and controversial questions are reviewed before being used for release thresholds; revise acceptance criteria for cases that cannot be scored consistently.

Isolate debugging and measurement

The development set is used to repeatedly tune Prompt, while the test set is reserved to limit exposure and be managed by version. Approximate rewrites and the same business events should be divided into groups to avoid half of the same event entering development and half entering testing. Only when the task content, standard answers, initial state of the tool and scorer are saved can it be explained whether the score change comes from the model or the test environment.

Keep expanding coverage rather than chasing scores

Online failures are redacted and verified before being added to the regression set, while retaining a fixed baseline to observe long-term trends. The success rates, risk violations, costs and uncertainty ranges by task type are given. 30 demos can start problem discovery, but are not enough to prove the performance of complex production task distribution; the sample size should increase with business differences and risks.

Engineering deduction

scene: Interview assumptions: The team will present 30 successful demos and prepare to connect to enterprise writing tools.
design decisions: Add real task distribution, fault and permission scenarios, and establish retention sets.
Verify target: The release basis covers tasks and risks and is not determined by the presentation's look and feel.
applicable boundary: The sample size needs to be designed according to confidence requirements and risks, and a fixed number of questions should not be regarded as quality assurance.

Continuous questions and answers

Continue reading along with the premises and constraints of the problem. Understand the reference answers first, then try to put away the answers and explain the cause and effect and trade-offs in your own words.

Level 1Can the same problem be assigned to the test set if it is rephrased?
Question bank partitioning requires checking whether answer information is still shared across different texts.
Reference answer
Usually cannot be taken apart randomly. Multiple expressions of the same semantic task should be classified into the same family, and the whole should be assigned to a debugging set or a test set; if the robustness of the rewriting is specifically tested, the expressions can be compared under independent protocols, but it cannot be claimed that this is a new task generalization. Shared documents, entities and conversation sources need to be checked at the same time.
Follow this answer further
1. Level 2After grouping by document, the test questions still use public framework documents. Is this considered a leak?
  Parent questions prevent task reuse and continue to distinguish between shared background knowledge and leakage of test answers.
  Reference answer
  Disclosing background knowledge does not automatically equal leakage. The key is whether the debugging process uses the test tasks or their answers to modify the system. It can be stated that what is being tested is the retrieval ability of known documents, and then use a task family that is not involved in debugging to test the changes; do not package public knowledge questions into the generalization of completely unfamiliar knowledge.
  Follow this answer further
  1. Level 3The team has already looked at the reserved question and changed the prompt accordingly. How to restore trustworthy comparison?
    The father asks to determine what kind of information caused the leak, and the child asks to deal with the remediation after the exposure has occurred.
    Reference answer
    Reduce this batch of questions to regression sets and retain their anti-regression value; create another test group that is not used for debugging, freeze the evaluation protocol, and then evaluate. Record the exposure time and modification history of old questions. You cannot rename the same problem to a retained set by renaming it, nor do you need to delete valuable old cases.
Level 1How to mark multiple valid answers?
Real tasks do not necessarily have unique outputs, and objects need to be clearly labeled.
Reference answer
Mark required facts, allowed actions, prohibited states, and several legal solutions, and record expert disagreements. The reference answers are examples to help determine, not word matching templates. If two annotators still have different conclusions about the same constraint, the task should be clarified or unknowns should be allowed, and there is no need to vote to cover up the ambiguity of the question.
Level 1What's the problem if it fails online and is directly stored in the database?
When using production data to compensate for failure coverage, first ensure legality and reproducibility.
Reference answer
The original failure may contain identity data, keys, or third-party private bodies, or it may rely on changed service state. First confirm the use permission and minimum redaction, and save bounded environmental test doubles, versions and expected facts; records that cannot be reproduced are retained as investigation clues and cannot be directly turned into definite truth values.

Draw inferences from one example: If the conditions change, how to deduce it?

First find out the conditions for change, and then determine which premises in the original plan still hold true. The following cases are teaching deductions to facilitate the transfer of principles to new problems.

New user tasks are shorter than old users

Changing conditions:The online input distribution has changed, while the original evaluation is still dominated by long tasks.

Extended question:Should we directly replace the old regression with online success rate?

Derivation and reference solutions

Keep the old fixed set and the new set sampled according to the new distribution, and report the success, cost and risk of each layer. The old set protects the original capabilities, and the new set evaluates the existing service objects. Changes in the total score need to indicate layer weights to prevent changes in user composition from being misinterpreted as algorithm improvements.

The principles that remain unchanged:The evaluation conclusion depends on task distribution and clear sampling definition.

Model evaluation is shared and written to the database

Changing conditions:Different attempts will see the previously created order or report.

Extended question:Why is increasing the number of repetitions unbelievable?

Derivation and reference solutions

Because environments are not independent, subsequent attempts may hit old results or be affected by resource competition. Use an isolated initial state and a unique business space for each run, verify the initial state after cleaning, and then compare versions; contaminated runs cannot be counted as independent successes.

The principles that remain unchanged:The task and the environment together constitute the sample, and the residual state cannot be hidden.

Easy to make mistakes

Select only successful cases
Repeatedly look at the reserved set when adjusting Prompt
Acceptance criteria change with model output

References

It is designed based on public technical information; the reference materials support the technical mechanism, and the scenarios and scoring standards are designed by this website and do not represent the original interview questions of a certain company. New Q&A and migration cases are added for principle explanation, and source verification and case operation verification are recorded separately.

Check how far you understand

After reading, you can explain the principles, boundaries, and trade-offs against these standards. It is up to you to evaluate your mastery; if further verification is needed, complete the small tasks below.

Basic standards met: Know the need for success, failure and rejection samples.
Intermediate and advanced signals: Able to define the initial state of the environment, grouping and labeling review.
Senior criteria: Make task distribution, risk thresholds and data versions into long-term mechanisms.

Continue to do advanced research experiments

Let reviews read events and actual effects

Differentiate between run success, result contracts, and unscored semantic quality along actual checkpoints, events, and servicer receipts for the same research task.

Read full text and fault analysis → · Download Reliability Experiment v3 ↓

python3 cli.py memory-put
python3 cli.py submit
python3 cli.py run
python3 evaluate.py
python3 -m unittest discover -s . -p test_lab.py -v

Keep evidence and check item by item

Explain the basis of passed in terms of events and checkpoints.
Point out credible observations of scope, references, idempotent effects, and step budgets respectively.
Semantic_support and model_quality are not_scored, and passed cannot be written as model quality score.

By default, deterministic summary functions and synthetic documents are used; real operating trajectories and mechanical constraints are evaluated, and there is no semantic support to verify the real model.

Hands-on verificationComplete on demand · Suggestions15 minutes

Design 12 evaluation sample quotas for enterprise research agents to illustrate various types of failure coverage.

Expand acceptance requirements and checkpoints

Contains permissions and tool failures
Each question has a judged acceptance
Debugging and Reserved Set Isolation

Key inspections

Stratified sampling by business and risk
The initial state of the task and acceptance can be reproduced
Prevent development test leaks