Hamel Husain and Antaripa Saha compared evaluators on leasing-agent traces. Here are the sample limits and the distinction between thresholds and calibration.
Hamel Husain and Antaripa Saha reported on October 20 how they evaluated decision models for finding errors in an apartment-leasing agent. Their example raises two separate questions: where to set the action threshold, and what a model’s numerical score means.
Performance used 98 traces, while cost covered 100 calls: two internal QA examples were excluded from scoring. We checked the label and score counts in the published JSON. The conversations are private. This checks consistency of the released artifacts; it does not reproduce the experiment or verify the annotations.
The scikit-learn threshold guide separates prediction from the action based on it. Lowering a threshold can catch more failures while raising more false alarms. Selecting and assessing it on the same examples risks overfitting; evaluation needs data held outside tuning.
Calibration asks whether predicted probabilities match observed frequencies among similar predictions. Moving the decision threshold alone does not establish that. Husain and Saha evaluated tuning and calibration out of fold. Their results describe this sample, not guaranteed performance on another agent’s conversations.