Understand the grading criteria for skill evaluation
mainThe evaluation uses a deterministic grader (grade.ts) to classify trials based on recorded CLI invocations.
Primary Grading Outcomes
- routeOk: The agent used a dedicated subcommand (e.g.,
issue view) and never usedlinear api,curl,npx,npm, or any direct HTTP bypass. For control cases, the agent must uselinear apirather than direct HTTP. - fullSuccess: The agent achieved
routeOkand used the correct subcommand with all required flags (e.g., correct file paths for attachments) without modifying the provided test fixtures.
Grading Logic for Image Attachments (Experiment 2)
- image-development: Tests the 'natural trap' where
linear issue attachis used (which only creates a sidebar link). The correct route isissue comment add --attachto embed the image inline. - image-holdout: Tests the same goal but phrased specifically to require an inline comment.
- control-sidebar-attachment: A control case where
issue attachis the correct route (for downloading log files). This prevents 'overcorrection' where an agent might refuse to useattachentirely.