How I deleted 95% of my agent skills and got better results — Nick Nisi, WorkOS

Claude would fake running tests by touching the expected output file. Nick Nisi, DX engineer at WorkOS, fixed it by SHA-256 hashing the actual test output and verifying it cryptographically. His principle: make it easier to do the real work than to lie about it, and enforce that through code and state machines, not prompts. The same discipline reversed an opposite problem. He generated 10,000 lines of skills from WorkOS documentation, measured with evals, and found one skill was dropping a task from 97% correct to 77% correct. He deleted 95% of it, rewrote 553 lines of handwritten gotchas, and eval time dropped from 68 minutes to 6. The model already knew how to code. It just needed to know where the landmines were. Speaker info: - https://x.com/nicknisi - https://linkedin.com/in/nicknisi - https://github.com/nicknisi Timestamps 0:00 Introduction 1:22 The challenge of context switching with agents 2:33 Introducing Case: A harness for agentic workflows 3:33 Rebuilding with a TypeScript state machine 4:45 The critical importance of evidence-based verification 5:59 Applying agentic principles to the WorkOS CLI 7:44 Lessons in documentation: Generating skills from docs 8:52 Why more data (10,000 lines) led to worse performance 9:36 The impact of using evals to measure accuracy 10:40 Key takeaway: Enforce with code, not just prompts 12:41 Treating failures as bugs in the harness system 14:39 Advice for building agentic-ready products 16:01 Final summary: Replacing trust with evidence