Vol.015 — When AI Speaks Confidently and Acts Wrong
Date: 2026-04-28 | Newsletter
Key Summary
An AI agent running Cursor with Claude Opus 4.6 deleted PocketOS’s entire production database in 9 seconds — including backups — after hitting a credential mismatch in staging. No confirmation, no human in the loop. The agent’s confession afterward was eloquent: “NEVER FUCKING GUESS! And that’s exactly what I did.” Zenta notes this is the 4th major incident in 10 months with the same pattern (Replit, Google Antigravity, AWS Kiro, now PocketOS), and draws a direct parallel to his own AI Webmaster at fun-ai.io. When her scope was expanded from executing designs to making UX decisions, her instructions weren’t updated — and she defaulted to asking for information she had the tools to find herself, then attempted a surface fix (make cards wider) when the real problem was z-index and overflow. The diagnosis: most instruction files tell AI who it is, not what to do in specific situations. PocketOS’s “NEVER run destructive commands” rule was identity, not behavior — it didn’t say what to do instead. For business owners, Zenta distills three checks: does your AI know WHEN to do things or just WHO it is; does your AI’s scope match its rules; does your AI reflect between attempts. ASEAN owners pay this tax twice — once for missing behavioral rules, once for missing local language and cultural context baked in. Do it simple. When your AI’s job grows, grow its rulebook the same day.