🤖 Turning Dread Into Numbers 📈
When it comes to artificial superintelligence, "optimism is not a policy."
Briefing
The risks of artificial superintelligence (ASI) begin and end with human nature. As Cornelia C. Walther writes in Forbes, “optimism is not a policy; quantified risk is.”
As AI accelerates, so do its dangers. But there is hope. “Turning dread into numbers is possible.” The institution of software development needs to plan for threats. “Three moves — align, scrutinize, incentivize — distill intuition into insight, and panic into preparation.”
Or to (over)simplify the thesis: humans must proactively and explicitly codify the right values into the technologies we create—and the processes we use to create them.
Links
Scaling Laws For Scalable Oversight, a recent paper on how lower-level control mechanisms might manage up. Co-authored by Engels, Baek, Kantamneni, and Tegmark