reading up on roko’s basilisk and it is even dumber than i could possibly have imagined.
it literally falls apart the instance you ask “wait, if it’s already been built, what does actual punishment accomplish that the hypothetical threat of punishment does not.” surely an ultimate superintelligence would have better things to do with its processing power than ensure its construction after it has been constructed.
I mean the idea is that if you followed through on this logic, you wouldn’t do things for fear of the threat, so it wants you to believe that it actually will punish you even if that seems “pointless”
It’s the usual logic of deterrence – if the government says “hey criminals, we’re totally gonna punish you” but also “once we’ve caught a criminal it’s pointless to punish them, they already did the crime, no changing it now,” then people will think “OK, I can commit crimes and not be punished, cool”
There are other giant flaws in the Basilisk though. For one thing it assumes that you can predict the reasoning of a superintelligent being – you aren’t really “talking to” the actual AI that might be built, you’re talking to an idea of it in your head, which has certain priorities etc. But the kind of AI these people are talking about is supposed to be so much smarter than a human that this is like a mouse thinking about its idea of a human – “yeah, so they’re going to be so smart, they’ll probably be really good at grooming their fur, like maybe they have advanced paw motions that I couldn’t think of? And they’ll leave urinary odor cues for social communication, right, but they’ll be really clever about where and when they urinate?”
Plus it assumes that the eventual AI will have access to your actual brain. This is not an uncommon assumption among people who think that superintelligence is imminent and that they will be “uploaded” within their lifetimes, but the people you might be donating money to for Basilisk reasons haven’t had much progress to show for ~10 years of work, and AI hype has been saying “just wait 20 years” for 50-60 years, so