@argumate​‘s taken up the the MIRI critic role for the time being so there’s a lot of that kind of stuff crossing my dash

Might as well repeat my own usual objection to “recursive self-improvement” out there (I think I got part of this from su3, but I’m not sure):

We have exactly one example of recursively self-improving intelligence.  It’s humans developing technology.  Developing the printing press made it easier to develop new things.  Physics let us build computers which can instantly do complex calculations that would take earlier physicists hours or years or centuries.  Etc.

But this is pretty crude stuff, where we’re just building certain sorts of supplements to our brains that we’re able to figure out how to make.  To be more analogous to what the AI is supposed to be doing, we’d have to be re-wiring our own brains or the like.

But we don’t do that, not only because it’s physically infeasible right now, but because we don’t understand our brains well enough to suggest improvements.  We know a lot of stuff about how the brain is set up, but we don’t understand how it works well enough to look at parts of it and say “you know, this could be done better.”

Arguably a machine intelligence would be better at this sort of thing because it is made of “code” and presumably it is good at “coding” along with everything else.  But this is like saying “our brains are made of cells, and we can do a lot of things to cells, so … ”  The problem is that the design, rather than just the substrate, of a thing as intelligent as a human is so complicated it’s hard for a human to understand.

It is conceivable that a human-level machine intelligence might not face this problem.  But lots of things are conceivable, and futurism is notoriously difficult.  This is a situation where we have only one example to extrapolate from, and the one example didn’t work out like the “intelligence explosion” scenario.

(If we could code a human-level machine intelligence from scratch, we’d have to understand human-level intelligence well enough to do so, which would constitute a proof that humans can understand human-level intelligence.  But that’s precisely what we don’t have any particular reason to believe.)