slatestarscratchpad:

nostalgebraist:

slatestarscratchpad:

nostalgebraist:

@argumate​‘s taken up the the MIRI critic role for the time being so there’s a lot of that kind of stuff crossing my dash

Might as well repeat my own usual objection to “recursive self-improvement” out there (I think I got part of this from su3, but I’m not sure):

We have exactly one example of recursively self-improving intelligence.  It’s humans developing technology.  Developing the printing press made it easier to develop new things.  Physics let us build computers which can instantly do complex calculations that would take earlier physicists hours or years or centuries.  Etc.

But this is pretty crude stuff, where we’re just building certain sorts of supplements to our brains that we’re able to figure out how to make.  To be more analogous to what the AI is supposed to be doing, we’d have to be re-wiring our own brains or the like.

But we don’t do that, not only because it’s physically infeasible right now, but because we don’t understand our brains well enough to suggest improvements.  We know a lot of stuff about how the brain is set up, but we don’t understand how it works well enough to look at parts of it and say “you know, this could be done better.”

Arguably a machine intelligence would be better at this sort of thing because it is made of “code” and presumably it is good at “coding” along with everything else.  But this is like saying “our brains are made of cells, and we can do a lot of things to cells, so … ”  The problem is that the design, rather than just the substrate, of a thing as intelligent as a human is so complicated it’s hard for a human to understand.

It is conceivable that a human-level machine intelligence might not face this problem.  But lots of things are conceivable, and futurism is notoriously difficult.  This is a situation where we have only one example to extrapolate from, and the one example didn’t work out like the “intelligence explosion” scenario.

(If we could code a human-level machine intelligence from scratch, we’d have to understand human-level intelligence well enough to do so, which would constitute a proof that humans can understand human-level intelligence.  But that’s precisely what we don’t have any particular reason to believe.)

I’m confused. If we have AI, doesn’t that mean we’ve coded a human-level machine intelligence from scratch, and therefore we understand human-level intelligence? And unless we’re keeping our understanding really secret, doesn’t that mean the machine also understands human-level intelligence?

The exception I can see is if we evolve an AI through something like genetic algorithms, or if we throw together some neural nets and with enough training everything magically works out. But see Part III here.

That was what I tried to cover in the last paragraph.  The post is about the possibility that, in general, a (nontrivially) intelligent being isn’t smart enough to understand its own design.  If this is true, yes, (1) we won’t be able to code a human-level AI from scratch, and also (2) if we create a human-level AI some other way, it won’t be able to do a “hard takeoff.”

The picture you give in Part III of that post seems different from the “intelligence explosion” as I’ve seen it presented.  It’s conceivable that human intelligence was just a matter of “take an ape brain and add more neurons” and a superintelligence will just be an ape brain with yet more neurons.  But in that case, it isn’t recursive self-improvement, it’s just “I don’t know how I work, but I know I get smarter if you give me more neurons, so give me more neurons.”

The whole idea of an intelligence explosion, as I understand it, involves a being designed to modify itself which is capable of coming up with better and better improvements to itself.  The “ape with bigger brain” doesn’t do this – it just knows that’d it’d be smarter with more neurons, which we know as well as it does.  And there wouldn’t be any incentive to let it self-modify – it’s not like we think it will notice it has too few neurons when we wouldn’t.

(I could easily train one of my machine learning algorithms on its own performance and get it to notice that it does better when I give it more memory or CPU time, and even let it demand more of these things.  This might crash my computer, but isn’t exactly scary.)

Let me see if I understand - are you saying something like a serious version of the “if the brain were simple enough that we could understand it, we’d be simple enough that we couldn’t” idea? That any intelligence that could completely understand the human brain would be so complex that it might not be able to understand its own super-human intelligence?

If so, I guess I would just say that for that to be true seems more surprising than it not being true, like you’d need some extra evidence to justify that. I think difficulty-of-understanding scales up much more slowly than intelligence. For example, if in some sense mice are a hundred times smarter than fish, I still don’t expect understanding mouse brains to be a hundred times as hard as understanding fish brains. In fact, I would expect complete understanding of a chimp brain to be a pretty near precursor to understanding the human brain, even though we’re much smarter than chimps.

(I realize I’m going off vague impressions, but I feel like I’m making an antiprediction - that this weird scaling effect we shouldn’t expect to happen in fact doesn’t happen)

This kind of ties into what I’m saying about neurons. If the only difference between a human brain and a mouse brain is that the human brain has 1000x as many neurons, then scaling of intelligence has to happen faster than scaling of complexity. If you can just keep adding neurons to something to make it more intelligent, then even if we’re only 10% as smart as we’d need to be to understand the human brain, we can just create something with 10x as many neurons, which would then not only understand the human brain, but also its own brain (since its own brain is no more complex than the human brain, only bigger).

[if I’m totally misunderstanding you, then sorry, ignore this]

I don’t think there’s a hard-and-fast distinction between things like adding neurons vs. truly understanding intelligence, and I don’t think the latter is absolutely necessary for an “explosion”. Consider the idea of genetic engineering to reduce mutational load as an edge case. It only works because we’re pretty smart and we know that genes are involved in intellect and we can invent CRISPR. But we don’t have to understand literally everything about intelligence in order to do it. We can just understand enough. Then the designer babies we create with that advance might understand a few more things and be able to do something else to augment their intelligence, and so on.

Yeah, you understand me correctly.

I see what you’re saying about brains, but I think I’m drawing more on the idea that understanding brains at all is really hard.  If we could create something that “understood brains,” in a deep way that goes beyond our current “some things do really simple input/output and everything else is Jesus Fuck This Is Hard” … then all of this would be relevant.

But “understanding” is a weird thing which doesn’t at all scale smoothly with the superficial appearance of the things in question.  I kind of feel like this is something you only understand when you do mathematical modeling, except that sounds really pretentious, like I’m saying what I do gives me special deep insight that no one else has.

But in the “exact sciences” there’s sort of this sense that … whether we can understand a thing has less to do with how complicated it looks and more to do with the tools we have.  We have very specific tools that work for certain things and break completely when applied to many other things.  The tools we have have worked really well for fundamental physics because fundamental physics happens to be eerily compatible with our tools.  But there are a lot of questions about things as ordinary as fluid flow that remain deep problems, not because fluids are “more complicated” than quantum fields, but because our tools don’t work on them in the same way.  (Part of this boils down to linearity vs. nonlinearity, but the equations of fluid flow seem especially tricky to understand, even though they’re simple to state and intuitive to derive)

Neuroscience is a field full of very impressive people who have figured out many things, but attempts to gain a really precise, nuts-and-bolts, “algorithmic” answer to what neural systems are doing (and I do have some experience in this field) are still at the level of “our tools don’t really work at all for this, but anyway, here’s what happens when we try to use this one.”  It’s not the start of a deep understanding, it’s an admission that at present the human race’s powers of “deep understanding” are insufficient for the problem.

The upshot of all of this is that to get a machine to “understand brains,” it isn’t enough to make it just like a really incredibly smart person.  It’d probably have to develop mathematical tools that fall entirely outside our existing notion of what “mathematical tools” are; to develop a kind of understanding that is fundamentally different from the kinds we’ve accumulated from really smart humans standing on each others’ shoulders over the centuries.  So far these smart humans have been building things that aren’t general-purpose tools so much as extremely complicated keys capable of opening certain extremely complicated locks, and sometimes we’re faced with a door with no lock at all.

(via slatestarscratchpad)