thinkingornot:

nostalgebraist:

thinkingornot:

nostalgebraist:

Also, at one point Bostrom uncritically describes AIXI (or something close to it) as an ideal reasoner, which is disappointing

I’m not talking about the computability issue here: even if AIXI is uncomputable (even if it were just slow), one could still say “it gets the right answer, and I want to get close to that answer computably and quickly,” just as a numerical analyst might say that they want their fast iterative method to closely approximate the answer given by a slow direct method

There is another problem, which is that AIXI has to learn everything from scratch.  Its early decisions, before it has learned how physics works, will be inferior to those made by something that came with a built-in intuitive physics.  Of course the advantage of AIXI is that it doesn’t rule anything out: given evidence for quantum mechanics, it can switch its belief to quantum mechanics without having to overcome pre-quantum intuitions.  But it’s not clear that this is any better than a system which starts with good intuitive heuristics, and can build itself new heuristics when they are needed and switch heuristics as needed.  (E.g. something that starts with human-like physical intuition, develops quantum mechanics, and then self-modifies to have a package of quantum intuitions which it can apply in appropriate cases.)  In short, pre-packaged info about one’s environment will help one early on, and need not hamper one forever if it turns out to be subtly flawed.

Why can’t you feed whatever workable version of AIXI you have all your intuitive physics? You’re only giving that data to the designers of whatever program you favor, but not to AIXI itself.

Part of the specification of AIXI (as I understand it) is that it uses the Solomonoff prior.  A prior putting (say) extra weight on Newtonian explanations would not be the Solomonoff prior, so it wouldn’t be AIXI.

Ok, but the only reason we have intuitions about Newtonian physics is because Newton was mostly right. In a world with different physics, we’d probably have intuitions more similar to the real physics than to Newtonian. And that’s because we (through thousands of years of natural selection, through hundreds of years of science and experiments) have accumulated a freaking huge amount of data. If AIXI is started out with that data, it would switch its posterior to one favoring Newtonian explanations quite soon, before ever being given a real problem. As to how to get that info into AIXI, just feed it a dump of the internet or something. You’re comparing apples and oranges by looking at a “naked” AIXI, but a human-prior program with lots of data. Start a human-prior out with wiped intuitions, or AIXI with the same data that led the humans to believe something, and AIXI will win hands down.

It seems we were thinking along the same lines – while you were writing this reply, I wrote a post about AIXI’s ability to learn from data, which retroactively works as a reply to your reply.

(via thinkingornot)

To expand a little on why AIXI is a problematic ideal, note that a computable approximation to AIXI has been taught to play Pac-Man (specifically a “partially observable” variant, where it only knows limited information about its surroundings).

You can read about this here (PDF) and watch MC-AIXI play (partially observable) Pac-Man on Youtube here.

It turns out that MC-AIXI is still learning and improving its performance after 250,000 “cycles,” where a cycle is a single step of interaction with the game (get input, choose a “button press”):

In particular, even after 250,000 cycles (if I’m reading the paper correctly), MC-AIXI still has not learned at least one crucial game mechanic:

Visual inspection of Pacman shows that the agent, whilst not playing perfectly, has already learnt a number of important concepts. It knows not to run into walls. It knows how to seek out food from the limited information provided by its sensors. It knows how to run away and avoid chasing ghosts. The main subtlety that it hasn’t learnt yet is to aggressively chase down ghosts when it has eaten a red power pill. Also, its behaviour can sometimes become temporarily erratic when stuck in a long corridor with no nearby food or visible ghosts.

There is no clear way to translate cycles into playtime, but using a clear underestimate of 0.1 seconds per cycle (this would require 10 button presses per second, faster than games for humans tend to work), 250,000 cycles translates into around 6.9 hours of playtime.  I suppose the experiment has not been done, but I imagine a Pac-Man-naive human could learn all the mechanics of partially observable Pac-Man in much less time (certainly they can do it with regular Pac-Man, an arcade game intended to be instantly accessible to new players).

What to make of MC-AIXI’s slowness?  One could chalk it up to the approximations involved, I suppose – they are complicated enough that it is hard to rule this out.

It could also be the case, though, that the AIXI approach simply takes a long time to learn Pac-Man because it has to rule out all possible sets of game rules, where humans would hone in on the sorts of rules we expect for games.  Thus, in some sense AIXI might do better than humans on the global set of “all possible games” (or “all possible sufficiently simple games”), while being pathetically slow, by our standards, to learn within the much smaller set of “games for humans.”

This has particular relevance for treating AIXI as an ideal reasoner in contexts where human-specific information is relevant (such as a hypothetical AI trying to manipulate a person, social group, society, etc.)  If it takes many hours to learn Pac-Man by ruling out all possible other games, how much harder must it be to learn language or social skills, which are orders of magnitude more complicated?  How many “cycles” of interactions with humans would AIXI need to hone in on human behavior in the space of all possible behavior?  Perhaps it would need more experience than is practically providable.  A reasoner that is ideal in some particular sense might thus never practically reach human-level capability to navigate a human world, which surely is not an ideal response to a human world.  (Meanwhile, it would have the ability to adapt – slowly – to any other sort of world, an ability that would never be relevant or necessary here on earth.)

thinkingornot:

nostalgebraist:

Also, at one point Bostrom uncritically describes AIXI (or something close to it) as an ideal reasoner, which is disappointing

I’m not talking about the computability issue here: even if AIXI is uncomputable (even if it were just slow), one could still say “it gets the right answer, and I want to get close to that answer computably and quickly,” just as a numerical analyst might say that they want their fast iterative method to closely approximate the answer given by a slow direct method

There is another problem, which is that AIXI has to learn everything from scratch.  Its early decisions, before it has learned how physics works, will be inferior to those made by something that came with a built-in intuitive physics.  Of course the advantage of AIXI is that it doesn’t rule anything out: given evidence for quantum mechanics, it can switch its belief to quantum mechanics without having to overcome pre-quantum intuitions.  But it’s not clear that this is any better than a system which starts with good intuitive heuristics, and can build itself new heuristics when they are needed and switch heuristics as needed.  (E.g. something that starts with human-like physical intuition, develops quantum mechanics, and then self-modifies to have a package of quantum intuitions which it can apply in appropriate cases.)  In short, pre-packaged info about one’s environment will help one early on, and need not hamper one forever if it turns out to be subtly flawed.

Why can’t you feed whatever workable version of AIXI you have all your intuitive physics? You’re only giving that data to the designers of whatever program you favor, but not to AIXI itself.

Part of the specification of AIXI (as I understand it) is that it uses the Solomonoff prior.  A prior putting (say) extra weight on Newtonian explanations would not be the Solomonoff prior, so it wouldn’t be AIXI.

(via thinkingornot)

Also, at one point Bostrom uncritically describes AIXI (or something close to it) as an ideal reasoner, which is disappointing

I’m not talking about the computability issue here: even if AIXI is uncomputable (even if it were just slow), one could still say “it gets the right answer, and I want to get close to that answer computably and quickly,” just as a numerical analyst might say that they want their fast iterative method to closely approximate the answer given by a slow direct method

There is another problem, which is that AIXI has to learn everything from scratch.  Its early decisions, before it has learned how physics works, will be inferior to those made by something that came with a built-in intuitive physics.  Of course the advantage of AIXI is that it doesn’t rule anything out: given evidence for quantum mechanics, it can switch its belief to quantum mechanics without having to overcome pre-quantum intuitions.  But it’s not clear that this is any better than a system which starts with good intuitive heuristics, and can build itself new heuristics when they are needed and switch heuristics as needed.  (E.g. something that starts with human-like physical intuition, develops quantum mechanics, and then self-modifies to have a package of quantum intuitions which it can apply in appropriate cases.)  In short, pre-packaged info about one’s environment will help one early on, and need not hamper one forever if it turns out to be subtly flawed.

I have read/skimmed some bits of Bostrom’s “Superintelligence.”

I was curious about it because Bostrom is a friend and collaborator of Yudkowsky’s, who makes Yudkowsky-like arguments about the dangers of AI (the book is about these arguments), but has an academic position and knows how to write in a standard academic style.  (I mean a good, lucid academic style – there are various “standard academic styles” that are pretty awful, but thankfully Bostrom doesn’t write in them.)  Calling Bostrom “the respectable version of Yudkowsky” would be a cheeky and incomplete, but not especially inaccurate, first-pass description of him.

From what I’ve read, the book is pleasant and contains some interesting information (the first few chapters summarizing the history of AI and the potential for things like whole brain emulation are very nice).

However, in many ways the meat of the book simply seems like a more academic, more measured, better organized, more caveat-filled version of the same arguments about AI dangers I’ve seen from Yudkowsky.  In many cases Bostrom cites Yudkowsky and endorses what he says (e.g. he cites the AI box experiment as a tentative indication that boxing an AI may not be safe).  My impression is that if I have not been impressed with Yudkowksy’s arguments, reading Bostrom has little to offer me; he is mostly re-packaging them for an academic audience.  (If anyone who’s read more of the book wants to weigh in on this, I’d be curious; at this point I’m not inclined to keep reading.)

One particular frustration is that Bostrom largely leaves “intelligence” (meaning in this context “general intelligence”) undefined.  He speaks of machines modifying themselves to become more and more “intelligent” and endorses the notion that there is a large ladder of potential levels of “intelligence” which ascends much higher than humans (so that the difference between an AI and a human might come to dwarf the difference between a human and, say, a chimpanzee).  Defining and investigating the concept of intelligence is a big project, and I can understand if Bostrom just wants to avoid it, but it seems to me that it can’t be avoided if you want to really assess how probable these scenarios are.  What do we really mean by “general intelligence,” and is it a thing that can really be made qualitatively much better than what humans have?

At various points Bostrom (like Yudkowsky) implicitly or explicitly uses a definition of intelligence that is something like “ability to achieve one’s goals.”  This is nicely clean, but problematic, because it doesn’t take into account the fact that some goals may have hard upper limits where others don’t.

In particular, this seems to apply to things having to do with social behavior.  I can imagine beings that are qualitatively better than humans at math, information recall, etc., since there are already orders of magnitude of variation in these abilities among humans.  (John von Neumann is a good example of a person who seems to have really been “superhuman” in these kinds of areas.)  However, social abilities like “ability to manipulate others” do not seem unbounded in these ways.  There are some people who are good at manipulation, and many of us have developed types of wariness, etc. to protect ourselves from these people, but it doesn’t seem like this is a “skill” like math ability that spans orders of magnitude.  Roughly speaking, are no “super-manipulators” out there who can manipulate ordinarily wary people (but not “super-wary” people?).

For instance, one of the most effective ways to get people to do your bidding is to start a cult: there are plenty of chilling stories about the level of devotion that cultists have had to their various leaders.  However, it’s not at all clear that it is possible to be any better at cult-creation than the best historical cult leaders – to create, for instance, a sort of “super-cult” that would be attractive even to people who are normally very disinclined to join cults.  (Insert your preferred Less Wrong joke here.)  I could imagine an AI becoming L. Ron Hubbard, but I’m skeptical that an AI could become a super-Hubbard who would convince us all to become its devotees, even if it wanted to.  If social abilities like this are subject to hard upper bounds that have already been nearly achieved, then there’s no potential for AIs to achieve their goals better by becoming superhuman at these abilities, which makes it problematic to just postulate an AI that’s “superhuman at achieving its goals.”  (In particular, the stories about AIs convincing programmers to let them out of the box because they are super-manipulative or whatever are much less convincing to me than the stories about, say, already free AIs developing powerful weapons and defense and then imposing their will through the threat of force, since technical ability may scale up in a way social ability doesn’t.)

slatestarscratchpad:

[snip]

Data point - one of the reasons I’ve never ventured into longer-form fiction is that I’m pretty sure my first effort will be nowhere near as good as HPMOR, and if that’s got people hate-blogging it and hate-Facebooking it all over the Internet, I don’t stand a chance.

I have a project along these lines I’m working on, but realistically I’ll keep editing and re-editing it forever in hopes of some version that I can one day publish without having a bunch of people tell me how stupid and terrible I am (which will never happen).

(I occasionally have people tell me how stupid and terrible I am because I’m blogging nonfiction, but I’m more likely to believe the people who tell me I’m stupid and terrible at fiction and start hating myself)

I think this might be a problem for people (like Eliezer) who are already Internet famous in one area before trying to learn a different one. Their first efforts in the new area will be less-than-perfect (because everyone requires practice). If you’re not famous that’s okay because usually nobody reads new stuff by new authors except a handful of their supportive friends. Eliezer combines starting out in a new territory with an already established base of haters willing to savage his first efforts and so gets the worst of both worlds.

As such, it’s astounding to me he’s done as well as he has, both literarily and psychologically. But it’s not a miracle I’m inclined to try to repeat.

(Having a hard time focusing on work so I’m taking a break for an hour or two, after which I will drink more coffee or something and try again.  Hence this reply)

I think you have a point about people who are already e-famous moving from one sphere to another.  However, I think it’s important to emphasize that the things that seem to drive HPMOR criticism aren’t lack of polish or writerly capability (though those things come up as a matter of course in the negative responses).

Instead, the criticism is largely driven by things that people also dislike about EY’s nonfiction writing: arrogant tone, scientific namedropping that seems intended to impress rather than communicate, focus on power over curiosity as a motivation, awkward references to hot-button topics that seem largely intended for shock value, etc.

(A long time ago someone was reading HPMOR who was familiar with EY, but didn’t realize he was the author, presumably because of the “Less Wrong” pseudonym.  She said the content was interesting but she found the tone very grating.  After I told her EY was the author, she said “oh, that explains so much.”)

Viewing this as a matter of “unpolished first-time fiction writing” misses the central issue.  The strongest distaste for HPMOR comes from qualities also present in EY’s nonfiction, so there’s no way the jump from nonfiction to fiction could explain that problem.  This is a very uncertain prediction, but I get the sense that if you made the jump to fiction, you’d get a similar pattern of responses – people who disliked your nonfiction would tend to dislike your fiction, and vice versa.  The dislikers might hit upon first-time writer mistakes like bad pacing and mention them in their critiques, but they wouldn’t be the motivating factors.

(A lot of HPMOR criticism is kind of like that John Kessel essay that said Ender’s Game was an attempt to make a perpetrator of genocide look innocent to the reader – it’s a very specific content criticism, so it’d be weird to see that essay and say, “well, I don’t think I can write anything as good as Ender’s Game, so I might as well not write fiction at all.”)

(via slatestarscratchpad)

nostalgebraist:

su3su2u1:

nostalgebraist:

Randall Munroe’s finally discovered the world’s greatest vein of nerd comedy material and he still can’t make a good joke

At least the throw-away alt text lead to this.

As always, the real thing tops any possible parody

I was thinking about this on the subway and was like “this really shows how bad EY is at PR, because he could have just made a shorter, nicer post that basically said ‘I am one of the main people associated with this idea and I actually think it’s BS, don’t believe everything you read’ and it would have been much more effective, instead he rants about propaganda campaigns”

Then I thought “wait, what if he’s noticed that fictional characters who rant about propaganda campaigns are often vindicated, and he’s just being Genre Savvy”

(via nostalgebraist)

su3su2u1:

nostalgebraist:

Randall Munroe’s finally discovered the world’s greatest vein of nerd comedy material and he still can’t make a good joke

At least the throw-away alt text lead to this.

As always, the real thing tops any possible parody

(via su3su2u1-deactivated20160226)