evolution-is-just-a-theorem:

So if MIRI redid logical induction for uniform instead of pointwise convergence, how good would that be? (Just in terms of technical difficulty, i.e. what would it say about MIRI’s mathematical competence relative to other groups of basically-post-docs.)

Also, how much does having the pointwise convergence proof help in generating a uniform convergence proof?

Also, has anyone talked to MIRI about this? (Gosh it’s almost like the review process occasionally does some good).

@nostalgebraist, @jadagul

The thing is, the analogous uniform convergence statements would be false for the criterion they’ve formulated, and my sense is you’d need something very different to get them. So you’re asking about a hypothetical paper which would be about some other criterion; everything would depend on the quality of the replacement.

For uniform convergence, you’d need to be resistant to *every* trader at once, but with some relaxed notion of resistance so you can tolerate losing some money and the algorithm is supposed to lose less and less (against all traders) over time. This may be possible, but it’s a different goal that would require its own distinct setup and a different kind of argument.

The friend I was talking to knows some people at MIRI, and I think he’s interested in talking to them about this stuff.

Right. Any finite set of traders is defeated after finite (but arbitrary) time. But if you need to defeat an infinite set of traders, there’s no guarantee that you ever beat all of them.

Yeah, that seems like a good way of looking at it.

on MIRI’s “Logical Induction” paper

jadagul:

nostalgebraist:

When I first saw this paper, I said it looked impressive, and seemed a lot more substantial than MIRI’s other work, but I never really looked at it in much detail.  In the past week I’ve been having an extended conversation with a friend about it, and that spurred me to read it more closely.  I now think it’s much less impressive than it seems at first glance.

It occurred to me that the criticisms I made to my friends might be of wider interest, so I’ll write a post about them.

Summary:

The authors state something they call the “logical induction criterion,” meant to formalize a kind of ideal inference.  To satisfy this criterion, an inference procedure only needs to have certain asymptotic properties in the infinite-time limit.  Rather than being sufficient for good inference but too strong for practical computation (as the paper suggests), the criterion is too weak for good inference: it tolerates arbitrarily bad performance for arbitrarily long times.

The easiest way to see why this is problematic in practice is to consider the criterion-satisfying procedure constructed in the paper, called LIA.  Speaking very roughly, LIA makes a countably-infinite list (in unspecified/arbitrary order) of all the “mistakes” it could possibly make, and at discrete time n, does a brute force search for a set of beliefs which avoids the first n mistakes in the list.

Depending on the ordering of the list, LIA can make an arbitrarily bad mistake for an arbitrarily long time (some mistake has to go in slot number 3^^^3, etc.)  Nonetheless, LIA can be proven to asymptotically avoid every mistake, since for every mistake there is some time N at which it begins to avoid it.  Thus, LIA enjoys a very large number of nice-sounding asymptotic properties, but it converges for very different reasons than most algorithms one is used to hearing about: rather than converging to nice properties because it moves toward them, it simply exploits the fact that these properties can only fail in countably many ways, and ticks off those ways one by one.

Thus, LIA is like an “author-bot” which generates all possible character strings in some arbitrary order, and at each step consults a panel of readers to weigh in on its best work so far.  One could argue that this bot’s “asymptotic best work” will have any aspect of writerly brilliance imaginable, but that does not mean that the bot satisfies some “ideal writing criterion,” and indeed we’d expect its output in practice to lack any aspects of writerly brilliance.  (LIA differs from author-bot in that it has an objective, if very slow, way to find its “best work so far.”  But garbage in means garbage out.)

More detail under the cut

Keep reading

Good post, for those who are interested in such things.

I wanted to pull out one bit that I thought was interesting. Content warning for analysis.

Keep reading

Oh, yeah, it totally is a pointwise vs. uniform convergence thing!  Thanks, that is illuminating.

In the OP I wrote “it [the LI criterion] tolerates arbitrarily bad performance for arbitrarily long times,” but now I realize that’s not stating the full case, since it could be true even if we had uniform (but arbitrarily slow) convergence.  I linked your post in a Discord chat and said this:

i think “it’ll take an extremely long time, but eventually, the probabilities will be sensible” is implicitly imagining that the convergence is uniform, when it’s really pointwise.  like, we imagine that there’s some huge N such that if we wait until then, the probabilities will be generally “good,” i.e. will have a whole bunch of great properties at once.

but what we really have are just guarantees that each good property arrives sometime.  it doesn’t have to arrive simultaneously with properties that seem related, and we’re not guaranteed an overall “package deal” at any point (unless we can show that there’s an e.c. trader that gets us that whole package at once).

(via jadagul)

on MIRI’s “Logical Induction” paper

When I first saw this paper, I said it looked impressive, and seemed a lot more substantial than MIRI’s other work, but I never really looked at it in much detail.  In the past week I’ve been having an extended conversation with a friend about it, and that spurred me to read it more closely.  I now think it’s much less impressive than it seems at first glance.

It occurred to me that the criticisms I made to my friend might be of wider interest, so I’ll write a post about them.

Summary:

The authors state something they call the “logical induction criterion,” meant to formalize a kind of ideal inference.  To satisfy this criterion, an inference procedure only needs to have certain asymptotic properties in the infinite-time limit.  Rather than being sufficient for good inference but too strong for practical computation (as the paper suggests), the criterion is too weak for good inference: it tolerates arbitrarily bad performance for arbitrarily long times, and more generally, provides no guarantees whatsoever about the quality of the inference procedure’s output at any finite time.  Moreover (see remarks below about pointwise convergence), there is a principled reason for the lack of guarantees which is baked deep into the paper’s approach, and which I don’t think could be fixed without completely overhauling that approach.

The easiest way to see why the LI criterion is problematic in practice is to consider the criterion-satisfying procedure constructed in the paper, called LIA.  Speaking very roughly, LIA makes a countably-infinite list (in unspecified/arbitrary order) of all the “mistakes” it could possibly make, and at discrete time n, does a brute force search for a set of beliefs which avoids the first n mistakes in the list.

Depending on the ordering of the list, LIA can make an arbitrarily bad mistake for an arbitrarily long time (some mistake has to go in slot number 3^^^3, etc.)  Nonetheless, LIA can be proven to asymptotically avoid every mistake, since for every mistake there is some time N at which it begins to avoid it.  Thus, LIA enjoys a very large number of nice-sounding asymptotic properties, but it converges for very different reasons than most algorithms one is used to hearing about: rather than converging to nice properties because it moves toward them, it simply exploits the fact that these properties can only fail in countably many ways, and ticks off those ways one by one.

Thus, LIA is like an “author-bot” which generates all possible character strings in some arbitrary order, and at each step consults a panel of readers to weigh in on its best work so far.  One could argue that this bot’s “asymptotic best work” will have any aspect of writerly brilliance imaginable, but that does not mean that the bot satisfies some “ideal writing criterion,” and indeed we’d expect its output in practice to lack any aspects of writerly brilliance.  (LIA differs from author-bot in that it has an objective, if very slow, way to find its “best work so far.”  But garbage in means garbage out.)

EDIT 6/28/17: I’ve added some remarks about pointwise vs. uniform convergence.  The relevance of this distinction to the LI paper was originally brought to my attention by @jadagul​ in a response to this post, and I think it’s important enough that it should be discussed here for completeness’ sake.  To find the part I added, search for “pointwise.”

More detail under the cut

Keep reading

I was talking to someone yesterday about my usual objections to representing beliefs/credences as probabilities, specifically the stuff about how IRL you don’t fully know the sample space and event space, and probability theory doesn’t tell you what to do about this.  

For instance, if you encounter an argument that “A implies B” – where A and B are the kind of ideas which you’d be assigning credences to – and the argument convinces you, you now know that A (as a set in the event space) is a subset of B.  You didn’t know that before.  Yet you had some concept of what “A” and “B” were, or you wouldn’t have gotten anything out of the argument – you needed to know which sets in your event space corresponded to the ones in the argument.  But although you knew about those sets, you didn’t know about that subset relation.  How do you “update” on this information, or formalize this kind of uncertainty at all?  It’s conceivable that you could and it would be very cool to do it, but probability theory itself doesn’t include this case – which to me is an argument (one of many) that probability theory is not the right set of tools for formalizing belief and inference.

Anyway, the person I was talking to mentioned this recent (Sept. 2016) preprint from MIRI, “Logical Induction,” which tackles the problem just mentioned.  (There’s also an abridged version, 20 pp. instead of 130 pp.)  I have not read it yet, beyond the first few pages, but it looks cool.  (Reportedly there’s a lot of cool math in there but the method for doing the thing is absurdly inefficient, doubly exponential time complexity or something.)

My understanding is that MIRI people want to formalize “logical uncertainty” in order to make TDT work (because TDT invokes the notion without formalizing it), not to make Bayesianism/Jaynesianism work.  But it’s refreshing to see people interested in this kind of problem, because, from my perspective, it is the sort of new math Bayesians/Jaynesians would need to have in order to make their perspective compelling.  There’s this giant looming problem with trying to apply results about “ideal Bayesians” / “Jaynes’ robot” to finite beings that keep learning new things about their sample and event spaces, and I would have expected people to notice this long ago and get to work developing new formalisms to deal with it.  And maybe that’d result in some super-powerful reasoning method, or maybe it’d result in something useless because it turns out the computational complexity is necessarily very high, but in any event there’d be cool math and an interesting line of thought to follow.

(I keep saying there is very little work about this stuff out there.  Maybe I’m wrong?  I haven’t been able to find it, in any event)

ETA: this also doesn’t have the problem @jadagul identified with earlier MIRI papers, that they read like crosses between papers and research proposals – they prove a whole bunch of different properties/implications of the criterion they define at the start, and I’d imagine there are at least several Least Publishable Units in there

slatestarscratchpad:

nostalgebraist:

I’m not sure who I could usefully get back to about this (@slatestarscratchpad may know?), but: I avoided clicking on the affiliate link to “Numerai” on SSC for a long time because the text description made it sound like an unpromising idea, and then last night I did click it out of random curiosity and it’s actually much more interesting than the text makes it sound.  It strikes me as bad advertising that could easily be better.

The text says:

“Numerai runs a global artificial intelligence competition to predict the stock market. Build a mind to earn what’s yours.”

This makes it sound like some crazy startup made by people who want to tackle a famously impossible problem (“predicting the stock market”) by harnessing a buzzword (“artificial intelligence”).  The “build a mind” bit sounds like something that someone would say if they didn’t know anything about AI but assumed it was kinda like the movies.  And the fact that it’s a competition suggests that the company may not have even hired anyone who knows about AI, since the work is being done by volunteers like you.  (Plus “earn what’s yours” gives me these weird Nietzschean vibes … )

What Numerai actually is: a hedge fund based on three ideas

(1) a lot of financial companies are hiring data scientists and the like, and these data scientists are working with proprietary data; there would be a lot more insight into the market to be gained if all the world’s data scientists could play with that data

(2) aggregating, in some sense, a bunch of models generally works better than just using one

(3) encryption can be used to distribute the sort of proprietary data mentioned in (1) so that it can be publicly distributed and modeled by anyone who wants it, even though the actual content remains secret

Whether or not this is a good/moral/interesting/etc. idea is of course up for debate, but it’s clearly an idea devised by people who know what they’re talking about.  No “AI” or “minds” – just models, whatever you want to use – and “predicting the stock market” only in the sense that hedge funds try to, with the kind of data they have

(On the weird tech startup front, it is true that they advocate donating to MIRI, to the extent that on the form you use to collect the payouts from good model predictions [in bitcoin, natch] there’s a cell that lets you type in an optional MIRI donation to make)

I told them I thought their text was terrible, and offered to write them a better ad. They refused. You’ll notice that’s the only one with the description in quotes, as if to say “Look, I just print the advertising copy they give me”.

Wow!  I guess that’s all one can do then :(

(via slatestarscratchpad)

I’m not sure who I could usefully get back to about this (@slatestarscratchpad may know?), but: I avoided clicking on the affiliate link to “Numerai” on SSC for a long time because the text description made it sound like an unpromising idea, and then last night I did click it out of random curiosity and it’s actually much more interesting than the text makes it sound.  It strikes me as bad advertising that could easily be better.

The text says:

“Numerai runs a global artificial intelligence competition to predict the stock market. Build a mind to earn what’s yours.”

This makes it sound like some crazy startup made by people who want to tackle a famously impossible problem (“predicting the stock market”) by harnessing a buzzword (“artificial intelligence”).  The “build a mind” bit sounds like something that someone would say if they didn’t know anything about AI but assumed it was kinda like the movies.  And the fact that it’s a competition suggests that the company may not have even hired anyone who knows about AI, since the work is being done by volunteers like you.  (Plus “earn what’s yours” gives me these weird Nietzschean vibes … )

What Numerai actually is: a hedge fund based on three ideas

(1) a lot of financial companies are hiring data scientists and the like, and these data scientists are working with proprietary data; there would be a lot more insight into the market to be gained if all the world’s data scientists could play with that data

(2) aggregating, in some sense, a bunch of models generally works better than just using one

(3) encryption can be used to distribute the sort of proprietary data mentioned in (1) so that it can be publicly distributed and modeled by anyone who wants it, even though the actual content remains secret

Whether or not this is a good/moral/interesting/etc. idea is of course up for debate, but it’s clearly an idea devised by people who know what they’re talking about.  No “AI” or “minds” – just models, whatever you want to use – and “predicting the stock market” only in the sense that hedge funds try to, with the kind of data they have

(On the weird tech startup front, it is true that they advocate donating to MIRI, to the extent that on the form you use to collect the payouts from good model predictions [in bitcoin, natch] there’s a cell that lets you type in an optional MIRI donation to make)

UC Berkeley — Center for Human-Compatible AI | Open Philanthropy Project →

lmao, I actually gave into the clickbait and watched that Steven Pinker video and he’s just like “if we don’t explicitly program an AI to subjugate humanity, there’s no reason to think it will ‘evolve’ in that direction … especially not if, like with every gadget we create, we build in safeguards”

it’s like listening to someone who clearly has never been on deviantart (etc) and doesn’t know that niche fetish porn exists.  the innocence.  so many things i have seen on the internet and he has not