hot-gay-rationalist:

nostalgebraist:

hot-gay-rationalist:

nostalgebraist:

hot-gay-rationalist:

nostalgebraist:

raginrayguns:

nostalgebraist:

“Where do Bayesians get their numbers from anyway,” installment (n+1)

[cut cut]

[snup]

[snop]

[snip]

[snrp]

[snep]

I think it’s because the uniform Ap distribution does not represent a state of “full uncertainty.” It in fact represents the statement “I think any probability is just as plausible as any other one,” and for most possible hypotheses, that is not a good description of most agents’ state of uncertainty about them! And once again we run into the problem of conflicting intuitions because I think that states of really total uncertainty are the ones that need to obey the rules of probability the most because they’re the easiest to fuck up.

And this problem is just isomorphic to the “finding priors” problem, which is the greatest weakness of the method. The Solomonoff solution, priors proportional to the negative exponential of the hypothesis’ complexity, is one that also appeals to me intuitively and mathematically - the basic argument that adding one bit makes twice as many hypotheses available and therefore should cut probability in half looks very good to me.

But there is no such thing as total uncertainty. Even in the ice cream case, you know something about the human population in the present and how it’s likely to change in the next 25 years, you know something about the prevalence of ice cream, etc - and even if you were completely ignorant, I hardly think you’d believe that “it’s as likely as not that that proposition is true,” or that literally any probability assignment is as justifiable as any other. The uniform Ap distribution is not special nor does it represent full uncertainty, because there is no such thing as full uncertainty, and because different amounts of information are represented by different Ap distributions (another example Jaynes uses is that of the existence of life in Mars at some point in the past, whose Ap distribution for himself he claims he’d describe as something like the Haldane prior).

—EDIT to add:

But also, as you and raginrayguns pointed out, it’s in practice impossible to really use the same heuristic rule about assigning more “reasonable” plausibilities to things when the thing is murky enough, as humans. But… all of my argumentation isn’t supposed to apply to humans :P I’m not concerned with what we can actually accomplish in the physical universe, I’m concerned with what’s the optimal way of reasoning, what’s the golden standard, hyper ideal, the Platonic Form of reasoning. Whether we can implement that and what we can do when we can’t is a completely different topic. But if I can determine that Bayes is indeed the ideal perfect unachievable golden standard of reasoning, then I can move on to see what approximations I can make and when and how I should deviate from the strict uncomputable solution.

OK, first of all a response to the last point: this conversation started with me objecting to some statements by a human, which are still floating up there above all the [snip]s and [snop]s.

As I’ve been interpreting it, this conversation has been about human approximations and not ideal reasoning, and to the extent that you’re only talking about ideal reasoning, you aren’t addressing the original question, which was “is the behavior of Robin Hanson (who is a human, not a JaynesBot) justified here?  Why or why not?”

Second: ultimately, I think the conflict here comes down to this question:

“Is it right to ‘add’ information we don’t feel like we know in order to make our representation of our uncertainty obey the probability axioms?”

It seems like our intuitions are diametrically opposed here.  I actively think this extra information shouldn’t be added, while you think that rationality demands that we add such information, and refer to not doing so as “fucking up.”

I’m going to give two examples of what I mean by “adding information.”  Note that I’m using “information” in an informal sense, not in the Shannon  sense or anything.  And also that, as always, I’m an amateur in these things and I wouldn’t ever want to imply that I’m the first to have ever thought of these objections – merely that I don’t know what the expert responses to them look like (though they presumably exist).

A thing I was getting at with my “conjunction” points yesterday is that if you know a probability distribution, you know information about the dependence structure of the events in it.  However, sometimes I don’t feel like I know the dependence structure of a set of “murky” events.  Representing my uncertainty as a distribution requires me to choose one, and this feels wrong.

For instance, let’s look at your example of the machine with two lights.  In that example there was no problem because the dependence structure was given (we know P(blue|red) = 0 and vice versa).

But now suppose I have the same machine, just as mysterious, except now I know any combination of the lights would be possible.  The possibilities are now {neither, red only, blue only, both}.  We could split this into various events like “red” which would be the set {red only, both}.  (I’ll give the event {both} the name “red&blue.”)

Knowing nothing about the machine, I don’t know what the dependence structure is.  Maybe the two lights are independent like two flipped coins, and P(red&blue) = P(red)*P(blue).  Or maybe they have some kind of dependence: maybe only “red only” and “blue only” are possible, or maybe only “neither” and “both” are possible.

What should my A_p distributions be here?  They can’t be uniform for each outcome because there are 4 outcomes and the means have to sum to 1, not 2.  (They still should have support over the whole interval [0,1] because maybe the machine just does blue every time or w/e.)  There’s probably a MaxEnt answer here?

In any case, whatever answer I choose, it will imply a dependence structure.  If I have a probability distribution over the space {neither, red only, blue only, both} then I can compute things like P(red|blue).

But actually I feel totally uncertain about those things, which I was not informed of at the outset.  They are “extra information,” and the idea that an object involving this information is the “right” representation of my state of uncertainty seems strange to me.

Is there any one dependence structure here that “correctly” represents my total lack of knowledge about the dependence structure of the machine’s behavior?  It feels counter-intuitive that there would be, though maybe there is.  Maybe I’m missing something here?

Here’s a second example of “extra information.”

Famously, if all you’re given are a mean and a variance, the MaxEnt prob. dist. on R is a Gaussian.

In going from “mean mu, variance sigma^2” to “N(mu,sigma^2)” I become able to compute many new things.  For instance, I can now compute any moment of this distribution.  I could tell you its fourth moment, say (and it would be finite).

However, the information provided is also consistent with other distributions, such as the Student’s t with nu > 2 (technically, a non-standardized Student’s t).  However, the Student’s t does not have defined nth moment for n >= nu.  So, the information provided is consistent with a Student’s t with nu = 3, but the fourth moment of that distribution is undefined (in the sense of being “infinite,” i.e. the integral diverges to +infinity).

So, suppose you come along and say, “Rob, there’s a distribution with mean mu and variance sigma^2.”  And I think, okay, there's some nonzero chance it’s a Student’s t.  After all, that’s a distribution that comes up in real life.

Now you ask “okay, Rob, what’s its fourth moment?”  And if I were a MaxEnt machine I’d happily spit out the fourth moment of N(mu,sigma^2).  But I, Rob, know that Student’s t is out there in the world, and that its fourth moment is infinite!  How do I incorporate this knowledge into a probability distribution?  If I have, say, some distribution over distributions in which the relevant Student’s t has a nonzero probability epsilon of being the right one, even if epsilon is tiny, the “expected value” of the fourth moment will still be infinite (infinity * epsilon = infinity), and I’ll spit out “infinity,” not “fourth moment of N(mu,sigma^2).”

So it seems like doing MaxEnt makes me forget that Student’s t is a possibility, in that it leads me to draw conclusions that seem unreasonable unless I think a Student’s t is actually impossible.  The information “mean mu, variance sigma^2” doesn’t just fail to tell me the fourth moment; it fails to tell me that it’s even finite.  The state of uncertainty I’m in, having received that information, seems very poorly represented by a Gaussian.

Again, I don’t doubt that people have thought about these issues and come up with answers to them; I just don’t know what the answers are.  And in sum, the disagreement here seems to come down to the question of whether states of uncertainty should or shouldn’t be “altered” to give me a prior that acts like a probability distribution.

(via hot-queer-rationalist-deactivat)

hot-gay-rationalist:

nostalgebraist:

hot-gay-rationalist:

nostalgebraist:

raginrayguns:

nostalgebraist:

“Where do Bayesians get their numbers from anyway,” installment (n+1)

[cut cut]

[snup]

[snop]

[snip]

tl;dr my subjective feelings about very inferentially distant propositions don’t feel like subjective plausibilities, and I get the sense that this is true for most people.  The way Bayesians quote numbers seems strange to many people, not just because it is unfamiliar, but because it seems to conflate the “sure uncertainty” one has about a fair coin with the “unsure uncertainty” one has about inferentially distant events.

About the difference in feeling, that’s indeed true, and in fact Jaynes has a whole chapter about this subjective difference and I wrote a post about it.

But I’m fairly certain that I wouldn’t say 50% to a question like that, because… well, I don’t know, full uncertainty doesn’t feel like “it could go either way” to me? And that’s part of the thing in the link, probabilities aren’t “brute numbers,” they have distributions. I’m also uncertain about my uncertainty, and if someone asked me that, I’d probably say something like “40% with a very very very wide tail.”

(By the way, I did read the whole thing, I just wanted to save space.)

It seems like you’re one of “nature’s plausibilists” — someone whose mind just naturally assigns a subjective plausibility to every proposition.  Which is pretty cool, don’t get me wrong, but my hunch is that this is not a common trait.  And the ultimate justifications for plausibilism are intuition-based, as ultimate justifications in philosophy tend to be.

Hmmm… yeah maybe. Though I don’t feel like those justifications are really intuition based, they’re more like “what I’d want to reason like”? I don’t know, maybe I’m projecting, or I’m just happy to have found my qualia represented in Bayesianism, but the Cox Axioms look like what I would want to reason like, even if I didn’t in fact reason like that - and of course I don’t really reason like that, I’m a biased human, and my reasoning deviates predictably from perfect Bayes, but it still looks like I’d want to be Bayesian and that whenever I reason in a way that’s inconsistent with that I’d feel bad about it. I do feel bad about it.

(Last point: I think there’s an even more fundamental state of uncertainty one can have, which is being uncertain about whether a proposition even describes a state of the territory at all.  For instance, if you asked me whether I thought Max Tegmark’s “Mathematical Universe Hypothesis” was true, I would feel a fundamental uncertainty caused in part by the fact that I’m not even sure yet what it would mean for it to be “true,” or whether that’s a meaningful question.  That is, uncertainty about whether or not a proposition is vacuous is a second kind of uncertainty that I don’t think can be captured well with plausibilities.  I have no idea if the Mathematical Universe Hypothesis is “plausible”; I don’t even know if it can be true or false, and will have to do more thinking to resolve that question.)

Well, here I think we sort of shrug and just go with logic? In logic, a meaningless sentence is always false (has no model, describes no possible world), so I think you can give that some form of probability. Maybe.

Though that’s in fact an Open Problem in FAI, in the same order of “how to reason about hypercomputers?” So this is a part where I confess complete epistemic confusion and say that this is an unsolved problem and that it may well be the point where Bayesianism unravels completely and we find out that we need some other, more universal form of reasoning to be our unattainable golden standard.

But I don’t know that Bayesianism can’t solve that, and I don’t know that simply giving potentially-meaningless propositions a probability will give me headaches. I haven’t yet read this paper but maybe there’s some potential stuff there.

I understand that such an A_p distribution can be constructed, and I guess then what I’d say is “for most futurological and other highly speculative propositions, my A_p distribution is nearly uniform.”

You can then take the first moment of all these uniform distributions and get 0.5 out and play games with the 0.5s, but this will at best just be a way of reflecting that I have no clue about the answers to any of these questions.  Saying “I have a subjective plausibility and it corresponds to probability 0.5” seems misleading; I don’t have a subjective plausibility.

Anyway, if we try to represent this state with uniform A_p distributions, doesn’t that run into the conjunction problem I mentioned?  "P and Q" should be (in the general case) less likely than P or Q alone, but supposing any of these are sufficiently far from my experience, I simply feel in a state of complete uncertainty about them.  So if you first asked me about “P and Q” I would give you “0.5” or “A_p is uniform” or whatever, but I would also have said that if you had presented me with P or Q alone.  (Taken literally, this set of judgments would seem to imply that all the events I’m totally uncertain about are really the same event, i.e. imply each other with probability 1, which is something I certainly don’t believe!)

For the above reason the A_p formalism seems like an awkward way to express the state of really total uncertainty I feel about many things; it still assumes my states of total uncertainty can play by the rules of probability when they can’t.  I don’t feel like they should, either – that would imply more knowledge than I really have, viz. some sense of which events are relatively big in the probability space and which are relatively small (to resolve the conjunction issue).

I guess what I’m looking for here is the mental state “more research is needed” – like the mental state I was in when I first learned about the “P=NP?” question but before I had learned that most people thought P != NP, and knew that if I really wanted to have a subjective sense of how plausible P=NP was, I should look up what experts thought about it.  I don’t think this can be captured by A_p because of the conjunction problem I mentioned, though maybe I’ve gotten that wrong.

(via hot-queer-rationalist-deactivat)