The previous post contains one example of why I’m skeptical when people say “X is my ideal, and I’m not doing X, but I’m approximating X, so I’m doing well.”

“Approximating X” can mean a lot of different things.  In some cases, it may actually make you worse, especially if you are still very far from X in other ways.  Reality is nonlinear; consuming elemental chlorine will not get you “halfway” to the effects of consuming nutritious NaCl.  (Instead, it will kill you.)

In physical science, much effort is put towards quantifying the quality of approximations and determining exactly when a given approximation will and will not work well.  I often feel like people outside of physical science could stand to spend more time doing this, and less time elaborating the details of perfect ideals that will never precisely hold true anyway.

do bayesian epistemic norms make you less bayesian?

(Yeah, it’s another one of these kinds of posts)

I’m sure someone else has said this before me, but it only just became clear to me, so I might as well write it down.

Subjectively, I feel like I’m only capable of a fairly small discrete set of “degrees of belief.”  I think I can distinguish between, say, things I am 90% confident of and things I am only 60% confident of, but I don’t think I can distinguish between being 60% confident in something and 65% confident in it.  Those both just fall under some big mental category called “a bit more likely to be true than false. ”  (I’m sure psychologists have studied this, and I don’t know anything about their findings.  This is just what seems likely to me based on introspection.)

I’ve talked before about whether Bayesian updating makes sense as an ideal for how reasoning should work.  Suppose for now that it is a good ideal.  The “perfect” Bayesian reasoner would have a whole continuum of degrees of belief.  They would typically respond to new evidence by changing some of their degrees of beliefs, although for “weak” or “unconvincing” evidence, the change might be very small.  But since they have a whole continuum of degrees, they can make arbitrarily small changes.

Often when the Bayesian ideal is distilled down to principles that mere humans can follow, one of the principles seems to be “when you learn something new, modify your degrees of belief.”  This sounds nice, and accords with common sense ideas about being open-minded and changing your mind when it is warranted.

However, this principle can easily be read as implying: “if you learn something new, don’t not modify your degrees of belief.”  Leaving your degrees of belief the same as they were before is what irrational, closed-minded, closed-eyed people do.  (One sometimes hears Bayesians responding to each other’s arguments by saying things like “I have updated in the direction of [your position],” as though they feel that this demonstrates that they are thinking in a responsible manner.  Wouldn’t want to be caught not updating when you learn something new!)

The problem here is not that hard to see.  If you only have, say, 10 different possible degrees of belief, then your smallest possible updates are (on average) going to be jumps of 10% at once.  If you agree to always update in response to new information, no matter how weak it is, then seeing ten pieces of very weak evidence in favor of P will ramp your confidence in P up to the maximum.

In each case, the perfect Bayesian might update by only a very small amount, say 0.01%.  Clearly, if you have the choice between changing by 0% and changing by 10%, the former is closer to the “perfect” choice of 0.01%.  But if you have trained yourself to feel like changing by 0% (i.e. not updating) is irrational and bad, you will keep making 10% jumps until you and the perfect Bayesian are very far apart.

This means that Bayesians – in the sense of “people who follow the norm I’m talking about” – will tend to over-respond to weak but frequently presented evidence.  This will make them tend to be overconfident of ideas that are favored within the communities they belong to, since they’ll be frequently exposed to arguments for those ideas, although those arguments will be of varying quality.

This provides one possible explanation for why there seem to be many Bayesians who are confident of various futurological ideas, like the idea that “the Singularity is near.”  To me, these kinds of ideas seem hard to have any strong confidence in because they involve chaining together a bunch of possible future events in science and technology, each of which are themselves hard to be confident about.  (A great many things have to all “go right,” so to speak, in order for a Singularity to happen by some specific date.)

An ordinary person is capable of the correct level of uncertainty here, because they can read a book by Ray Kurzweil (or whatever) and say to themselves “well, sure, there was a lot of evidence in there, but this whole thing just still seems really unlikely” and unashamedly keep their beliefs in the same confidence tier they were in before.  But a Bayesian who reads a number of similarly minded futurist texts and dutifully “updates” in response to each one will soon find themselves feeling irrationally confident.

There are only so many times you can sincerely instruct your mind to “believe a little more in P!” before P starts to feel like a pretty sure bet, no matter what P is.

thank you, eliezer yudkowsky, for what you’ve done for me. Even though your reasons were weird

raginrayguns:

Eliezer Yudkowsky: Everybody is too irrational to understand why they need to give me money to save them from robots. I must make them rational
Eliezer Yudkwosky: *ends up creating evidence-based self-help community*

(via pluspluspangolin)

I’m still elated about “The Ballad of Big Yud” coming back

Riffing on that post from last night:

There’s a certain sort of person who sets their ideas up as explicitly “intellectual” yet explicitly opposed to academia.

A few examples of the sort of person I’m thinking of: Yudkowsky, Ayn Rand, Vladimir Nabokov.

The way these people talk has a number of appealing qualities, e.g.

• an overarching hard-mindedness, a refusal to make facile compromises or pretend that the truth is always somewhere in between (enabled by outsider status, independence from a social community of thinkers)
• A fondness for clear, precise, straightforward language that avoids cliches or jargon (except for jargon of their own invention!)
• A refusal to bow down to revered figures of the past unless those figures live up to their own standards

These people also tend to fail in characteristic, related ways:

• Overly strong focus on their own personal points of interest or anxiety
• Refusal to acknowledge the psychological and historical origins of their own beliefs or tendencies
• Attracting acolytes, when the last thing they want to do is start a “school of thought”
• Mistaking simplicity or apparent vividness of expression for actual clarity, and producing statements that sound “clear” but are difficult to actually make sense of (I can’t find the exact quote, but someone one wrote something to the effect that that Nabokov’s style was “so crystal clear it is difficult to see anything,” which also describes a lot of Yudkowsky’s blog posts; compare to Rand’s 4-point summary of objectivism, which is seemingly straightforward yet nearly meaningless)

I feel more sympathetic to these people than I do to traditional academics, but I have never found one of them that I could agree with whole-heartedly.

Today’s subway station thoughts:

I originally got into reading Less Wrong because it seemed a lot like academia except with people whose minds worked more like mine.  A version of academia where I wouldn’t have to act like I agreed with ideas that made no sense to me, like:

• Once a thinker is famous, you can (or should) never rightfully displace them from fame by arguing or by citing new information; the best you can do is produce a “critique.”  If sufficiently damning, your “critique” will eventually be read alongside the original thinker to achieve some sort of synthesis (?).
• The respectable way to think about a topic is not to look at information about it and think about that information sensibly; it’s to read an extremely wide range of famous thinkers, then “apply” them to the issue.  You must do this to be “an intellectual.”
• Any thinker can be “applied” to any issue, but they must be treated as stand-ins for static, fixed sets of ideas rather than people of their time.  Aristotle and David Hume can be brought back from the grave to speak upon, say, modern political issues.  But they must say exactly the things they said centuries ago, even though they would surely say other things if they were alive today.
• All famous books are famous for a reason.  There is something good somewhere in Plato’s Republic, even if you can’t see it.
• There are no stock phrases that shouldn’t be reified.  If you can say the words “free will” or “the self” then there must be a set of True and False propositions about these things, and we can usefully talk about them by using these phrases.  Asking whether such a concept makes sense in the first place is anti-intellectual.
• As a consequence of the previous point, “the hard problem of consciousness” is a sensible and non-misleading term.
• You should accept and enjoy, and ideally practice, a turgid, imprecise style of writing that is totally inconsistent with any standard of good writing that exists outside of academia.  You don’t need to think carefully about the definitions of the words you use – in fact, it is better if you don’t, as long as you use big words.  Say “ontology” instead of “stuff,” “irreducible” instead of “unavoidable,” “instigate” instead of “cause.”  (This sounds like I’m attacking humanists, but most scientists can’t write either.  If that sounds arrogant of me to say, well, it probably is.  But I just mean relative to ordinary standards of writing – what you’d find in average-quality magazines, newspapers, essay collections, etc.)
• Etc., etc., etc.

Then I learned about Friendly AI, prediction markets, pop Bayesianism, the bad fanfic, “politics is the mind-killer but we’re all libertarians somehow,” and many other things in Less Wrong that didn’t make sense to me either and seemed like in-group shibboleths that I would never be comfortable with.

(The point of writing all this is that it’s funny to think of HPMoR as Less Wrong’s equivalent of Plato’s Republic.)

what is bayesianism? we (i) just don’t know

hot-gay-rationalist:

somervta:

nostalgebraist:

OK, apparently part of what I am going to do with this sick day, while the caffeine is still convincing me I’m not really sick, is to write this post, which I’ve had in my head for a long time but still haven’t written down.

tl;dr: after a bit of personal narrative this will turn into “reasons I’m not a Bayesian” or “reasons I don’t understand why other people are Bayesians, although maybe they have good reasons and I just haven’t heard about them yet.”

Maybe 6 years ago or so, I learned the word “Bayesian.”  This word seemed to refer to a particular philosophical position that was believed by a lot of smart people (note: me from six years ago was much more keen than current me on investing great weight in concepts like “smart people”).  Some of these people were bloggers I was reading at the time, but some of them were academics.  I knew that there were rival positions, like something called “frequentism,” but all the smart people seemed to be Bayesians.  I wanted to be a Bayesian too, but I told myself that first I should probably figure out what Bayesianism was.

I only properly tried to do this a number of years later.  I had read some popular resources about Bayesianism, but they weren’t very satisfying, so I checked out John Earman’s academic book “Bayes or Bust?” from the library and started reading it.  I didn’t get very far.  Partially this was because I was trying it to read it during my first semester of grad school while taking a full courseload and studying for imminent quals.  But partly it was because Earman’s book was full of numerous exceedingly complicated and subtle arguments both in favor of and against Bayesianism.  The amount of heavy shit — both mathematical and philosophical — I’d have to think through before reaching a position on Bayesianism was very intimidating.

But if this was the state of affairs, why were there so many Bayesians?  Had they passed through these trials by fire unscathed?  Did they have lower philosophical standards than the ones that I, perhaps quixotically, was trying to maintain?  Was there a middle road between the pop presentations, like Eliezer Yudkowksy’s — which weren’t nearly serious enough for me — and the presentations like Earman’s, which were so serious they scared me off?  And again: if this was all so hard to make sense of, whence all these Bayesians I kept meeting?

I still don’t know the answer to any of these questions.  Below, I’m going to try to talk a little bit about what Bayesianism appears to be, to me, and why it doesn’t seem to be intuitive (according to the arguments in its favor I think I actually understand, which is not all of them).

Read More

Are you familiar with things like Cox’s Theorem and Jaynes’ derivation of probability theory? In other words, people start with what seem to be fundamental principles of rational thought/belief and then prove that based on these (and sometimes certain other) assumption that you must use probabilities or something equivalent to them

I didn’t read all of this because it’s enormous, I just read parts and glossed over, but yeah, without grokking Cox’s Theorem one wouldn’t necessarily see how this makes sense.

Furthermore, there was a lot of talk about “what probability is“ (unless I misunderstood those parts) which is… a very silly thing to ask? Probability isn’t anything, that’s like asking what a blorgh is. The division between frequentists and bayesians is exactly what meaning one should ascribe to the word “probability.” When I talk about probability, I’m talking about subjective degrees of belief that obey the Cox axioms.

And then this person said that probability distributions in cases like the examples listed are “physically embodied” by those frequencies but… that’s assuming the consequent. If you assume that a frequency is a physical thing (it’s not) and that probabilities should only exclusively talk about that then sure bayesianism makes absolutely no sense. But since it turns out that probability-as-frequencies is a special case of probability-as-logic, I don’t know why anyone would be remotely interested in talking only about that subset.

And there was also some objection about how subjective degrees of belief are not a good description of human reasoning and, well, yeah? I mean, they really aren’t, probability-as-logic is prescriptive not descriptive. It’s not supposed to say how we do reason, it talks about how we ought to.

But as I said, I haven’t actually read the whole thing because it’s long, so I might be misrepresenting OP’s position here. I’ll read it later and reply on my main blog at length, possibly pointing to some other sources but… well, unless you grok Cox’s Theorem or at least believe that the axioms lead uniquely to the conclusion, this may not make much sense.

A few points:

1. The only thing that Cox’s theorem does, AFAIK, is to convince us that if we have a set of synchronic "plausibilities,” they should obey the probability axioms.  (It does the same work as synchronic Dutch book arguments, which I mentioned briefly in the post.)  In particular, it doesn’t say anything about anything diachronic, like conditionalization.
2. If you want to see someone more qualified than me making the same distinctions I am making here (synchronic/diachronic, Cox doesn’t get us conditionalization), so that you have evidence I’m not just some crackpot and thus have more reason to read my post, see Jonathan Weisberg’s “Varieties of Bayesianism,” available here (particularly section 3).
3. The justification of synchronic probabilism (i.e. what Cox purportedly does, though not everyone agrees that it actually does so) is the least questionable aspect of all of this to me.  I’m willing to accept intuitively that if I should be assigning a “plausibility” to every proposition, then my “plausibilities” should obey the probability axioms.  What I am less sure of is, first, that I should be assigning plausibilities, and second, that I should update these plausibilities by conditionalization (the “Bayesian update”).
4. I’m confused by what you mean when you say frequency is “not a physical thing.”  If I have three lemons and one apple in front of me, surely it’s a physical fact that ¾ of the things in front of me are lemons?  (Yes, it’s defined in a somewhat abstract way relative to the “brute” physical facts, but so is the fact that “there is an apple in front of me”; the only real brute physical fact is a bunch of subatomic particles or strings or whatever, and I don’t see how “¾” is less of a real physical thing than “apple.”)
5. The reason I talk about “what probability is” in the post is not that I think it has some real pre-existing definition in the world that we simply have to find.  That would be silly!  I do it because I’m trying to remind the reader that frequencies in physical samples are not obviously the same sorts of objects as subjective probabilities, and so an intuitive update rule for the former case is not obviously intuitive for the latter.  Of course if you use the conditionalization rule in all cases, then my use of it in only some cases must look parochial and silly.  But my point is that it needs to be shown that the same conditionalization rule does work in all cases!  What matters is not what is most general, but what is correct.  "All ants are insects" could be derived as a special case of “All animals are insects” if one believed the latter, but that doesn’t make the latter any less false.
6. I agree with you that I wasn’t very clear on the prescription vs. description issue.  Maybe I’ll write a follow-up post about that?  In short, what I was trying to say is that if we begin with an ideal of reasoning that is very far from what humans naturally do, then our approximations of it are likely to be very poor approximations.  This makes the “well, at least we’re approximating the right thing” argument unconvincing to me.  I wonder whether there might be an ideal of reasoning more natural to us (perhaps not involving assigning plausibilities to all propositions) from which we could derive, through a parallel set of arguments, another equally ideal system which we would actually be able to approximate well.

(via hot-queer-rationalist-deactivat)

jonomancer:

nostalgebraist:

jollityfarm:

nostalgebraist:

[snip]

If you’d like, I can send you a really good deconstruction of a bunch of ‘traditionalist’/neoreactionary arguments, but 

a) you’ve probably already read it, and
b) it’s made by one of those big yud fans iirc (slate star codex or something)(which doesn’t make the arguments presented any less true, but I’m more likely to take something with a grain or two of salt if it stinks of lesswrong dot com on the internet)

Basically these dudes remind me of Porfiry from The Golovlyov Family, who throws words at you until you’re like “oh my god FINE WHATEVER YOU’RE NOT SAYING ANYTHING BUT IF YOU SHUT UP I’LL DO WHAT YOU WANT.”

Oh yeah I’ve read that one, it’s great!

This may also be of interest to jonomancer.

I don’t like every post on Slate Star Codex, but as a whole it is one of the most (unironically) good things I’ve seen come out of Less Wrong.

Yeah, i read Slatestarcodex. His writing seems to alternate between brilliant and really aggravatingly naiive. And sometimes he has to use pages and pages of very advanced Logic in order to reach a conclusion that other people seem to have figured out by going outside and talking to people. But at his best he invents some really useful tools for cutting through bullshit, like the concept of the “bravery argument”.

You have to steer clear of the comments, though. They remind me of that Onion article “ACLU defends neo-Nazi group’s right to burn down ACLU headquarters”

I know what you mean, but the “re-deriving common sense by academic argument” aspect of his blog doesn’t bother me much, because it’s also something I do myself.

I don’t have a very good track record with just trying to follow common sense because it is common sense.  Sometimes it’s because it just seems plain wrong to me.  More insidiously, I’ll sometimes try to pick up a social norm (about what actions “are common sense”) but end up with a distorted version of it in my head and end up doing something that I thought was common sense, but wasn’t.  And then when I end up doing something weird and people ask me why, I end up having this resentment towards society in general for “misleading” me.  Except society isn’t a person I can just go and talk to about this, so the resentment just sits there, unresolved.

At least with an academic argument, you can respond to “why are you doing that?” with something other than “I thought society wanted me to.”  And if the argument really does go wrong, you can look under the hood and try to figure out why.

The part of Slate Star Codex I don’t like, personally, is that Scott’s social group affiliations don’t really seem in line with his views to me.  For instance, even when he writes pages and pages about how Moldbug makes no sense, he still talks about Moldbug as though he’s some sort of chummy in-group member, like he’s “one of us” who just happened to have strayed a bit afield.  Meanwhile, whenever he talks about “tumblr” he talks about it in collective form as some almost incomprehensible and far-off entity, like an alien planet, even though he tends to agree with the typical tumblr post he links and disagree with the typical Moldbug post he links.

In general, he chooses a lot of the way he talks and frame things to suggest close kinship with people like the neo-reactionaries or with doctrinaire libertarians, even though he doesn’t share kinship in worldview with these people.  This seems to create an artificial constriction of his audience: a large subset of people read his stuff and think

“even if I agree with this, the cultural signifiers are screaming THIS IS NOT FOR YOU”

(e.g. whenever I link one of his posts on tumblr I usually get a few “WTF am I reading?” responses, partially just from culture clash)

and another large subset of people read his stuff and think

“the cultural signifiers are screaming THIS IS FOR YOU, yet I think it’s all complete bullshit”

(This may go some length to explaining the comments section.)

Meanwhile, I end up reading the same posts and thinking “I agree with this but it keeps making me cringe, because with each new sentence I envision more potentially sympathetic readers closing the tab.”

ETA: I think this would be the right way to write if his main goal was convincing certain groups of people, like neoreactionaries, to agree with him.  If you want to convince neoreactionaries to change their minds, you need to read stuff neoreactionaries will read.  But that ignores the large subset of his blog that is supposed to be advice/insight for like-minded people.

(via accordion-druid)