Install Theme

Some quick NAB notes (taking a break from work):

(I have stuff to say about Sandifer’s discussion of Moldbug, but that’ll have to wait for next time – this is all Yudkowsky stuff)

Sandifer’s presentation of the AI Box Experiment gets a number of things wrong, and ends up being much too favorable to Yudkowsky (!).  He writes:

Unlike the Basilisk, however, [the AI Box] is not a problem for Yudkowsky’s thought, but an actually kind of cool idea. Indeed, it’s one of the reasons why intelligent people with actual achievements have taken Yudkowsky seriously. 

This could well be true, and it’d be fascinating if it was, but it set off a huge [citation needed] warning in my head.  In all my time reading about LW and interacting with LWers, I’ve rarely seen the AI Box come up, and generally it tends to be treated as another embarrassing skeleton in Yudkowsky’s closet (which is probably why it doesn’t come up).

This isn’t conclusive evidence by any means, but let’s just check one quick way to get our finger near the pulse of the current LW community: searching for the phrase “AI risk” on SSC gets 37 hits including a number of actual posts on the topic, while “AI box” gets only 7, most of them in comments.

OK, but Sandifer is talking about outsiders, and maybe they’re more impressed by the AI Box than LW itself is?  (That’d be a first.)  But who does Sandifer have in mind here?  Scott Aaronson is brilliant, has plenty of real achievements, and takes Yudkowsky seriously, and I can’t find him talking about the AI Box anywhere; he seems to think that AI risk is too currently murky to be worth thinking about at all.  Peter Norvig?  Can’t find anything linking him to the Box on Google.  Who am I missing here?

Sandifer then describes the AI Box Experiment as follows:

In it, two people make a monetary bet and then roleplay out a dialogue between a boxed AI and a person given the authority to decide whether to let it out or not in which the AI tries to talk its way out of the box. And it is important to stress that it is roleplayed: valid exchanges include things like “give me a cure for cancer and I’ll let you out.” “OK here.” “You are now free.”

This puts the emphasis entirely in the wrong place.  It’s true that the stated exchange is valid, but the whole point of the experiment is that it can’t work via those sorts of techniques.  The “Gatekeeper” player is indeed RPing a person from the future, but there are no constraints on what sort of ethics this person must have; they can simply decide that a cure for cancer (or whatever) isn’t worth it, and that’s that.  They can just refuse any request the AI makes.  This is spelled out explicitly in the rules:

The Gatekeeper party may resist the AI party’s arguments by any means chosen - logic, illogic, simple refusal to be convinced, even dropping out of character - as long as the Gatekeeper party does not actually stop talking to the AI party before the minimum time expires.

(Note the “even dropping out of character” part.)

If the AI Box Experiment were merely a kind of science fiction roleplaying about an attempt to mind-hack a player character, it’d make for interesting stories but wouldn’t have much argumentative force – which is exactly what Sandifer says about it.  But it’s not.  The claim is that Eliezer Yudkowsky can literally get you, not some imagined character, to say something specific by chatting with you online, even if you decide beforehand that you’ll never do it, and with nothing on the line except that he’ll pay you money if you don’t say the thing.  This isn’t about collaboratively writing SF stories, this is about Yudkowsky’s claimed ability to hack your actual brain, right now, in real life.  As I joked a long time ago, it’s kind of like this:

image

This is what makes the AI Box a skeleton in Yudkowsky’s closet – because people just can’t imagine how this could actually be done, and he won’t release the logs, and so people tend to assume it involved some sort of tricky rule-gaming rather than anything at all relevant to futurism.

philsandifer:

achairforjane:

philsandifer:

jbeshir:

nostalgebraist:

reddragdiva:

argumate:

The reaction (ha!) to Neoreaction a Basilisk from the local rationalist(-adjacent) community has been narrowly focused on these core issues:

1. Is this book accusing Yudkowsky of being neoreactionary?

2. No really, is it? I mean why else would it group him with Moldbug?

3. That fuckin’ Basilisk story, that was totally misinterpreted.

Having read it, I think it’s helpful to understand that this book is not attempting to be the annotated history of Internet politics circa 2k10, and the claims that it does make in service of its overall trajectory are modest and reasonable.

It is also worth remembering that not every work of literature is a textbook intended to be interpreted as a sequence of logical propositions. A community that sees value in communicating information in the form of fanfiction, poetry, and jokes should be well aware of this.

Finally the book does not just discuss Yudkowsky, Moldbug, and Land, but also the Matrix, Hannibal, and the works of Milton and Blake, among other things. Tying these topics together in no way implies that Yudkowsky is neoreactionary, any more than it implies that Nick Land is one of the Wachowski siblings or that Moldbug is a good writer.

uh

wirehead-wannabe said: I haven’t read it but if it’s not trying to make fun of people for falling for the basilisk, and it’s not tying to accuse us (or Yud) of being Neoreactionaries, and it’s not trying to be a history, then what is it about?

we have a possible prizewinner for rational.txt

you know, there’s several thousand words of extracts on http://www.eruditorumpress.com/ perhaps you could read them and it might inform you

it was obvious in october that the reaction would be at best to wilfully misread the book (rationalism literally trains people in bad thinking) and harp on trivially disprovable actually made-up nitpicks, but for some reason i hoped i wouldn’t be right

???? where is the problem with the quoted text?

Can a guy not ask another guy what a book is about around here, or what

(Talking to someone who has actually read a book is often a lot more reliable than reading some extracts and making inferences, I think)

I don’t get why anyone is engaging with this stuff.

It’s the trolliest troll title in the history of trolling titles, discussing a topic good for little other than trolling. I don’t see why anyone engages with something which tells you that upfront for longer than it takes to thank them for the upfront warning that enables you to filter them out so readily.

Really, if I’d wanted a trolling title I’d have gone with the mooted Yudkowsky, Moldbug, Land: An Eternal Golden Cuckball

I tend to think of the first word of the book’s title as a verb.

I definitely think of them both as nouns, but don’t particularly mean the title to suggest a specific relationship between them. 

I am amused by the “trolling title” accusation, though, just for the narcissism of assuming rationalists were the primary audience. My assumption is that the word “Basilisk” will, for most people who see the title, seem strange and raise questions, questions that are deepened by the title’s lack of punctuation and the resultant uncertain relationship between its two main words.

I’d honestly never even thought of how rationalists would react to the title, and am more than slightly surprised at how objectionable people find it. 

It’s not that rationalists are the primary audience, but rather than the title looks exactly like a sequence of words one might put together to toss a bomb into an already tense and heated discursive situation.

See my analogy here.  This is one of those “intent doesn’t determine the meaning of words” things – the author of my fictional book about Corbyn and anti-semitism might in fact have been totally unaware that they were throwing a bomb, but words once written do the things they do and not just the things one forecasted.

Like, to be clear, I don’t consider any of this especially important.  I don’t identify as a rationalist but plenty of my friends do – but I’m not worried about mean old Phil Sandifer ruining their reputation forever with his ~virally spreading, insidious book~.  I mean jeez.  Certainly if anyone actually reads the book they’ll see that you explicitly disclaim guilt-by-association at a number of points (and indeed I’ve seen a few posts by people who were disappointed that you portrayed Yudkowsky as not reactionary when they expected the opposite).

Indeed, while reading the book it’s become clear that my Corbyn analogy was way too strong, not just because it used a weighty thing to explain a trivial thing, but because the book’s actual text contains none of the detonable material that the hypothetical Corbyn book would have contained, even if the words “HIGH EXPLOSIVE” happen to be printed on the packaging.

It’s more that, if you’re surprised and curious, I’m trying to explain the thing.

(via eruditorumpress)

nightpool replied to your postSome notes on Neoreaction A Basilisk (I’ve read up…

question. When you say “Yudkowsky is not trying to build a positive system” but “He does tend to act like he’s deriving everything from the bedrock of Bayes’ Theorem"—it makes some amount of sense to me that Sandifer would focus on this from a narrative perspective? Even if Sandifer doesn’t think that he’s succeeding in that, its obviously part of what yudkowsky *believes* (or believed) he was doing.

To be clear, I don’t object to the idea that Yudkowsky thinks everything he said can be traced down to a solid foundation.  That’s a sensible thing to pick up on and run with.

What I object to is the idea that he’s building up from that framework, rather than tackling a series of puzzles more or less independently, although always with the (not much elaborated or justified) idea that it all “comes down to Bayes.”  He really is not building a positive system.  He’s trying to solve all the big analytic philosophy puzzles so well that no one will ever have to solve them again.

(He is fond of referring to things as “solved problems” if he has devised resolutions of them that satisfy him.  He thinks he’s the master puzzle-solver.  But not by virtue of any system of his, just by virtue his personal approach, much as any analytic philosopher will have their own characteristic way of going about the puzzle-solving task.)

other question. The patreon says that the book is “written in the form of an internet comment”. Yet you tend to refer to the book in a way that basically equates the narrator with Sandifer. Is there any attempt by the book to characterize/give an amount of unreliability to the narrator? Could that affect how some of this comes off?

There hasn’t been anything like this device in what I’ve read so far, or if there has been, I’ve missed it.

(Having looked at the statement on Kickstarter – “it is a horror novel written in the form of a lengthy Internet comment” – I think it’s just a less literal characterization of the book than you’re thinking.)

Some notes on Neoreaction A Basilisk (I’ve read up to p. 53 of 153):

So far, the book seems most interested in studying its chosen trio (Yudkowsky, Moldbug, Land) as, so to speak, “characters in horror stories” – as people who encounter certain related disturbing-and-apocalyptic philosophical concepts, and respond to them in varying ways.

This is an interesting angle, although it puts Sandifer in the unfortunate position of having to set up at least a rough account of what each of these writers is “doing” before he can really talk about his chosen subject.  (“Before” in the “logically prior” sense rather than “temporally prior” sense – in practice he weaves the two together, but the analysis of the three as “characters in horror stories” requires, as a necessary backdrop, an account of what it is that they are each constructing.)

The reason this is an unfortunate position is, of course, that the question “what is this writer doing?” has – for each member of the trio – been quite controversial, and a great deal has already been written about it in each case.  To really make a case, on any one of these questions, that engages with all the concerns and counter-concerns of interest to the die-hard buffs, one would need to write a whole book as long as Sandifer’s or longer, and of a much different kind.

As a result, to “die-hard buffs” of any of these issues, this book will be frustrating reading, because Sandifer makes – and has to make – a bunch of controversial claims without defending them them fully, or even at all.  In this he is (I imagine self-consciously) a lot like the trio, making rash strides across contested territory, which are likely to make people with a pre-existing interest in these topics say “wait!– but what about this nuance!–”  (I have been doing this about once per page, making the book pretty slow reading.)

I suspect this will get better as the book goes on, since it’s the scaffolding that frustrates me, and scaffolding once built doesn’t have to be built again.  In any case, it doesn’t seem like the purpose of the book is to present Sandifer’s account as definitive, but to use it as a springboard to other things.


The tone is highly evaluative, praising here and blaming there.  There is quite a lot of blame, but also quite a lot of praise.  The most basic formal unit in the book goes something like “[a member of the trio] says something correct or interesting here.  But then, in this other place, he falls flat on his face.”  Sandifer clearly finds all three members of the trio (sometimes) interesting as thinkers, (sometimes) appealing as creative writers, and (sometimes) “relatable” in their psychology.

So far, the book most closely resembles a (very long) book review.  As in a book review, it is very focused on which bits of a text Sandifer thinks are good and which he thinks are bad, and as in a book review, it means to introduce the texts in question to people who have not read them.


A lot of the specific things I’ve noted down while reading the book are of the “wait!– but what about this nuance!–” variety, and I’m not sure how many of these are really worth mentioning.  I will just mention one relatively large-scale case where I think Sandifer has gotten something wrong.

This is Sandifer’s account of what Yudkowsky’s Sequences are.  (I should note at the outset here that I don’t think Sandifer makes these look worse than they really are, just different.)  He writes that the Sequeneces are

of the largely abandoned genre of from-first-principles systemic philosophical worldviews, of a genuine intellectual heft comparable to Kant’s Critiques, assuming you don’t much care for Kant’s critiques.

He goes on to compare them to Spinoza’s Ethics, an attempt to derive a great deal of consequential philosophy via strict deduction from a small set of axioms.

But this isn’t what Yudkowsky is doing at all.  He has no list of axioms, and rarely makes anything as clean as a strict logical deduction.  He does refer to his earlier posts quite a lot, but mostly in the manner of an academic writing “as I argued in [reference 7] … ” or “as I argued in section 2.1 … ”  (He does tend to act like he’s deriving everything from the bedrock of Bayes’ Theorem, and Sandifer buys this self-characterization, but in fact the relation of most of Yudkowsky’s writing to Bayes is quite murky.)

One post in which he refers to a great many of his earlier conclusions is “Heading Towards Morality,” the start of the relatively late “Metaethics Sequence.”  Note how none of the conclusions are framed as established propositions, which will now have direct logical implications for the subject of metaethics.  Instead, they are simply a set of previous investigations whose twists and turns might be helpful to keep in mind – in particular, many of them identify (ostensible) mistakes which the reader can now be trusted to avoid.  When Yudkowsky goes on to develop his metaethics, he does so not by building on a previously established positive system, but as if from scratch – except with those previous investigations as helpful experience.

Yudkowsky is not trying to build a positive system in the manner of Spinoza, or Kant – or Hume, or Hegel, or any of those classic old system-builders.  What is he trying to do?  He’s trying to be a contemporary analytic philosopher.

Analytic philosophers do not tend to build their own positive systems.  Instead, they “work” on “problems” – things thought to be puzzling, paradoxical, or in need of clearer description – and thus build a set of individual investigations, like Yudkowsky’s.  Insofar as a positive worldview emerges here, it does so only implicitly, as the picture formed from large number of these investigations.

The extremely prolific David Lewis, for instance, advocated a number of positive positions, but not as part of a project of building a worldview.  Lewis’ work as a whole is often associated with “naturalism” or “physicalism,” but he didn’t set out to analyze the whole world on the basis of such things – rather, these things are characteristic of his approach to the various “problems” he “worked on.”  Even his most audacious proposal – that all possible worlds actually exist – was advocated on the basis of all the cute little puzzles it can solve.  (”Traditional treatments of modal talk in terms of operators face several difficulties… . All of these are easy to understand in terms of quantification across possibilities.”)  Much the same could be said of Saul Kripke, Hilary Putnam, and various other big names in the field.

Yudkowsky’s approach in the Sequences makes if you figure he’s trying to be like one of these guys.  This also, simultaneously, makes sense of his topics of interest.  Almost all of his big topics are puzzles of great interest among analytic philosophers: the nature of induction, the relation of statements to their referents, p-zombies, reductionism, metaethics, utilitarian aggregationcausal decision theory and related issues, the interpretation of quantum mechanics.

(This may also provide answers for those wondering why Yudkowsky focuses on some things that seem unimportant.  Sandifer writes that EY is “weirdly obsessed with” Newcomb’s Problem, but here he’s just acting like an analytic philosopher, working on a familiar puzzle.  Analytic philosophers care about some really goofy-seeming shit: former tumblr user ogingat, a very smart analytic philosopher who found Yudkowsky’s approach to his field naive and infuriating, was himself very interested in the puzzles produced by the concept of a spinning disk.  Not that I’m judging him – I had fun arguing with him about it.)

Now, is any of this important for Sandifer’s book?  Well, maybe not.  I’m not sure that viewing Yudkowsky my way rather than his would change more than a sentence here or there.  But this is one of the various cases in which it looks to me like Sandifer hasn’t done his homework – not just about weird internet history, but about the history of ideas.  (He did not seem to be aware, for instance, of the mainstream intellectual context for Yudkowsky’s TDT idea, presumably for the broader reason that he is not aware Yudkowsky is trying to be an analytic philosopher.)

Whether or not this kind of thing matters depends entirely on what you’re looking to get out of the book.  But in any event I don’t think it’s right, from what I’ve read so far, to frame this – as some have – as the book by the guy who’s done his intellectual history homework.

All Right Let’s Sort Many Worlds/TDT Out Then

reddragdiva:

philsandifer:

nostalgebraist:

philsandifer:

Here’s the relevant passages. The first is actually the paragraph right before the Newcomb/Prisoner’s Dilemma thing - it’s where I’m explaining the odd premises behind Roko’s Basilisk. (Which is in many ways just restating this bit of the RW article, though I went back to the primary sources for it.) 

The first and most straightforward weird premise is one that Yudkowsky establishes through some intense contortions of the many-worlds interpretation of quantum mechanics, which is a belief that one ought treat any copies of one’s self that exist in any possible future timelines not only as real, but as really being one’s self to the extent that one should actually care what happens to one’s hypothetical future duplicate. The means by which Yudkowsky reaches this are obscure; he explicitly cites it as one of those things that won’t make sense to the unenlightened masses. But the appeal of the conclusion is obvious: it allows the utopian vision to apply directly to the present day in spite of the profound and potentially insoluble technological barriers between us and strong AI.

That’s not TDT. The bit in the next paragraph where I talk about how “it is meaningfully possible to negotiate with a future superintelligent AI if it can predict your actions and you can predict its” is TDT, but that’s explicitly the second weird premise, i.e. not the same one as Many Worlds.

Meanwhile, here’s what I actually say about TDT many pages later, the paragraph after I say Newcomb’s Problem is silly. (Where I’ll be rephrasing the “only one correct answer” bit as per @nostalgebraist’s suggestion.)

The result of this is Timeless Decision Theory, which suggests that the prediction and the problem of picking a box are actually just two iterations of the same problem - an abstract computation roughly of the form “is this person going to pick one box or two.” Accordingly, instead of thinking about one’s actions in terms of “what am I going to do” one should think about it in terms of “what is the output of the abstract computation of what I’m going to do going to be.”

Nothing whatsoever about quantum mechanics.

I am at a loss for how the claim that I invoke Many-Worlds in my explanation of TDT can possibly be justified. 

Yes, I agree that you aren’t saying TDT is grounded in MWI.  I think @theungrumpablegrinch​ is just wrong here.

I should probably address this sort of thing when I write my own post(s) about the book, but since we’re on the topic now: I think your first paragraph here is still not quite right.  As far as I can tell (and I may be wrong), Roko doesn’t use the concept of timeless identity in his Basilisk at all, although he does in the Quantum Billionaire Trick.

The RationalWiki page adds a number of specifics to the Basilisk that aren’t actually stated in Roko’s rather terse post, or in his comments on that post.  For instance, he never actually says that the Basilisk will be punishing simulations of people (!).  Nor does Roko rely on any arguments in favor of “caring about” future/alternate/hypothetical versions of you that would otherwise seem remote.

Roko’s statement of the Basilisk in the original post – the paragraph starting with “in this vein” – really strikes me as pretty mundane.  It doesn’t depend on anything about simulations, and I’m not sure it really depends on TDT.  He seems to think it’s likely that the singularity will happen within our lifetimes, so there’s no reason the punishment has to happen to a copy, rather than simply to us when we’re older.

It’s pretty much analogous to worrying that something will be made retroactively illegal in the future – and that when the future law is enforced, it will only punish those who were worried about it earlier, but still did the illegal thing.  (The idea being that it only acted as deterrence against people who realized it was a possibility; the possibility of such a law had no effect whatsoever on people who never considered the concept at all.)

Such a law would be weird in the real world.  But not inconceivable – not with the NSA snooping on internet conversations, where people might discuss worries about things being made retroactively illegal.  (Ex post facto laws are unconstitutional in the U.S., but not in various other countries the NSA might share data with.)  There might never actually be a reason for a government to do this, but it’s certainly possible without any sci-fi stuff.

I realize the above isn’t the “standard” view, so take this with a grain of salt.

Right, and to some extent this becomes a matter of “which Basilisk,” as ultimately everyone who’s ensnared has their own bespoke monster, part of the point of the book being that Basilisks are actually very common. The archeology of precisely what Roko had in mind is tricky, and isn’t actually the same as what freaked Yudkowsky out so much nor as what disturbed any given person who posted on RW because LW was censoring the discussion. But none of these passages constitute a close-reading of Roko’s post, and the Standard Interpretation (i.e. the RW version), while possibly bulked out and expanded a bit from Roko’s original post via reference to related concepts in Yudkowskian thought, very much does the job the book actually needs it to, although I ultimately reconstruct it from primary sources instead of just citing RW because that’s just the sort of book it is. But further precision seems more likely to confuse than clarify for most readers.

timeless identity, and simulation of a person reconstructed via handwave, was standard fare in the lesswrong memeplex at the time. the simulation hypothesis (we are living in an ancestor simulation), which works on reconstruction entirely via handwave, was accepted as worth serious consideration.

from the basilisk post - which is written very densely in local jargon, so arguing over precisely what was meant could be endless - roko says “In your half, you can then create many independent rescue simulations of yourself up to August 2010 (or some other date), who then get rescued and sent to an optimized utopia.”

(note by the way there that in the first half he threatened hell, here he offers heaven.)

the totally friendly ai is presumed capable of creating “rescue simulations”. which can of course in its basilisk mode be torture simulations, as PeerInfinity immediately realises in the comments.

oh, this was posted to lw twelve hours after the basilisk post. note discussion of “rescue simulations” in the comments there too.

so no, i think that (even without a direct “i am talking about later reconstructions” or similar) the basilisk post, and the assumptions in the readers’ minds as to what it was about, are indeed about the basilisk ai recreating a simulation of you to send to hell or heaven, and to say otherwise i think you’d need positive evidence that roko did not subscribe to this particular lw trope.

Ah, no, Roko definitely did believe in rescue simulations and that’s definitely what freaked out PeerInfinity et. al.  I’m probably making far too much of this.

My point was that that the “talking to the God from the future” aspect of the Basilisk isn’t actually that weird or Yudkowskian at all – which matters here because that aspect is deep with potential resonances if we’re seeing this all as a horror story.

But then, even if that aspect doesn’t depend on specifically Yudkowsian premises, it’s definitely the sort of thing that fascinates him and LW – there’s a reason why “acausal blackmail” was an existing term that could be immediately brought to bear on Roko’s post.  So, resonance-wise, we’re still in the clear.

(via reddragdiva)

All Right Let’s Sort Many Worlds/TDT Out Then

philsandifer:

Here’s the relevant passages. The first is actually the paragraph right before the Newcomb/Prisoner’s Dilemma thing - it’s where I’m explaining the odd premises behind Roko’s Basilisk. (Which is in many ways just restating this bit of the RW article, though I went back to the primary sources for it.) 

The first and most straightforward weird premise is one that Yudkowsky establishes through some intense contortions of the many-worlds interpretation of quantum mechanics, which is a belief that one ought treat any copies of one’s self that exist in any possible future timelines not only as real, but as really being one’s self to the extent that one should actually care what happens to one’s hypothetical future duplicate. The means by which Yudkowsky reaches this are obscure; he explicitly cites it as one of those things that won’t make sense to the unenlightened masses. But the appeal of the conclusion is obvious: it allows the utopian vision to apply directly to the present day in spite of the profound and potentially insoluble technological barriers between us and strong AI.

That’s not TDT. The bit in the next paragraph where I talk about how “it is meaningfully possible to negotiate with a future superintelligent AI if it can predict your actions and you can predict its” is TDT, but that’s explicitly the second weird premise, i.e. not the same one as Many Worlds.

Meanwhile, here’s what I actually say about TDT many pages later, the paragraph after I say Newcomb’s Problem is silly. (Where I’ll be rephrasing the “only one correct answer” bit as per @nostalgebraist’s suggestion.)

The result of this is Timeless Decision Theory, which suggests that the prediction and the problem of picking a box are actually just two iterations of the same problem - an abstract computation roughly of the form “is this person going to pick one box or two.” Accordingly, instead of thinking about one’s actions in terms of “what am I going to do” one should think about it in terms of “what is the output of the abstract computation of what I’m going to do going to be.”

Nothing whatsoever about quantum mechanics.

I am at a loss for how the claim that I invoke Many-Worlds in my explanation of TDT can possibly be justified. 

Yes, I agree that you aren’t saying TDT is grounded in MWI.  I think @theungrumpablegrinch​ is just wrong here.

I should probably address this sort of thing when I write my own post(s) about the book, but since we’re on the topic now: I think your first paragraph here is still not quite right.  As far as I can tell (and I may be wrong), Roko doesn’t use the concept of timeless identity in his Basilisk at all, although he does in the Quantum Billionaire Trick.

The RationalWiki page adds a number of specifics to the Basilisk that aren’t actually stated in Roko’s rather terse post, or in his comments on that post.  For instance, he never actually says that the Basilisk will be punishing simulations of people (!).  Nor does Roko rely on any arguments in favor of “caring about” future/alternate/hypothetical versions of you that would otherwise seem remote.

Roko’s statement of the Basilisk in the original post – the paragraph starting with “in this vein” – really strikes me as pretty mundane.  It doesn’t depend on anything about simulations, and I’m not sure it really depends on TDT.  He seems to think it’s likely that the singularity will happen within our lifetimes, so there’s no reason the punishment has to happen to a copy, rather than simply to us when we’re older.

It’s pretty much analogous to worrying that something will be made retroactively illegal in the future – and that when the future law is enforced, it will only punish those who were worried about it earlier, but still did the illegal thing.  (The idea being that it only acted as deterrence against people who realized it was a possibility; the possibility of such a law had no effect whatsoever on people who never considered the concept at all.)

Such a law would be weird in the real world.  But not inconceivable – not with the NSA snooping on internet conversations, where people might discuss worries about things being made retroactively illegal.  (Ex post facto laws are unconstitutional in the U.S., but not in various other countries the NSA might share data with.)  There might never actually be a reason for a government to do this, but it’s certainly possible without any sci-fi stuff.

I realize the above isn’t the “standard” view, so take this with a grain of salt.

about that title, though

(this is a comparison of one thing to a vastly more important thing; the point is not to suggest any sort of proportionality, but to express my sense of one discursive context by bringing up another with some formal similarities)

The situation does strike me as a bit like writing a book titled This Year Will Be Tougher Than Last Year: The Steady Rise of the New Anti-Semitism, which weaves together discussion of Corbyn and several anti-semitic extremists, but then explicitly rejecting the “weird theory” that the book was intended to depict Corbyn as anti-semitic or as a supporter of anti-semites – since in fact you just included him because he’s the sort of fringe-contrarian-with-cultlike-fanbase political figure you’ve always been fascinated with, and as a “good guy” to pair with bad guys like Jean-Marie Le Pen, and anyway, it makes perfect sense to discuss him in the context of anti-semitism, since he goes way back with Ken Livingstone, etc.

The book is nonetheless widely viewed as a contribution to the current Labour antisemitism controversy, a bizarre misinterpretation that no one could have anticipated,