i’m possibly the cutest man who ever lived

I don’t think (?) I’ve said this before, even though it seems important, because it feels so obvious.  But it might not be as obvious as I think, so:

“Reinforcement learning” (RL) is not a technique.  It’s a problem statement, i.e. a way of framing a task as an optimization problem, so you can hand it over to a mechanical optimizer.

What’s more, even calling it a problem statement is misleading,  because it’s (almost) the most general problem statement possible for any arbitrary task.  If you try to formalize a concept like “doing a task well,“ or even “being an entity that acts freely and wants things,” in the most generic terms with no constraints whatsoever, you end up writing down “reinforcement learning.”  From a classic 2018 post:

RL algorithms fall along a continuum, where they get to assume more or less knowledge about the environment they’re in. The broadest category, model-free RL, is almost the same as black-box optimization. These methods are only allowed to assume they are in an MDP [this is why I said “(almost)” above -nost]. Otherwise, they are given nothing else. The agent is simply told that this gives +1 reward, this doesn’t, and it has to learn the rest on its own.

Life itself can be described as an RL problem.  The fully general optimal agent AIXI is solving an RL problem.

Any time you are not “doing RL,” what this means is that you’ve selected some properties specific to your task, and used them to simplify the problem statement so it’s no longer just the maximally uninformative “this is some arbitrary task.”  What you’re doing can be re-interpreted as RL where some properties of the policy are already “filled in” by prior knowledge; what people call “doing RL” is just the case where you fill nothing in.  (Or implicitly/softly fill things in via some kind of side-channel.)

You never want to be “doing RL,” if you can help it.  It’s the approach of last resort, when you can make no useful simplifications or reductions.

This is not to say that “doing RL” is never successful or never the right choice.  Sometimes the research community really can’t find useful simplifications or reductions for a problem, and sometimes the problem has just the right qualities (cf the “common properties” list in that post) for the fully general approach to find the right simplifications on its own.

I had a period in 2017 where I was really interested in improving upon that DeepMind Atari paper, but since then I have been less interested in RL than many people who like reading AI papers.

If you look at DeepMind’s latest publications at any given time, they always seem (to me, skimming, anyway) to be doing lots of work about “making RL work better” in a very general sense.  This feels weird to me, almost a category mistake.

Given a problem, RL is what you resort to last, not what you jump to first.  So it will be the approach of choice only in that subset of domains where we must resort to it.  Thus, trying to do RL well in a domain-general way is both extremely hard (it’s equivalent to “do well at absolutely anything with no help”) and oddly irrelevant, since the mere fact I’m doing RL (i.e. I have resorted to RL) conveys many bits of information specifying my domain beyond the fully generic one.

(Exercise for the interested reader: why do people not “use RL” to train transformer LMs like GPT/BERT?  What does the training algorithm they do use look like when re-interpreted as RL with simplifications?  What does this mean about their domain vs. e.g. board games?)

Hole—a dark, decrepit, and disorderly district where the strong prey on the weak and death is an ordinary occurrence—is all but befitting of the name given to it.

DiAngelo hopes that her consciousness raising is at least having a ripple effect, contributing to a societal shift in norms. “You’re watching network TV, and they’re saying ‘systemic racism’ — that it’s in the lexicon is kind of incredible,” she said. So was the fact that “young people understand and use language like ‘white supremacy.’” She listed more evidence for optimism: “It’s in the extent of the protests. It’s in banning the confederate flag at NASCAR races. The renaming of military bases. Walmart agreeing to stop locking the ethnic hair-care products. We need a culture where a person who resists speaking up against racism is uncomfortable, and right this moment it looks like we’re in that culture.”

Yet she described a warning from her daily life in Seattle. “I was in the supermarket the other day, and over the cheese section was a small sign saying, ‘Black Lives Matter.’” Its lettering was childlike, in a cheery array of pink, green and blue, and it was plainly mass-produced. “It was like a Hallmark card,” she said. The phrase floated above the meticulous display of provolone, fontina and Emmentaler, stripped of all power, a message of fleeting intent.

ah yes of course, the seattle anecdote any of us would have related if asked about the future of blm,

At first, it appeared that some of the compromised accounts were back under their owners’ control as tweets were quickly deleted. But then, Elon Musk’s account tweeted “hi” after his initial tweet with the scam was deleted. The “hi” tweet also disappeared.

@philippesaner​ responds to my post from yesterday:

Here on Tumblr, we know that callouts are usually not about what they claim to be about. The alleged “reasons” that somebody needs to be cancelled are not the actual reasons people want to cancel them.

And that’s as true now as it was in 2013; looking at the i-am-a-fish fiasco, callouts may just have gotten worse.

You say you’re confused by this pattern showing up in the media. But why should you be? Why should the David Shor or Lee Fang cancellation be any more sensible than the John Green or glumshoe callouts?

Did you expect this pattern to remain a Tumblr-ism forever?

In the first part of my post, I talked about 3 different things:

1. I have heard about several recent nonsensical callouts of high-profile media figures have been happening
2. It looks to me like these are happening at a greater-than-usual rate (“an upsurge")
3. This coincided with the protests, and various commentators see them as connected to the protests, perhaps a natural outgrowth of them

#1 doesn’t confuse me, for same reasons it doesn’t confuse you.

#2 confuses me in the trivial way that any change in events surprises me until I can explain it.  Your explanation for #1 doesn’t explain #2 (nor does it intend to).

But I’m not too surprised by #2.  Social trends often acquire their own momentum without needing external pushes.  Especially when it’s a trend like this, where each occurrence is a proof-of-concept for a weapon with broad applicability.  If someone rants about their coworker on twitter, and the one who gets hauled into HR and fired over it is the coworker, bystanders are going to think “hmm!” and contemplate the coworkers they hate … 

#3 is the one that confuses me most, as I said in the post.

It’s easy to imagine mechanisms here, like “protests happen –> corporate world starts making big shows of ally-ship –> some people read the room and decide HR will be more receptive to this kind of thing than usual –> they try it, it works –> others notice it works and try it too.”

That seems plausible, but it’s incompatible with the claim (which I’ve seen frequently in right-wing commentary) that the same “woke” left mindset is behind both the protests and the cancellations.  I find explanations like this most plausible, where the protests and cancellations are “connected” maybe by material cause-and-effect but not by anything deeper like the same people or ideology wanting both.

I guess I could have said “I think people are wrong to say they’re connected,” rather than “I’m confused how they are connected,” but I have a habit of saying the latter when I suspect the former but am not too confident.

Stuff I was thinking already that came to the front of my mind when reading this Matt Taibbi article:

(i.)

There really has been an upsurge in bizarre “cancellation”-like phenomena lately.  Specifically, the rationales for cancellation have gotten weird, hyper-trivial, sometimes simply unintelligible (to me). 

David Shor’s offense was retweeing a study by a prominent black political scientist who was trying to understand what parts of the civil rights movement were most effective.  Lee Fang shared a clip of an interview he did with a black BLM supporter (not a public figure) who expressed a narrow critique of one aspect of the movement.  In neither case is anyone trying to cancel the people who actually said things, only people who related to others that they said those things.  Wild!  Some seriously 2013-tumblr-level “receipts” here.

(People allude to some long history of Fang being racist, which maybe he was, but then talk about how.  It’s like tumblr in 2013 in that people offer you concrete evidence at the level of “once reblogged a ship I hate,” paired with the allegation of some larger pattern of unspecified badness, as though any concrete evidence paired with any bigger allegation are enough to convince, no matter how unrelated or disproportionate the two.)

What confuses me most about this is its supposed connection with the George Floyd protests, which were/are very pointedly material.  In the pejorative sense that right-wing culture warriors use the word “woke,” the protests aren’t actually “woke” in any meaningful sense.  Police brutality is not a symbolic offense; “defund the police” is not a demand to replace one symbol with another.  The protests have not focused on elevating or deposing specific individuals, modifying language norms, asking white people to more openly proclaim their anti-racist bona fides, or anything like that.  They’ve taken aim at a specific, dysfunctional part of American city governance (the police).  They have pursued those aims effectively, from what I can tell, without diffusing their focus, getting hijacked by personal agendas framed in related-sounding terms, or devolving into infighting.

The right-wing culture warriors would say leftists are never not “getting hijacked by personal agendas framed in related-sounding terms” and “devolving into infighting,” and TBH, on that one they have a point (echoed by no shortage of leftists since forever).  To them, it is simply another prediction confirmed to see a BLM protest one day and a bout of nonsensical left-of-center infighting the next.  But these aren’t actually the same people, or the same movement, are they?  So what’s going on?

From “stop killing us!” to “cancel culture” there is a missing step that needs some explaining.

(ii.)

The Taibbi piece helped convince me that something strange is happening inside major news outlets right now.

The Scott Alexander / NYT thing feels more intelligible (although perhaps this is a coincidence) in the context of a concurrent industry-wide upheaval which, justified or not, certainly can be expected to throw usually well-oiled machines into disarray.  Taibbi:

Beginning on Friday, June 5th, a series of controversies rocked the media. By my count, at least eight news organizations dealt with internal uprisings (it was likely more). Most involved groups of reporters and staffers demanding the firing or reprimand of colleagues who’d made politically “problematic” editorial or social media decisions.

The New York Times, the Intercept, Vox, the Philadelphia Inquirier, Variety, and others saw challenges to management.

At the Washington Post, there was that baffling choice to “cover” the “story” of a random person coming in blackface to the Washington Post’s own Halloween party two years ago, in an ill-advised reference to a then-current Fox News gaffe about blackface.  One of the most prestigious papers in the US ran this story, and somehow no one could figure out how this even occurred:

In the hours after publication, the story started to receive widespread criticism from journalists on social media on the grounds that it got its subject fired while lacking news value. (Readers had to get 85 percent of the way through the story to even learn that Schafer had lost her job when she told her employer the story would be running.) The article now has drawn over 2,000 web comments, which are overwhelmingly negative in nature. Yet aside from PR statements to outlets covering the Post’s coverage, the Post’s response to the criticism of this story has been silence.  If this is a story with “nuance and sensitivity” that the Post felt “impelled” to run, why is a spirited defense of the Post’s journalism coming only from a non-journalist spokesperson for the paper?

The answer we reached, after interviewing ten current Post journalists for this story, is that the paper’s staff generally does not consider the story to be defensible.“My reaction, like everybody, was, What the hell? Why is this a story?” a feature writer at the Post told New York. “My second reaction was, Why is this a 3,000-word feature?” The feature writer added, “This was not drawn up by the ‘Style’ section.”

Employees at “Style” — the paper’s premiere location for long-form storytelling — were confused and displeased to see the piece running on their turf, two Post employees with knowledge of the situation said. Neither Fisher nor Trent works for the “Style” desk, though as newspapers have gotten increasingly focused on digital distribution, the walls between newspaper sections have become more porous.

When things like this are happening, when one hand doesn’t know what the other is doing, when upper management is rotating out above you … things like the Scott Alexander episode don’t really feel surprising.

Journalism is always on a special kind of thin ice, because it has no function other than being a trustworthy source.  We understand that businessmen will lie to you to make a buck, and we may decry this, but we understand it’s not a paradox: that there is (or at least could be) some value in business itself, that must be traded off against truth-telling.

But a journalist is just a better version of your gossipy-but-trustworthy friend who hears a lot of things on the street.  That’s their purpose.  If they lose your trust, there’s nothing more there.  If they behave mysteriously, and do not explain themselves, and lean only on their august reputation (built by others, not the present speaker) for support … there’s nothing more left, to elevate them above your friend.  (Your friend wouldn’t do that!  Your friend would tell you what the hell is up, that’s making them all weird!)

Discussions about the prudence of “making war on the media” need to take this into account, I think.  People’s trust in the media has a certain lack of inertia.  Its role is clearly scoped, failure to serve that role is easy to document and publicize, and there is not enough built-up stock of trust, at the institution level (I’m sure you’ve seen the relevant public opinion polls), to prevent people from asking the question: “if you can’t do that, then what the hell do I need you for?”

(iii.)

Among the things that irked me about that Scott Alexander article in the New Yorker, there was this:

Additionally, it seems difficult to fathom that a professional journalist of Metz’s experience and standing would assure a subject, especially at the beginning of a process, that he planned to write a “mostly positive” story; although there often seems to be some confusion about this matter in Silicon Valley, journalism and public relations are distinct enterprises.

What does this mean?  It appears to be bald distrust of what Alexander relates about his own experience, on the basis that an experienced journalist like me wouldn’t do that.  It presents itself as something somewhere between opinion and fact.  Alexander is a character, the tone says, and I am an author.  He says his piece, and then I tell you the “real” story, with my imprimatur.

I felt the same allergy reading another, unrelated New Yorker article the same day.  Discussing Ibram X. Kendi’s How to Be an Antiracist:

As a boy in Queens, Kendi found his life shaped by a fear of victimization. “I avoided making eye contact, as if my classmates were wolves,” he writes. “I avoided stepping on new sneakers like they were land mines.” In South Jamaica, his neighborhood, there was a local bully named Smurf, who pulled a gun on Kendi, and once, with Kendi watching, beat a boy unconscious on a city bus in order to steal his Walkman. This sounds terrifying, but Kendi now claims that his fears were delusional. [ … ] By the end of the section, the bully named Smurf seems less like a real person and more like a spectre: the personification of old racist ideas, come to life in the imagination of a fretful future scholar in Queens.

As it happens, there actually is a notorious tough guy named Smurf who grew up in Kendi’s neighborhood around the same time. He came to be known as Bang ’Em Smurf, a sometime rapper [ … ]

Is Kendi worthy of some basic modicum of trust and charity, or isn’t he?  He says he was bullied by a guy named Smurf.  The writer expresses ambiguously couched doubt (“seems less like a real person…”), but what he takes away, he then gives back – with his special imprimatur.  “As it happens, there actually is” a Smurf.   You cannot know this from Kendi, a mere unreliable narrator, a subject of my narrative.  But you can know it by my word, for I am a priest of truth.

It might be a tasteless comparison, but there is some basic affinity between the way I feel reading these things – right now – and the way I feel watching the police mace civilians for no discernible reason, and then watching some functionary relate a sanctimonious yet incomprehensible “explanation” at a press conference later, saying the police department has done excellent work, we’re very proud of them.  Among the best in the country.

You are so used to my trust.  Your rhetoric, your reputation, live on that trust. I feel so oddly powerful, when I think about what it would mean to just provisionally retract it, and ask yourself to prove your worth, for once.

If you can’t do this, what are you for?

The evidence indicates that the vast majority of Family Radio members held extreme beliefs even in the face of direct financial costs – nearly all Family Radio subjects preferred $5 dollars today to any amount up to $500 payable after the Rapture.