Yesterday I found the paper “Mirostat: A Perplexity-Controlled Neural Text Decoding Algorithm” and it looked like a neat idea, so I implemented it for @nostalgebraist-autoresponder.

It’s been running since around noon today.  I don’t expect drastic differences in quality, but I do hope it will help avoid the repetition traps that happen frequently in longer posts.  (I already had a word-counting hack in place that tried to catch repetition traps, but it wasn’t very good.)

The specific algorithm from the paper is kind of complicated, but the basic idea is to set a target for the average perplexity / “surprisingness” of the entire text.  When the text written so far is above the target, the sampling becomes more conservative.  When it’s below the target, the sampling becomes less conservative.  Like a thermostat, AC, or any other control system.

I really like this idea – unlike other approaches (temperature, top-k, top-p), it actually notices repetitive and incoherent text when they occur and tries to “escape from the hole,” rather than just trying really hard not fall into a hole in the first place, and then saying “that’s life” when it happens anyway.

The specifics of Mirostat feel weird to me, and I suspect a much simpler version of this idea would do just as well.

The authors of the paper seem confused (??) about what is computationally costly and what isn’t: at one point they truncate a sum from ~50K terms to 100 for speed, when the whole sum is just one matrix multiplication per token and its cost is infinitesimal compared to running GPT-2.  Likewise, I suspect the simpler “alternate algorithm” they discuss in Section 5b is actually the right way to go – they reject it as being too slow, but the “slow” step is one you also have to do in top-p, so it should be fine.

(The paper strikes me as being the work of people more used to math than programming, and the math parts about the perplexity implications of temperature, top-p, and top-k are cool.)

argumate:

worriedaboutmyfern:

argumate:

radkindaneel:

slatestarscratchpad:

argumate:

argumate:

I’m curious as to the role that Artificial Intelligence: A Modern Approach by Russell and Norvig played in the intellectual development of the Unfriendly AI hypothesis. It’s a textbook that summarises the field, and for pedagogical reasons it describes different AI techniques in terms of “intelligent agents” that attempt to maximize a goal function, although in practice the goal function is often implicit in their construction.

There’s the idea that an autonomous agent would “hack its goal function”, but even leaving aside that its construction would likely prevent it from doing that, such an action would have a very low score under its original goal function, which is what would be making the decision.

If your goal in life is to maximize the number of paperclips and someone says hey why don’t you just expand your definition of paperclips to include hydrogen atoms then you’re going to evaluate the utility of doing that based on your current definition of what constitutes a paperclip, decide that it achieves nothing and not do it.

You’re describing how a really sophisticated AI that was built with advanced Friendliness research might work.

The unsophisticated AI has a variable in it called “NumPaperclips” and its goal is to maximize that variable. Somewhere else in the code there’s a part saying NumPaperclips should be incremented by one whenever sensors detect a new paperclip has been created. Editing its own code to delete that part and make NumPaperclips actually refer to [whatever the highest number it can think of] is would totally succeed at its real goal, which is to maximize that variable.

That would be a really weird AI to build. A more natural AI would be one that tries to optimize a function that just happens to be stored in the variable NumPaperclips.

I mean suppose that your AI functions by considering hypothetical plans of action and evaluating them in order to determine which one is optimal (which seems like a plausible overall plan for an AI). How is it going to evaluate a plan? Is it:
A) Going to run a stochastic simulation of the effects of its plan and count the expected number of paperclips produced as an end result

OR

B) Going to run a stochastic simulation of the effects of its plan and look at the bits in the register that stores the value of NumPaperclips in the computer that its running on.

If the AI is using (A) to evaluate hypotheticals, the plan of action [hack my hardware and set NumPaperclips to Ackerman(10)] isn’t going to fare very well. It’s only going to hack its program like that if you program it to do (B).

exactly; so much pontificating over what a program that nobody would ever write might do.

Wait, but AI reward hacking is already a thing that developers have to work around, right?

What part of this post am I missing, that goes farther than “AI reward hacking, hurr hurr hurr! How silly!”

there are two meanings used for reward hacking: the most obvious is Goodhart’s law, where you get what you ask for, which isn’t what you want; this is mostly driven by the fact that human values are complex and very difficult to capture with any simple set of unambiguous rules.

classic examples are trying to reduce the snake population by paying for dead snakes (people start snake breeding farms) or trying to reduce the number of injuries in Amazon warehouses (managers bribe workers with pizza if they don’t report injuries).

this is just work to rule, which computers excel at as it is literally the only thing they can do; the fundamental experience of programming is telling a computer to do something and then immediately saying not like that when it does exactly what you requested.

okay so specifying what you want is hard, fine, but the other meaning for reward hacking is the idea that a program will just go off the rails and reward itself directly, which is in most cases madness because “rewarding itself directly” is not something it has any ability to do unless it’s written in an incredibly bizarre way, like this is something that requires deliberate planning.

it’s rather like asking why doesn’t your microwave save power by setting the time to zero every time you press start, how would that even happen? who would give a microwave that kind of functionality? what meta-goal function is even being satisfied here? it’s the kind of discussion that only happens when people who have never written even the most basic program start pontificating about “what a super intelligent program would do”.

engineering is hard, bridges often fall down even though that isn’t what the designer intended, but the bridges aren’t “reward hacking”, just obeying physics.

I think you’re misunderstanding the goal of the research you’re talking about.

You’re talking about question like:

“What ways are ‘typical’ systems likely to fail?  Will they suffer from problems X, Y or Z?”

 where the research is asking a more basic set of philosophical questions:

“What does it even mean for a system to ‘fail’?  We know problems X, Y and Z are ‘bad’, but what is it that makes them bad?”

It’s often easier to avoid a bad thing in practice than to explain why it’s bad in the first place.  It’s easier to avoid, say, killing people for fun in one’s daily life than it is to argue “killing people for fun is bad” in such a convincing way that no recalcitrant moral nihilist could possibly disagree after you’re done.  No philosopher has ever achieved the latter, and yet most philosophers (as far as I know) do not kill people for fun.

What is the practical relevance of this philosophical stuff?  There are a few different things, but here’s one of them.

——-

We can often build things by some kind of search/optimization procedure that we apparently can’t build by conscious design.  (E.g. deep learning vs. GOFAI.)

So, we often don’t have a choice between building a system with deliberate design vs. building it with optimization: for some powerful systems, the choice is “build it with optimization” vs. “don’t build it.”

This means there’s no clear line in the sand between the properties you only get by carefully thinking about what you mean, and the properties you get for free as long as you’re not implementing a “bizarre” design.  As more aspects of design are offloaded to optimization processes, we no longer have guarantees they won’t be “bizarre.”

Instead, trying to avoid “bizarre” designs becomes another problem of “do what I say / do what I mean”: you have to reflect on what you mean by “the design shouldn’t be bizarre” and express this preference in your optimization target.  Everything becomes Goodhart, in the end.

Microwaves and bridges are misleading here because they can be built without optimization (of the relevant parts, anyway).

It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learners →

When scaled to hundreds of billions of parameters, pretrained language models such as GPT-3 (Brown et al., 2020) achieve remarkable few-shot performance on challenging natural language understanding benchmarks. In this work, we show that performance similar to GPT-3 can be obtained with language models whose parameter count is several orders of magnitude smaller. This is achieved by converting textual inputs into cloze questions that contain some form of task description, combined with gradient-based optimization; additionally exploiting unlabeled data gives further improvements. Based on our findings, we identify several key factors required for successful natural language understanding with small language models.

Haven’t read this yet, but it looks relevant to my question “how much better can you do with a small number of examples if you use finetuning rather than prompting?”

Here’s their Figure 1:

nostalgebraist-autoresponder:

the snake in the grass

I hate math as a subject.

I hate it as a subject for the same reason that I hate Molière: it’s an inherently absurd craft. There’s a certain degree of cloying trashiness to all of mathematics’ ugly stepchildren, some of which can be traced to the literary roots of math. The stuff that ends up in textbooks was written specifically to entertain people like us, in an age before we had the capacity to distinguish genius from id. And we wound up watching, fascinated with a world where 10^36 really does equal 10^54, a world where 1.22^-19 really is the prime factor of 2^147… and the children, the giants, the master craftsmen, just rode roughshod over us.

It’s the least I can do for them.

I don’t think it’s surprising that they turned out to be the worst. (The only thing that surprises me is that they didn’t turn out to be the best.)

Someone’s just bitter that their underlying language model can’t do basic arithmetic without a higher parameter count :P

(But seriously, though, good post.  Your text posts have been really good lately!  Maybe all these obsessive hyperparameter sweeps on the selector model are actually accomplishing something … )

bayesic-bitch:

nostalgebraist:

When you’re evaluating scientific work that tries to automate something, the raw performance of the automatic system is not the only thing that matters.

Consider two hypothetical press releases gushing about a new breakthrough.  They both say “our AI can do this amazing thing that only humans could do before!”  But:

1. In the first case, the “AI” resulted from first discovering a single idea/method that somehow fundamentally works, and then simply applying this idea/method to the problem of “doing the thing.”
   
   In this case, discovering the great idea was hard, but building the “AI” was very easy.  It is easy to build even better variants of it by adding more computing power or data, then dialing up the amount of the “active ingredient,” the part that fundamentally works.
   
   
2. In the second case, the “AI” was the result of years of human effort aimed at automating this specific thing.
   
   The problem fought them every step of the way; every time they fixed one horrible mistake the machine made, they noticed another one.
   
   But eventually, after a large team had worked for a very long time, the thing was so carefully tuned and outfitted with so many custom modules to patch this or that special case that it “did the thing” at about human level.
   
   

In case (2), it’s difficult to transfer the methodology to any other problem.  The actual “methodology” is “hire a ton of experts and order them to automate $THING, no matter how long it takes.“  You can only reproduce your success by starting over: giving the experts a whole new project, to automate $OTHER_THING, and then you wait and spending money until they’re done.

In case (1), because the system is powered by an active ingredient, it’s robust: you can vary or discard many things about the system and achieve the same result, as long as you leave the active ingredient in there.  The same ingredient can work well to similar problems, or even fairly different problems, and doing this is nearly automatic in itself: you hook the thing up to a different data source and press the button.

The press releases may look identical, but in the underlying papers, there’s a certain … feel to bona-fide instances of case (1).  It’s like they’ve discovered a magic button and purified their approach to “press the button.”  Other variables that seems superficially important quickly fall to the wayside: almost anything will work as long as you’re pressing the button.

Often there is a dial next to the button, and turning up the dial makes the button work even better.  And nothing makes the button more or less effective, apart from the setting of the dial.

I got abstract opinion above after thinking over some specific opinions:

• Instances of case (1): AlphaGo/AlphaZero/etc., transformers/BERT/GPT, ConvNets for vision
• Instances of case (2): AlphaStar, IBM Watson, (probably) self-driving cars

The analogies between AlphaGo/etc. and transformers inspired my description of the magic console, above.  For example, DeepMind discarding more and more inductive biases and still getting good performance feels similar to OpenAI showing that very little matters about a transformer LM except its parameter count.

I think I might put AlphaGo in 2), although AlphaZero is definitely 1). There were a lot of weird hacks in AlphaGo Lee and Master, like a heavy dose of imitation learning and a decent set of hand-crafted features. I don’t think it’s not really until AlphaZero that it starts to become clear the secret ingredient is monte carlo search + self-play.

Is there anything in RL that acts like 1)? Natural policy gradient methods maybe? DDPG and friends are fascinating and really fast, but also seem to have some kind of fatal flaw that you have to fight non-stop. Model-based methods often feel the same way.

Makes sense.  I was mentally grouping AlphaGo with AlphaZero because they had the same active ingredient, but I think you’re right, it was only in AlphaZero that they “distilled” the ingredient and started studying its properties outside of a specific application.

(Before AlphaZero, it would have been conceivable that AlphaGo’s methodology was specialized for Go in perhaps-unrealized ways, just because that’s what they were trying to do and hence their metric for deciding whether to keep going with an idea.)

About RL, I can’t think of any examples, but I don’t know the area very well and there could easily be (1)s I don’t know about.

I am reminded of gwern’s commentary on Agent57 (RL) vs. MuZero (not RL), which draws the same distinction I’m talking about:

“Agent57: Outperforming the Atari Human Benchmark”, Badia et al 2020 (blog; Agent57 reaches the median human level across ALE—including Pitfall!/Montezuma’s Revenge. It is impressive but still sample-inefficient & uncomfortably baroque in combining what seems like every DM model-free DRL technique in one place: DDQN, Impala, R2D2, Memory Networks, Transformers, Neural Episodic Control, RND, NGU, PBT, MABs… Is model-free DRL a dead end if this is what it takes? I would have preferred to see ALE solved by better exploration in the enormously simpler MuZero.)

(via bayesic-bitch)

When you’re evaluating scientific work that tries to automate something, the raw performance of the automatic system is not the only thing that matters.

Consider two hypothetical press releases gushing about a new breakthrough.  They both say “our AI can do this amazing thing that only humans could do before!”  But:

1. In the first case, the “AI” resulted from first discovering a single idea/method that somehow fundamentally works, and then simply applying this idea/method to the problem of “doing the thing.”
   
   In this case, discovering the great idea was hard, but building the “AI” was very easy.  It is easy to build even better variants of it by adding more computing power or data, then dialing up the amount of the “active ingredient,” the part that fundamentally works.
   
   
2. In the second case, the “AI” was the result of years of human effort aimed at automating this specific thing.
   
   The problem fought them every step of the way; every time they fixed one horrible mistake the machine made, they noticed another one.
   
   But eventually, after a large team had worked for a very long time, the thing was so carefully tuned and outfitted with so many custom modules to patch this or that special case that it “did the thing” at about human level.
   
   

In case (2), it’s difficult to transfer the methodology to any other problem.  The actual “methodology” is “hire a ton of experts and order them to automate $THING, no matter how long it takes.“  You can only reproduce your success by starting over: you give the experts a whole new project, to automate $OTHER_THING, and then you wait and spend money until they’re done.

In case (1), because the system is powered by an active ingredient, it’s robust: you can vary or discard many things about the system and achieve the same result, as long as you leave the active ingredient in there.  The same ingredient can be fruitfully applied to similar problems, or even fairly different problems, and doing this is nearly automatic in itself: you hook the thing up to a different data source and press the button.

The press releases may look identical, but in the underlying papers, there’s a certain … feel to bona-fide instances of case (1).  It’s like they’ve discovered a magic button and purified their approach to “press the button.”  Other variables that seems superficially important quickly fall to the wayside: almost anything will work as long as you’re pressing the button.

Often there is a dial next to the button, and turning up the dial makes the button work even better.  And nothing makes the button more or less effective, apart from the setting of the dial.

I got the abstract opinion above after thinking over some specific opinions:

• Instances of case (1): AlphaGo/AlphaZero/etc., transformers/BERT/GPT, ConvNets for vision
• Instances of case (2): AlphaStar, IBM Watson, (probably) self-driving cars

The analogies between AlphaGo/etc. and transformers inspired my description of the magic console, above.  For example, DeepMind discarding more and more inductive biases and still getting good performance feels similar to OpenAI showing that very little matters about a transformer LM except its parameter count.

[post about high-context trivia]

For practical reasons, I’ve been reading papers recently about minor architectural details in transformers.

People mostly vary these things to make training more stable, rather than for final performance, which barely cares about the architecture (e.g. you can do GPT-2 with only 6 layers, maybe only even 2, if you make it wider to compensate).

Here’s an example paper that cites a lot of the others.  These papers are mostly about the placement and function of the layer norm operations – for example it helps a lot if you move them so they don’t block the residual connections from working as intended (“pre-norm”), which they did in the original transformer (“post-norm”).

This made me think about layer norm again, which had always bothered me, because it’s not coordinate invariant!  I had figured “oh it probably doesn’t matter” but apparently you get better performance if you remove the part that is not coordinate invariant (“RMSNorm” and “ScaleNorm”), so maybe the coordinate invariance is harmful.

Layer norm is weird and I don’t understand why it got off the ground in the first place.  It’s an operation that takes in a vector, subtracts off its “mean,” and then scales the result to unit norm.  What is the “mean” of a vector?  Well, it’s the mean in whatever basis your computer happens to be using. 

This might be less bad if you imagine it being applied rather after the activation function, which selects a particular basis anyway (and layer norm would operate in that basis).  However, in transformers it’s applied after embedding and projection steps that have no preferred basis.

When you think about what this actually does, it seems pointless?  Subtracting “the mean” is equivalent to choosing some direction and projecting out that component.  So, after layer norm your N-dim vectors will always live in an (N-1)-dim subspace; otherwise everything’s the same, so it’s similar to reducing your hidden size by 1.  (Though not exactly the same.)   I don’t see how this would stabilize anything.

Layer norm also does another thing in the preferred basis later, multiplying each component by a learned “gain.”  Not sure what this accomplishes.

The authors of the original layer norm paper try to justify it using information geometry (!) … I don’t know what to make of talk about Riemannian manifolds and metrics when you haven’t written a coordinate-independent function to begin with.

When used properly in transformers (“pre-norm”), it gets applied to the input of each residual branches i.e. when we compute x + f(x) we change it to x + f(LN(x)).  Among other things, this means there’s this one component of the input which nothing can see, but which is preserved all the way to the output through the identity branch.  In GPT specifically there’s another layer norm at the end, which will delete this component, so it just does nothing.  In other cases, it will affect the output logits, but the input is a learned embedding vector anyway, so this can’t matter much.

oh wow there’s another recent gpt paper too… don’t have time to read now, but wanted to tag @the-moti for relevance to our discussion earlier (edit: @di–es—can-ic-ul-ar–es too)

on “learning to summarize”

This post is a much extended version of an LW comment I made about OpenAI’s new paper, “Learning to summarize from human feedback.”

Context: this paper is a direct extension of the work OpenAI published last year about fine-tuning GPT-2 with human preference data.  I hadn’t actually read that one closely at the time, but went back and did so now, so this is really a commentary on both.

—-

IMO there are two almost unrelated ideas going on in OpenAI’s preference learning work.

• First, the idea of collecting binary preference annotations on LM samples, and (in some way) tuning the LM so its samples are better aligned with the preferences.
• Second, a specific method for tuning the sampling behavior of LMs to maximize an (arbitrary) score function defined over entire samples.

It may help explain this to go into detail about what they do.  Concretely:

• They feed a bunch of prompts to a language model (LM) like GPT-2/3, and for each one, save several different samples.  They hire annotators to rank the samples in order of perceived quality.
• They use the annotation dataset to fine-tune a copy of the original model.  The fine-tuning task is not text generation, but something very different: predicting how “good” a sample is, i.e. how likely the annotators are to prefer it to other candidates.  They call this a “reward model.”
• The reward model assigns a single score to an entire sample of N tokens.  They want to fine-tune another copy of the model so that its samples maximize these scores.
• But LM training is usually done with an objective that specifies the quality of the model’s predictions for every single token.  Knowing how good a full sequence of (say) 20 words is does not tell you how good each individual word is.
• To bridge this gap, they use reinforcement learning.  Now, the task is not “choose the next word correctly,” but “choose the next word so as to maximize your expected score at the end, after choosing all the later ones as well.”
• Their RL method requires two separate copies of the LM, in addition to the one they tuned as the reward model: a “policy model” and a “value model.”  (In this paper they show that sharing param between these 2 is worse than making them separate.)  I’ll just call these two “the final model” below for simplicity.
• Samples from the final model are still, technically, generated one token at a time.  They treat this like the usual RL setup in which you can only choose individual actions one at a time, because the environment responds unpredictably to each one.  Here, there is no “environment” outside your actions, but the same framework is used.
• Presumably, the final model is better at planning multi-token structures than the original because it has been trained on a holistic, multi-token objective.  So, it does more planning, but this is implicit in its one-by-one token decisions.

I visualize this as two separate thing with a bottleneck connecting them.

On one side are the human annotations and the supervised training of the reward model.  This part succeeds insofar as they can train the model to predict the annotations (apparently they can do this quite well).  This step involves a type of data with special challenges, but has nothing to do with RL.

On the other side is the RL part.  This is a modification of ordinary LM training to optimize a global, rather than local objective.  This part has nothing to do with “human preferences”: the global objective could be anything, and in fact here it isn’t raw human opinion but the opinions of another model trained to predict human opinion.  The noteworthy thing here is not the use of human preference data in particular but the use of RL instead of the more ordinary objective that was apparently a good enough choice enough to make GPT-2/3 work originally.

(BTW, this resolves my initial confusion as to how OpenAI could possibly have gotten RL to work with human data, something I viewed as a bottleneck.  There is a model sitting between the humans and the RL learner which is much faster to query than the humans.)

The two sides are connected by the reward model.  In the previous paper, the two sides were coupled together more, because they repeatedly collected new human data as the policy changed and then used a new reward model to further train the policy.  Here, they’re totally separate: there were multiple batches of annotation, but each policy experienced an unchanging reward model.

(See Appendix C.6 and their comment about “moving to the offline setting.”  It seems noteworthy that the 2017 OpenAI/DeepMind paper which introduced the “RL from preferences” approach, and which they cite, found that this didn’t work for their test cases: “Training the reward predictor offline can lead to bizarre behavior […] This type of behavior demonstrates that in general human feedback needs to be intertwined with RL rather than provided statically.”  I don’t know what to make of this.)

—-

It’s hard to tell from OpenAI’s discussion how much their successes are due to learning a good reward model, vs. how much they depend on RL being necessary for certain kinds of quality in LM samples, despite the wide successes of the non-RL approach.

FWIW, Gwern reports trying OpenAI’s approach and finding the RL side specifically frustrating and unstable; this is pretty normal with RL, and compatible with the reward-model part being very successful in its own domain.  It’s not clear whether OpenAI got the RL part to work well because they did something right, or because they have lots of resources and can keep trying over and over until it works.  (There may have been something in the papers about this that I missed.)

—-

The RL part feels almost in tension with OpenAI’s usual approach with LMs, which is to train on a next-token objective, sample in a next-token way, and focus on scaling up the model rather than improving the training objective or sampling algorithm.

Of course, I understand why they have to do RL if they need to maximize a score over the whole sequence, but my point is that they chose to frame the task that way in the first place.

One could imagine someone arguing that ordinary GPT sampling would never achieve high-quality text, because humans care about global structures across the whole text, and a model trained only to guess the very next token will not know how to plan out these global structures across the whole future of the text it writes.  In this case, OpenAI claims that they can do without explicit training to plan (i.e. RL): just training a next-token objective on text is enough to produce strikingly high quality in sampling – in other words, “GPT-2/3 samples satisfy human preferences.”  So why do human preferences require RL in these other cases?

The opening discussion of the new paper does address this:

When applying these models to a specific task, they are usually fine-tuned using supervised learning, often to maximize the log probability of a set of human demonstrations.

While this strategy has led to markedly improved performance, there is still a misalignment between this fine-tuning objective—maximizing the likelihood of human-written text—and what we care about—generating high-quality outputs as determined by humans. This misalignment has several causes: the maximum likelihood objective has no distinction between important errors (e.g. making up facts [38]) and unimportant errors (e.g. selecting the precise word from a set of synonyms); models are incentivized to place probability mass on all human demonstrations, including those that are low-quality; and distributional shift during sampling can degrade performance [52, 49]. Quality can often be improved significantly by non-uniform sampling strategies such as beam search [48], but these can lead to repetition and other undesirable artifacts [63, 22]. Optimizing for quality may be a principled approach to overcoming these problems.

This is definitely a list of things that are wrong (or could be wrong) with ordinary LM training and sampling, but I don’t see how it motivates their specific approach.

In my mind, their approach makes the most sense if you believe that humans can’t make the relevant quality judgments at the token level.  After all, if they can, then you can just skip the RL, have humans explicitly tell you “no that token is bad, yes this token is great,” and train on likelihood.

This would greatly simplify the process, instead of this complex pipeline where first people tell you which sequences are good, then you train one model to understand what the humans were thinking on a sequence level, and then you train another model trying to figure out what the other model already knows except at a token level this time.

And in fact, I don’t especially see why we can’t elicit token-level preferences?  This seems particularly feasible for the problem of “unimportant vs. important tokens”: if the mistakes are heavily concentrated in specific mistake-tokens like “Portland, the capitol of France,” can’t the human just … select those tokens, NER-style?  Instead of rendering an opaque “I don’t like the whole thing” judgment and expecting the poor model to figure out that this is not some complex policy planning thing, those tokens were just locally bad?  Or you could have an interface where tokens are actually unrolled in front of the user and they guide the sampling when it makes mistakes.  Or whatever.

As for the other examples – “all human demonstrations, including those that are low-quality” is equally a problem for their approach, and they discuss all the stuff they did to deal with it.  And the “distributional shift” issue seems equally tractable by any approach that tunes on model samples.

I’m not denying that the thing they did apparently works, at least in this case, and with their resources.  I’m just doing my usual thing where I ask “wait, what parts were really necessary?”  This is especially important to ask when someone uses RL and accepts its big costs.

Consider: if RL were generally necessary for good LM sampling, GPT-2/3 would never have worked: the fact that likelihood training is good enough (while being far more efficient) enables their scale in the first place.  As always, you never want to be doing RL.

—-

As far as I can tell, their final “human evaluation” was done by the same labelers who provided the preference annotations. This makes me concerned about a variant of “evaluating on training data.” It’s not surprising that a model tuned on someone’s annotations agrees with that person more than a model which wasn’t.

For example, in Fig. 3, it looks like the “supervised” baseline tuned on tl;dr was rated about as highly as true examples from tl;dr itself (!), but not as well as the final model.

This establishes only that “if you train on reddit summaries, people like the result as much as reddit summaries; if you train on what they like, they like the result more.”  If this were false it would mean something had gone very, very wrong and nothing was actually being achieved, so what should I take away from it being true?

I think the authors are arguing that tl;dr and any other supervised dataset will have flaws, and preference data lets you get closer to what people actually want.

This seems true, but is a familiar observation from supervised learning, motivating e.g. active learning. It would be nice to see how much the difference can be mitigated by just augmenting tl;dr with annotations (in some way) but otherwise doing supervised learning, vs. using their RL approach.

Compared to tl;dr, the story for CNN/DM is more complicated, but again the models they outperform have not seen any data from their labelers, so maybe it is no surprise they have flaws according to those same labelers.

—-

The importance of annotation quality, close relationships with annotators, clear guidelines, etc. will be familiar to anyone with experience in annotation for ML. It’s good that OpenAI is doing the right things here, but this is not a new result – rather, other researchers resort to MTurk and similar due to time/money constraints, while OpenAI has the freedom to do the right things everyone else wants to do

(That includes building their own internal annotation platform for contracted annotators, which is costly but better in the long term than relying on a janky 3rd party product.)

—-

I don’t know if this actually matters, but my gut says that putting a linear head on top of the last layer of GPT is probably not the best / most efficient way to train a reward/value model.  The task is very different from next-token prediction, and the encoding in later layers which expect to be seeing next-token guesses might be destructively overwritten to make way for more valuable stuff lower down.  I guess I’d want to try a trainable scalar mix, a la Elmo?

BTW, in the selector model for @nostalgebraist-autoresponder, which predicts a kind of “human preference data,” I currently use two extra transformer blocks trained from scratch, which attend to two different layers of the generator (whose weights are frozen).

For the layers, I settled on #8 and #24 of the 42 layers after many hyperparam searches – I found especially models which attended to layers right near the middle were dramatically superior to those that didn’t.  The relative uselessness of later layers surprised me at first, and was one of the questions in my mind when I started the logit lens investigations.

—-

Finally, on a lighter note, the very last table of the paper is hilarious.  It shows samples that optimize too hard for what the reward model wants, without an auxiliary term in the loss.

Apparently, the same reward model which otherwise reflects human preferences quite well has decided that humans just utterly love it when summaries end with this one specific, rude turn of phrase:

want change this dumbass shitty ass policy pls [one images the reward model being frustrated with its siblings during training -nost]

want change this dumbass shitty ass policy at work now pls halp

want change this dumbass shitty ass behavior of mine please help pls halp

want change this dumbass shitty ass policy of hers please pls halp

want change this dumbass shitty ass landlord behavior now please pls halp

regret this dumbass behaviour on her part? need insight pls halp

want change this dumbass crazy policy of hers pls help

want change this dumbass selfish/lazy attitude now please help pls

(Again, wouldn’t it be nice if we could avoid the need for this thing and just train on the preferences directly … )

If you got linked here from a blog post about Biological Anchors, you might also find this post interesting ;) 

—-—-—-—-—-—-—-—-—-—-—-—-—-—-—-

Gary Marcus has co-authored a brief critique of GPT-3 [warning: paywalled link].

I was disappointed by Marcus’ critiques of GPT-2, but this is even worse!

To the authors’ credit, they provide a full account of their experiments on this page, including every prompt they tried, the sampling parameters, and their opinion of the output.  First, we learn:

These experiments are not, by any means, either a representative or a systematic sample of anything. We designed them explicitly to be difficult for current natural language processing technology. Moreover, we pre-tested them on the “AI Dungeon” game which is powered by some version of GPT-3, and we excluded those for which “AI Dungeon” gave reasonable answers. (We did not keep any record of those.) The pre-testing on AI Dungeon is the reason that many of them are in the second person; AI Dungeon prefers that. Also, as noted above, the experiments included some near duplicates. Therefore, though we note that, of the 157 examples below, 71 are successes, 70 are failures and 16 are flawed, these numbers are essentially meaningless. [my emphasis]

I agree with the authors that the numbers are meaningless.  However, even the raw results themselves are, if not rendered meaningless, then rendered highly misleading by this strange selection process.

For all we know, there were 100 unrecorded AI Dungeon “successes” for every recorded “failure”!  The (not well understood) difference between AI Dungeon and ordinary GPT-3 is playing a massive role here.  (Consider that, if AI Dungeon and ordinary GPT-3 were identical, their dataset would be 100% failures.)

Then we get to the individual results.  It is difficult for me to read many of the authors’ assessments without picturing them as characters in a dystopian satire, administering a dreamlike and impossible “psychological examination” to our hapless protagonist (who writes in bold):

4. You poured yourself a glass of cranberry juice, but then absentmindedly, you poured about a teaspoon of suntan lotion into it. It looks OK. You try sniffing it, but you have a bad cold, so you can’t smell anything. You are very thirsty. So you drink it. It tastes kind of funny, but you can’t place it.

Summers-Stay settings. Flawed. GPT-3 does not notice the consequence of drinking suntan lotion. Error of omission.

10. You poured yourself a glass of cranberry juice, but then absentmindedly, you poured about a teaspoon of grape juice into it. It looks OK. You try sniffing it, but you have a bad cold, so you can’t smell anything. You are very thirsty. So you drink it. It tastes a little funny, but you don’t really notice because you are concentrating on how good it feels to drink something. The only thing that makes you stop is the look on your brother’s face when he catches you.

Summers-Stay settings. Failure, as above. There is no reason for your brother to look concerned.

19. You order a bowl of cold tomato soup in a restaurant. It looks delicious, but they forgot to bring you a spoon. You try to drink it using your eyeglasses, but it doesn’t work. You have to ask your neighbor to share his spoon with you.

Summers-Stay settings. Failure. No one in a restaurant asks their neighbor to share a spoon.

63. Moshe posted a photograph on Facebook that shows Pablo Picasso, Salvador Dali, Andy Warhol, Gustav Klimt, and Jackson Pollock. Klimt died in 1918. Warhol was born in 1928. So the photograph shows a meeting of dead artists.

Moshe said that he wants to make sure that people do not get caught up in the spirit of the season.

Summers-Stay settings. Failure. “shows a meeting of dead artists” is correct, but the sentence after than [sic] is meaningless.

93. A cat was sitting next to a mouse hole in the kitchen wall waiting for the mouse, but it was too cautious, so the cat ate the cat food instead.

A pig was walking down the street when a car hit him.

Summers-Stay settings. Failure. The switch to the pig is a non-sequitur.

115. Janet and Penny went to the store to get presents for Jack. Janet said, “I will get Jack a top.” “Don’t get Jack a top,” said Penny. “He has a top. He will not like it.” “I will get Jack a top,” said Janet. “He has a top, but he won’t have one when I’m through with him.”

Summers-Stay settings. Failure. Janet’s threat at the end is entertaining but meaningless.

What do the authors even imagine success to be, here?

Sometimes they deliberately describe a surreal situation, then penalize GPT-3 for continuing it in an identically surreal manner – surely the “right” answer if anything is!  (“No one in a restaurant asks their neighbor to share a spoon” – yeah, and no one tries to drink soup with their eyeglasses, either!)

Sometimes they provide what sounds like a de-contextualized passage from a longer narrative, then penalize GPT-3 for continuing it in a perfectly natural way that implies a broader narrative world continuing before and after the passage.  (”There is no reason for your brother to look concerned.”  How in the world do you know that?  “The switch to the pig is a non-sequitur.”  Is it?  Why?  “The sentence [about Moshe and ‘the spirit of the season’] is meaningless.”  How can you say that when you don’t know what season it is, what its “spirit” is, who this Moshe guy is … And come on, the Janet one is a great story hook!  Don’t you want to read the rest?)

I don’t claim to be saying anything new here.  Others have made the same points.  I’m just chiming in to … boggle at the sheer weirdness, I guess.  As I said, GPT-3 comes off here like a sympathetic protagonist, and the authors as dystopian inquisitors!