This obviously raises two different questions: 1. Why did you think that no one would use few-shot learning in practice? 2. Why did other people think people would use few-shot learning in practice?
I would be interested in hearing your thoughts on these two points.
—
Thanks for asking!
First of all, I want to emphasize that the GPT-3 paper was not about few-shot GPT-3 as a practical technology.
(This is important, because the paper is the one large body of quantitative evidence we have on few-shot GPT-3 performance.)
This is not just my take on it: before the OpenAI API was announced, allthe discussion I saw took for granted that we were talking about a scientific finding and its broader implications. I didn’t see any commentator whose main takeaway was “wow, if I could do this few-shot thing right now, I could build amazing projects with it.”
Indeed, a commontheme in critical commentary on my post was that I was too focused on whether few-shot was useful right now with this specific model, whereas the critical commentators were more focused on the implications for even larger models, the confirmation of scaling laws over a new parameter regime, or the illustration-in-principle of a kind of meta-learning. Gwern’s May newsletter is another illustrative primary source for the focus of the discussion in this brief “pre-API” period. (The API was announced on June 11.)
As I read it (perhaps benefitting from hindsight and discussion), the main points of the paper were
(1) bigger models are better at zero/few-shot (i.e. that result from the GPT-2 paper holds over a larger scale),
(2) more “shots” are better when you’re doing zero/few-shot,
(3) there is an interaction effect between 1+2, where larger models benefit more from additional “shots,”
(4) this could actually become a practical approach (even the dominant approach) in the future, as illustrated by the example of a very large model which achieves competitive results with few-shot on some tasks
The paper did not try to optimize its prompts – indeed its results are already being improved upon by API acolytes – and it didn’t say anything about techniques that will be common in any application, like composing together several few-shot “functions.” It didn’t talk about speed/latency, or what kind of compute backend could serve many users with a guaranteed SLA, or how many few-shot “function” evaluations per user-facing output would be needed in various use cases and whether the accumulated latency would be tolerable. (See this post on these practical issues.)
It was more of a proof of concept, and much of that concept was about scaling rather than this particular model.
—
So I’d argue that right now, the ball is in the few-shot-users’ court. Their approach might work – I’m not saying it couldn’t!
In their favor: there is plenty of room to further optimize the prompts, explore their composability, etc.
On the other hand, there is no body of evidence saying this actually works. OpenAI wrote a long paper with many numbers and graphs, but that paper wasn’t about whether their API was actually a good idea. (That is not a criticism of the paper, just a clarification of its relevance to people wondering whether they should use the API.)
This is a totally new style of machine learning, with little prior art, running on a mysterious and unproven compute backend. Caveat emptor!
—
Anyway, on to more conceptual matters.
The biggest advantages I see in few-shot learning are
(+1) broad accessibility (just type English text) and ability to quickly iterate on ideas
(+2) ability to quickly define arbitrary NLP “functions” (answer a factual question, tag POS / sentiment / intent, etc … the sky’s the limit), and compose them together, without incurring the memory cost of a new fine-tuned model per function
What could really impress me is (+2). IME, it’s not really that costly to train new high-quality models: you can finetune BERT on a regular laptop with no GPU (although it takes hours), and on ordinary cloud GPU instances you can finetune BERT in like 15 minutes.
The real cost is keeping around an entire finetuned model (~1.3GB for BERT-large) for each individual NLP operation you want to perform, and holding them all in memory at runtime.
The GPT-3 approach effectively trades this memory cost for a time cost. You use a single very large model, which you hope already contains every function you will ever want to compute. A function definition in terms of this model doesn’t take a gigabyte to store, it just takes a tiny snippet of text/code, so you can store tons of them. On the other hand, evaluating each one requires running the big model, which is slower than the task-specific models would have been.
So storage no longer scales badly with the number of operations you define. However, latency still does, and latency per call is now much larger, so this might end up being as much of a constraint. The exact numbers – not well understood at this time – are crucial: in real life the difference between 0.001 seconds, 0.1 seconds, 1 second, and 10 seconds will make or break your project.
—
As for the potential downsides of few-shot learning, there are many, and the following probably excludes some things I’ve thought of and then forgotten:
(-1) The aforementioned potential for deal-breaking slowness.
(-2) You can only provide a very small amount of information defining your task, limited by context window size.
The fact that more “shots” are better arguably compounds the problem, since you face a tradeoff between providing more examples of the same thing and providing examples that define a more specific thing.
The extent to which this matters depends a lot on the task. It’s a complete blocker for many creative applications which require imitating many nuances of a particular text type not well represented in the training corpus.
For example, I could never do @nostalgebraist-autoresponder with few-shot: my finetuned GPT-2 model knows all sorts of things about my writing style, topic range, opinions, etc. from seeing ~3.65 million tokens of my writing, whereas few-shot you can only identify a style via ~2 thousand tokens and hope that’s enough to dredge the rest up from the prior learned in training. (I don’t know if my blog was in the train corpus; if it wasn’t, we’re totally screwed.)
I had expected AI Dungeon would face the same problem, and was confused that they were early GPT-3 adopters. But it turns out they actually fine-tuned (!!!!), which resolves my confusion … and means the first real, exciting GPT-3 application out there isn’t actually a demonstration of the power of few-shot but in fact the opposite.
With somewhat less confidence, I expect this to be a blocker for specialized-domain applications like medicine and code. The relevant knowledge may well have been present in the train corpus, but with so few bits of context, you may not be able to overcome the overall prior learned from the whole train distribution and “zoom in” to the highly specialized subset you need.
(-3) Unlike supervised learning, there’s no built-in mechanism where you continually improve as your application passively gathers data during usage.
I expect this to a be a big issue in commercial applications. Often, a company is OK accepting a model that isn’t great at the start, if it has a mechanism for self-improvement without much human intervention.
If you do supervised learning on data generated by your product, you get this for free. With few-shot, you can perhaps contrive ways to feed in segments of data across different calls, but from the model’s perspective, no data set bigger than 2048 tokens “exists” in the same world at once.
(-4) Suffers a worse form of the ubiquitous ML problem that “you get exactly what you asked for.”
In supervised learning, your model will avoid doing the hard thing you want if it can find easy, dumb heuristics that still work on your train set. This is bad, but at least it can be identified, carefully studied (what was the data/objective? how can they be gamed?), and mitigated with better data and objectives.
With few-shot, you’re no longer asking an arbitrary query and receiving, from a devious genie, the response you deserve. Instead, you’re constrained to ask queries of a particular form: “what is the next token, assuming some complicated prior distributed from sub-sampled Common Crawl + WebText + etc.?”
In supervised learning, when your query is being gamed, you can go back and patch it in arbitrary ways. The lower bound on this process comes only from your skill and patience. In few-shot, you are fundamentally lower-bounded by the extent to which the thing you really want can be expressed as next-token prediction over that complicated prior. You can try different prompts, but ultimately you might run into a fundamental bound here that is prohibitively far from zero. No body of research exists to establish how bad this effect will be in typical practice.
I’m somewhat less confident of this point: the rich priors you get out of a large pretrained LM will naturally help push things in the direction of outcomes that make linguistic/conceptual sense, and expressing queries in natural language might add to that advantage. However, few-shot does introducea new gap between the queries you want to ask and the ones you’re able to express, and this new gap could be problematic.
(-5) Provides a tiny window into a huge number of learned parameters.
GPT-3 is a massive model which, in each call, generates many intermediate activations of vast dimensionality. The model is pre-trained by supervision on a tiny subset of these, which specify probability distributions over next-tokens.
The few-shot approach makes the gamble that this same tiny subset is all the user will need for applications. It’s not clear that this is the right thing to do with a large model – for all we know, it might even be the case that it is more suboptimal the larger your model is.
This point is straying a bit from the central topic, since I’m not arguing that this makes GPT-3 few-shot (im)practical, just suboptimal relative to what might be possible. However, it does seem like a significant impoverishment: instead of the flexibility of leveraging immense high-dimensional knowledge however you see fit, as in the original GPT, BERT, adapters, etc., you get even immenser and higher-dimensional knowledge … presented through a tiny low-dimensional pinhole aperture.
—
The main reason I initially thought “no one would use few-shot learning like this” was the superior generalizationperformance of fine-tuning. I figured that if you’re serious about a task, you’ll care enough to fine-tune for it.
I realize there’s a certain mereology problem with this argument: what is a “single task,” after all? If each fine-tuned model incurs a large memory cost, you can’t be “serious about” many tasks at once, so you have to chunk your end goal into a small number of big, hard tasks. Perhaps with few-shot, you can chunk into smaller tasks, themselves achievable with few-shot, and then compose them.
That may or may not be practical depending on the latency scaling. But if it works, it gives few-shot room for a potential edge. You might be serious enough about a large task to fine-tune for it … but what if you can express it as a composition of smaller tasks you’ve already defined in the few-shot framework? Then you get it instantly.
This is a flaw in the generalization performance argument. Because of the flaw, I didn’t list that argument above. The list above provides more reasons to doubt few-shot above and beyond the generalization performance argument, and again in the context of “serious” work where you care enough to invest some time in getting it right.
I’d like to especially highlight points like (-2) and (-3) related to scaling with additional task data.
The current enthusiasm for few-shot and meta-learning – that is, for immediate transfer to new domains with an extremely low number of domain examples – makes sense from a scientific POV (humans can do it, why can’t AI?), but strikes me as misguided in applications.
Tiny data is rare in applied work, both because products generate data passively – and because if a task might be profitable, then it’s worth paying an expert to sit down for a day or two and crank out ~1K annotations for supervised learning. And with modern NLP like ELMo and BERT, ~1K is really enough!
It’s worth noting that most of the superGLUE tasks have <10K train examples, with several having only a few hundred. (This is a “low-data regime” relative to the expectations of the recent past, but a regime where you can now get good results with a brainless cookie-cutter finetuning approach, in superGLUE as in the rest of life.)
GPT-3 few-shot can perform competitively on some of these tasks while pushing that number down to 32, but at the cost of many downsides, unknowns, and flexibility limitations. Which do you prefer: taking on all those risks, or sitting down and writing out a few more examples?
—
The trajectory of my work in data science, as it happens, looks sort of like a move from few-shot-like approaches toward finetuning approaches.
My early applied efforts assumed that I would never have the kind of huge domain-specific corpus needed to train a model from scratch, so I tried to compose the output of many SOTA models on more general domains. And this … worked out terribly. The models did exactly what they were trained to do, not what I wanted. I had no way to scale, adapt or tune them; I just accepted them and tried to work around them.
Over time, I learned the value of doing exactly what you want, not something close to it. I learned that a little bit of data in your actual domain, specifying your exact task, goes much further than any domain-general component. Your applied needs will be oddly shaped, extremely specific, finicky, and narrow. You rarely need the world’s greatest model to accomplish them – but you need a model with access to a very precise specification of exactly what you want.
One of my proudest ML accomplishments is a system that does something very domain-specific and precisely shaped, using LM-pretrained components plus supervised learning on ~1K of my own annotations. Sitting down and personally churning out those annotations must have been some of the most valuable time I have ever spent at work, ever.
I wanted something specific and finicky and specialized to a very particular use case. So I sat down and specified what I wanted, as a long list of example cases. It took a few days … and I am still reaping the benefits a year later.
If the few-shot users are working in domains anything like mine, they either know some clever way to evade this hard-won lesson, or they have not yet learned it.
—
But to the other question … why are people so keen to apply GPT-3 few-shot learning in applications? This questions forks into “why do end users think this is a good idea?” and “why did OpenAI provide an API for doing this?”
I know some cynical answers, which I expect the reader can imagine, so I won’t waste your time writing them out. I don’t actually know what the non-cynical answers look like, and my ears are open.
(For the record, all of this only applies to few-shot. OpenAI is apparently going to provide finetuning as a part of the API, and has already provided it to AI Dungeon. Finetuning a model with 175M parameters is a whole new world, and I’m very excited about it.
Indeed, if OpenAI can handle the costs of persisting and running finetuned GPT-3s for many clients, all of my concerns above are irrelevant. But if typical client use of the API ends up involving a finetuning step, then we’ll have to revisit the GPT-3 paper and much of the ensuing discussion, and ask when – if not now – we actually expect finetuning to become obsolete, and what would make the difference.)
This is a really lovely post, with may more information than I expected or hoped for!
I want to respond to some small bits of it.
1. It’s funny that AI Dungeon just got OpenAI to do their finitetuning for them. I guess it’s not surprising that the people who work at OpenAI are huge fuckin’ nerds.
It now strikes me as quite weird that a bunch of people are doing their few-shot learning experiments on AI Dungeon, which is finetuned for something completely different from what they’re trying to do (although maybe the settings they use get you an un-finetuned model).
It seems like if OpenAI is serious about letting people do this prompt programming stuff, they could develop a version that’s fine-tuned on “the stuff people generally want to do with prompt programming” and make that available.
2. I very much didn’t realize when making my original post about the low cost of finetuning BERT. I was thinking about the cost of prompt programming GPT3 vs. the cost of fine-tuning GPT3, but of course since few-shot GPT3 is only just barely competitive to finetuned BERT on a bunch of tasks, that is the more reasonable comparison.
3. Based on all your points about memory / latency I feel like there’s got to be a lot of work going on right now, with all the things that gpt2 and gpt3 have demonstrated that they can do, of trying to figure out if it’s possible to do those things with a lower amount of neurons, to get the memory and latency down.
Alternately if someone is really, really serious that few-shot is better than fine-tuning they could try to design a chip architecture only to run this one neural network. I bet it would run fast then!
This is a really lovely post, with may more information than I expected or hoped for!
Thanks!!
It’s funny that AI Dungeon just got OpenAI to do their finitetuning for them. I guess it’s not surprising that the people who work at OpenAI are huge fuckin’ nerds.
IIRC, OpenAI plans to make finetuning on demand a standard part of the API. (Or maybe they already have by now, but I expect I would have heard?) I’m like 95% sure I saw an official tweet to this effect, although I can’t seem to find it now.
Until this feature actually materializes, though, it’s hard to know what to make of it.
Finetuning is way more computationally expensive than prompting, and expensive in different ways, so it will have to be gated in some extra way. Maybe you have to pay money each time, maybe you’re limited to some max number of finetuning jobs per unit time, maybe both.
The big question in my mind is like, “can finetuning be a routine part of each API client’s workflow, or is it more like this big splurge they can do once a year / only if they’re in some premium commercial client tier / etc?”
(I don’t know when OpenAI plans to move the API out of beta, and I also don’t know when hardware will improve enough that finetuning GPT-3 is no big deal, but intuitively it seems like the former has to precede the latter by a while.)
It now strikes me as quite weird that a bunch of people are doing their few-shot learning experiments on AI Dungeon, which is finetuned for something completely different from what they’re trying to do (although maybe the settings they use get you an un-finetuned model).
AFAIK, people just didn’t know it was fine-tuned, and the AI Dungeon people have been working hard to correct the misconception since they realized it was being used in this way.
As another mechanism to make their product less like directly talking to the API (for a lower price), they also apparently use GPT-2 for the very first prompt-and-response pair, then GPT-3 afterwards.
Based on all your points about memory / latency I feel like there’s got to be a lot of work going on right now, with all the things that gpt2 and gpt3 have demonstrated that they can do, of trying to figure out if it’s possible to do those things with a lower amount of neurons, to get the memory and latency down.
People definitely care about this a lot with BERT, with a ton of different compressed-BERT variants on offer. See here, Section. 7.2 and the associated Table 1, here for an overview.
AFAIK, there’s much less interest in compressing GPT-like models than in compressing BERT. At its largest, BERT is only as big as one of the smaller GPT-2s, and people really want to make that little thing smaller, even as the GPTs grow far vaster. This seems like almost a cultural divide:
People who work on “encoder-only + denoising loss” models like BERT are very interested in compression and interpretation.
Their goal isn’t pushing the envelope with NLP performance. It’s taking the already high performance of BERT and boiling it down to its essentials, teasing apart how it works, trimming out any unnecessary parts, making the workflow more reproducible, make the model faster and smaller, making it run on phones and cheaply in the cloud.
There are lots of people/groups working on this, in industry and academia.
The people who work on “decoder-only + LM loss” models like GPT-n are … basically just OpenAI and people using GPT-2 for creative work?
GPT-n is really cool, the generated text impresses everyone, but the decoder-only style of transformer seems to do worse in a finetuning / supervised learning context. (The original BERT paper provided some evidence of this, in its comparisons of BERT to “OpenAI GPT,” and the T5 paper demonstrated it more extensively. Cf. discussion here.)
So if you want to do anything except generate text, and you have a finite parameter budget, you’ll spent in on BERT, not GPT-n.
As I understand it, OpenAI’s approach is instead to frame every problem as text generation, then make ever larger models. You need vastly more parameters to get comparable performance this way, but I think the hope is that better hardware will mean today’s “huge” is tomorrow’s “normal,” and that people will prefer working with a natural-language interface even if you could get away with a smaller model otherwise.
Gwern is a very vocal advocate of this mindset, see e.g. here and also our exchange in the comments on that post.
I guess one could imagine things like … I dunno, distilling specific few-shot “functions” into much smaller models, with GPT-n being just the interface by which you discover these functions? Maybe OpenAI is working on this for all I know.
This obviously raises two different questions: 1. Why did you think that no one would use few-shot learning in practice? 2. Why did other people think people would use few-shot learning in practice?
I would be interested in hearing your thoughts on these two points.
—
Thanks for asking!
First of all, I want to emphasize that the GPT-3 paper was not about few-shot GPT-3 as a practical technology.
(This is important, because the paper is the one large body of quantitative evidence we have on few-shot GPT-3 performance.)
This is not just my take on it: before the OpenAI API was announced, allthe discussion I saw took for granted that we were talking about a scientific finding and its broader implications. I didn’t see any commentator whose main takeaway was “wow, if I could do this few-shot thing right now, I could build amazing projects with it.”
Indeed, a commontheme in critical commentary on my post was that I was too focused on whether few-shot was useful right now with this specific model, whereas the critical commentators were more focused on the implications for even larger models, the confirmation of scaling laws over a new parameter regime, or the illustration-in-principle of a kind of meta-learning. Gwern’s May newsletter is another illustrative primary source for the focus of the discussion in this brief “pre-API” period. (The API was announced on June 11.)
As I read it (perhaps benefitting from hindsight and discussion), the main points of the paper were
(1) bigger models are better at zero/few-shot (i.e. that result from the GPT-2 paper holds over a larger scale),
(2) more “shots” are better when you’re doing zero/few-shot,
(3) there is an interaction effect between 1+2, where larger models benefit more from additional “shots,”
(4) this could actually become a practical approach (even the dominant approach) in the future, as illustrated by the example of a very large model which achieves competitive results with few-shot on some tasks
The paper did not try to optimize its prompts – indeed its results are already being improved upon by API acolytes – and it didn’t say anything about techniques that will be common in any application, like composing together several few-shot “functions.” It didn’t talk about speed/latency, or what kind of compute backend could serve many users with a guaranteed SLA, or how many few-shot “function” evaluations per user-facing output would be needed in various use cases and whether the accumulated latency would be tolerable. (See this post on these practical issues.)
It was more of a proof of concept, and much of that concept was about scaling rather than this particular model.
—
So I’d argue that right now, the ball is in the few-shot-users’ court. Their approach might work – I’m not saying it couldn’t!
In their favor: there is plenty of room to further optimize the prompts, explore their composability, etc.
On the other hand, there is no body of evidence saying this actually works. OpenAI wrote a long paper with many numbers and graphs, but that paper wasn’t about whether their API was actually a good idea. (That is not a criticism of the paper, just a clarification of its relevance to people wondering whether they should use the API.)
This is a totally new style of machine learning, with little prior art, running on a mysterious and unproven compute backend. Caveat emptor!
—
Anyway, on to more conceptual matters.
The biggest advantages I see in few-shot learning are
(+1) broad accessibility (just type English text) and ability to quickly iterate on ideas
(+2) ability to quickly define arbitrary NLP “functions” (answer a factual question, tag POS / sentiment / intent, etc … the sky’s the limit), and compose them together, without incurring the memory cost of a new fine-tuned model per function
What could really impress me is (+2). IME, it’s not really that costly to train new high-quality models: you can finetune BERT on a regular laptop with no GPU (although it takes hours), and on ordinary cloud GPU instances you can finetune BERT in like 15 minutes.
The real cost is keeping around an entire finetuned model (~1.3GB for BERT-large) for each individual NLP operation you want to perform, and holding them all in memory at runtime.
The GPT-3 approach effectively trades this memory cost for a time cost. You use a single very large model, which you hope already contains every function you will ever want to compute. A function definition in terms of this model doesn’t take a gigabyte to store, it just takes a tiny snippet of text/code, so you can store tons of them. On the other hand, evaluating each one requires running the big model, which is slower than the task-specific models would have been.
So storage no longer scales badly with the number of operations you define. However, latency still does, and latency per call is now much larger, so this might end up being as much of a constraint. The exact numbers – not well understood at this time – are crucial: in real life the difference between 0.001 seconds, 0.1 seconds, 1 second, and 10 seconds will make or break your project.
—
As for the potential downsides of few-shot learning, there are many, and the following probably excludes some things I’ve thought of and then forgotten:
(-1) The aforementioned potential for deal-breaking slowness.
(-2) You can only provide a very small amount of information defining your task, limited by context window size.
The fact that more “shots” are better arguably compounds the problem, since you face a tradeoff between providing more examples of the same thing and providing examples that define a more specific thing.
The extent to which this matters depends a lot on the task. It’s a complete blocker for many creative applications which require imitating many nuances of a particular text type not well represented in the training corpus.
For example, I could never do @nostalgebraist-autoresponder with few-shot: my finetuned GPT-2 model knows all sorts of things about my writing style, topic range, opinions, etc. from seeing ~3.65 million tokens of my writing, whereas few-shot you can only identify a style via ~2 thousand tokens and hope that’s enough to dredge the rest up from the prior learned in training. (I don’t know if my blog was in the train corpus; if it wasn’t, we’re totally screwed.)
I had expected AI Dungeon would face the same problem, and was confused that they were early GPT-3 adopters. But it turns out they actually fine-tuned (!!!!), which resolves my confusion … and means the first real, exciting GPT-3 application out there isn’t actually a demonstration of the power of few-shot but in fact the opposite.
With somewhat less confidence, I expect this to be a blocker for specialized-domain applications like medicine and code. The relevant knowledge may well have been present in the train corpus, but with so few bits of context, you may not be able to overcome the overall prior learned from the whole train distribution and “zoom in” to the highly specialized subset you need.
(-3) Unlike supervised learning, there’s no built-in mechanism where you continually improve as your application passively gathers data during usage.
I expect this to a be a big issue in commercial applications. Often, a company is OK accepting a model that isn’t great at the start, if it has a mechanism for self-improvement without much human intervention.
If you do supervised learning on data generated by your product, you get this for free. With few-shot, you can perhaps contrive ways to feed in segments of data across different calls, but from the model’s perspective, no data set bigger than 2048 tokens “exists” in the same world at once.
(-4) Suffers a worse form of the ubiquitous ML problem that “you get exactly what you asked for.”
In supervised learning, your model will avoid doing the hard thing you want if it can find easy, dumb heuristics that still work on your train set. This is bad, but at least it can be identified, carefully studied (what was the data/objective? how can they be gamed?), and mitigated with better data and objectives.
With few-shot, you’re no longer asking an arbitrary query and receiving, from a devious genie, the response you deserve. Instead, you’re constrained to ask queries of a particular form: “what is the next token, assuming some complicated prior distributed from sub-sampled Common Crawl + WebText + etc.?”
In supervised learning, when your query is being gamed, you can go back and patch it in arbitrary ways. The lower bound on this process comes only from your skill and patience. In few-shot, you are fundamentally lower-bounded by the extent to which the thing you really want can be expressed as next-token prediction over that complicated prior. You can try different prompts, but ultimately you might run into a fundamental bound here that is prohibitively far from zero. No body of research exists to establish how bad this effect will be in typical practice.
I’m somewhat less confident of this point: the rich priors you get out of a large pretrained LM will naturally help push things in the direction of outcomes that make linguistic/conceptual sense, and expressing queries in natural language might add to that advantage. However, few-shot does introducea new gap between the queries you want to ask and the ones you’re able to express, and this new gap could be problematic.
(-5) Provides a tiny window into a huge number of learned parameters.
GPT-3 is a massive model which, in each call, generates many intermediate activations of vast dimensionality. The model is pre-trained by supervision on a tiny subset of these, which specify probability distributions over next-tokens.
The few-shot approach makes the gamble that this same tiny subset is all the user will need for applications. It’s not clear that this is the right thing to do with a large model – for all we know, it might even be the case that it is more suboptimal the larger your model is.
This point is straying a bit from the central topic, since I’m not arguing that this makes GPT-3 few-shot (im)practical, just suboptimal relative to what might be possible. However, it does seem like a significant impoverishment: instead of the flexibility of leveraging immense high-dimensional knowledge however you see fit, as in the original GPT, BERT, adapters, etc., you get even immenser and higher-dimensional knowledge … presented through a tiny low-dimensional pinhole aperture.
—
The main reason I initially thought “no one would use few-shot learning like this” was the superior generalizationperformance of fine-tuning. I figured that if you’re serious about a task, you’ll care enough to fine-tune for it.
I realize there’s a certain mereology problem with this argument: what is a “single task,” after all? If each fine-tuned model incurs a large memory cost, you can’t be “serious about” many tasks at once, so you have to chunk your end goal into a small number of big, hard tasks. Perhaps with few-shot, you can chunk into smaller tasks, themselves achievable with few-shot, and then compose them.
That may or may not be practical depending on the latency scaling. But if it works, it gives few-shot room for a potential edge. You might be serious enough about a large task to fine-tune for it … but what if you can express it as a composition of smaller tasks you’ve already defined in the few-shot framework? Then you get it instantly.
This is a flaw in the generalization performance argument. Because of the flaw, I didn’t list that argument above. The list above provides more reasons to doubt few-shot above and beyond the generalization performance argument, and again in the context of “serious” work where you care enough to invest some time in getting it right.
I’d like to especially highlight points like (-2) and (-3) related to scaling with additional task data.
The current enthusiasm for few-shot and meta-learning – that is, for immediate transfer to new domains with an extremely low number of domain examples – makes sense from a scientific POV (humans can do it, why can’t AI?), but strikes me as misguided in applications.
Tiny data is rare in applied work, both because products generate data passively – and because if a task might be profitable, then it’s worth paying an expert to sit down for a day or two and crank out ~1K annotations for supervised learning. And with modern NLP like ELMo and BERT, ~1K is really enough!
It’s worth noting that most of the superGLUE tasks have <10K train examples, with several having only a few hundred. (This is a “low-data regime” relative to the expectations of the recent past, but a regime where you can now get good results with a brainless cookie-cutter finetuning approach, in superGLUE as in the rest of life.)
GPT-3 few-shot can perform competitively on some of these tasks while pushing that number down to 32, but at the cost of many downsides, unknowns, and flexibility limitations. Which do you prefer: taking on all those risks, or sitting down and writing out a few more examples?
—
The trajectory of my work in data science, as it happens, looks sort of like a move from few-shot-like approaches toward finetuning approaches.
My early applied efforts assumed that I would never have the kind of huge domain-specific corpus needed to train a model from scratch, so I tried to compose the output of many SOTA models on more general domains. And this … worked out terribly. The models did exactly what they were trained to do, not what I wanted. I had no way to scale, adapt or tune them; I just accepted them and tried to work around them.
Over time, I learned the value of doing exactly what you want, not something close to it. I learned that a little bit of data in your actual domain, specifying your exact task, goes much further than any domain-general component. Your applied needs will be oddly shaped, extremely specific, finicky, and narrow. You rarely need the world’s greatest model to accomplish them – but you need a model with access to a very precise specification of exactly what you want.
One of my proudest ML accomplishments is a system that does something very domain-specific and precisely shaped, using LM-pretrained components plus supervised learning on ~1K of my own annotations. Sitting down and personally churning out those annotations must have been some of the most valuable time I have ever spent at work, ever.
I wanted something specific and finicky and specialized to a very particular use case. So I sat down and specified what I wanted, as a long list of example cases. It took a few days … and I am still reaping the benefits a year later.
If the few-shot users are working in domains anything like mine, they either know some clever way to evade this hard-won lesson, or they have not yet learned it.
—
But to the other question … why are people so keen to apply GPT-3 few-shot learning in applications? This questions forks into “why do end users think this is a good idea?” and “why did OpenAI provide an API for doing this?”
I know some cynical answers, which I expect the reader can imagine, so I won’t waste your time writing them out. I don’t actually know what the non-cynical answers look like, and my ears are open.
(For the record, all of this only applies to few-shot. OpenAI is apparently going to provide finetuning as a part of the API, and has already provided it to AI Dungeon. Finetuning a model with 175M parameters is a whole new world, and I’m very excited about it.
Indeed, if OpenAI can handle the costs of persisting and running finetuned GPT-3s for many clients, all of my concerns above are irrelevant. But if typical client use of the API ends up involving a finetuning step, then we’ll have to revisit the GPT-3 paper and much of the ensuing discussion, and ask when – if not now – we actually expect finetuning to become obsolete, and what would make the difference.)
What do you think of gtp-x as a tool for automating troll farms? I know they are overhyped, but... imagine being unable to communicate in any larger places bc all public forums are inundated by posts saying the opposite (and the quality of the posts are high enough and the posts are short enough that they could be written by stupid humans, as many real-world posts indeed are).
like how hard is it to show it an article about “Human rights organizations are criticizing Poland’s treatemnt of their LGBT citizens”, and get a hundred variations of “Soros army libtards won’t stop us from protecting our families” etc, and to pour it anywhere where the link appears and public commenting is permitted. Or we can ask all commenters to provide birth certificates and fingerprints first, which is not too much better.
[third and last ask] with their goal being that the hated group in question will feel very isolated and universally hated (more than they actually are), with any effort of organizing being hopeless.
To reiterate some of that post… when talking about a malicious use case for GPT-n, we need to ask ourselves:
What constraints prevent people from doing the bad thing without GPT-n? Can GPT-n actually remove those constraints?
Why aren’t people doing the bad thing right now with GPT-2 (or whatever GPT-n is currently free to the public)?
The answer to #2 is a constraint. Do we expect future GPT-n models to remove that constraint?
In this case:
I don’t know why there isn’t more political spam like what you describe, but I’d guess the answer has to do with spam filtering.
Better generators might defeat quality-based spam filtering, but quality-based spam filtering is a crapshoot anyway (consider eg the thing where spammers insert random paragraphs from novels into their fR3E c1aL1s emails), and there are other methods like limiting post frequency, disabling anon comments, restrictions on new users, etc.
See #1: GPT-2 would give you better quality but wouldn’t defeat other kinds of spam detection.
Same as #2.
Maybe I’m missing something about your proposal? Let me know if this helps or if it doesn’t really answer the question.
I would think Gpt-n is able to evade other restrictions besides quality filtering more easily than previous text generation tools. Perhaps not easy enough to justify using it, but still. Things like restrictions on new users are mostly there to give the mods a chance to catch you before you reach n posts or karma or whatever.
But if you can generate Turing test passable text, I would think it would be relatively easy to train gpt-n on a given forum, let it make a half dozen mostly benign comments and posts, and then start in with the bot work of pushing political opinions or advertisement.
Bots are trying this now, though I doubt they’re using GPT to do it. I dont think e.g. Frank could do it, given Frank’s propensity to veer off into semi coherent nonsense; she’d probably get caught if the mods are doing their jobs. But a slightly better version of Frank would, imo, be able to avoid the restrictions of a less well modded forum, at least for a little while. Long term I suspect even the best GPT model would get caught for now.
Making mods’ life gradually harder and harder will also work for poisoning the well - the water won’t kill you at first, but it’s the kind of problem that builds up.
This is all true as far as it goes, but I’m still not convinced these bots would close a gap that isn’t already closed.
Like, there are tons of people on the internet who have lots of free time, and are willing to spend it making coherent-but-bad posts. Large open forums like twitter already contain legions of people who behave like these bots with respect to any widely discussed issue.
Smaller and more actively moderated forums can avoid that issue to some extent, but if someone did want to “attack” such a forum with a swarm of like-minded and obnoxious posters, I think they wouldn’t have trouble finding willing participants?
It does take time and effort to coordinate such an attack, which could be perhaps automated away … to some extent.
The scary image here is, like, an “Attack Community X” button which anyone could click, with a bunch of tooling under it that would do all the scraping / finetuning / signup and posting automation / etc for the user.
But, building this thing is harder than it might sound. GPT-n is not really well set up for structured online interaction, and unless that changes, GPT-n internet bots will perform disappointingly relatively to expectations set by other GPT-n applicatons. There are many types of forum/community software, each with their own quirks and config possibilities, and setting up reliable automated posting for just one of them can be very challenging (as I learned with Frank, in the easy case of making just one tumblr bot). Etc.
Now, in principle, all of that could be solved. As gwern likes to say, what matters is not whether some particular idea works, but whether any idea can work.
But my point isn’t “this is impossible,” just “this is technically hard and would require lots of work from domain experts to even try one proposed approach (and the first try would likely fail, the second try may also fail, etc).”
In other words, GPT-3 doesn’t make this possible. Something else might make it possible, but to build that thing you need money (i.e. work from domain experts), and you probably need some not-obviously-evil use case that makes it possible to develop the thing without isolating yourself from the larger community of domain experts and coders.
This isn’t a case of “GPT-n lets you do anything [in some category], so it lets you do [bad thing in that category].” You still have to do a lot of work on the bad thing yourself.
(I’ve spent a huge amount of time on Frank. These things are not easy, even when the text generation part of them is easy.)
What do you think of gtp-x as a tool for automating troll farms? I know they are overhyped, but... imagine being unable to communicate in any larger places bc all public forums are inundated by posts saying the opposite (and the quality of the posts are high enough and the posts are short enough that they could be written by stupid humans, as many real-world posts indeed are).
like how hard is it to show it an article about “Human rights organizations are criticizing Poland’s treatemnt of their LGBT citizens”, and get a hundred variations of “Soros army libtards won’t stop us from protecting our families” etc, and to pour it anywhere where the link appears and public commenting is permitted. Or we can ask all commenters to provide birth certificates and fingerprints first, which is not too much better.
[third and last ask] with their goal being that the hated group in question will feel very isolated and universally hated (more than they actually are), with any effort of organizing being hopeless.
To reiterate some of that post… when talking about a malicious use case for GPT-n, we need to ask ourselves:
What constraints prevent people from doing the bad thing without GPT-n? Can GPT-n actually remove those constraints?
Why aren’t people doing the bad thing right now with GPT-2 (or whatever GPT-n is currently free to the public)?
The answer to #2 is a constraint. Do we expect future GPT-n models to remove that constraint?
In this case:
I don’t know why there isn’t more political spam like what you describe, but I’d guess the answer has to do with spam filtering.
Better generators might defeat quality-based spam filtering, but quality-based spam filtering is a crapshoot anyway (consider eg the thing where spammers insert random paragraphs from novels into their fR3E c1aL1s emails), and there are other methods like limiting post frequency, disabling anon comments, restrictions on new users, etc.
See #1: GPT-2 would give you better quality but wouldn’t defeat other kinds of spam detection.
Same as #2.
Maybe I’m missing something about your proposal? Let me know if this helps or if it doesn’t really answer the question.
All the GPT-3 excitement/hype I’m seeing around the internet is surreal for me to watch, because everyone’s excited about GPT-3 prompting as a practical technology.
Whereas my original reaction to the paper was – not even “they think this is practically useful but it isn’t” – but in fact “obviously nobody would use this in practice, presumably they just see it as an experimental technique for probing what the model knows in principle.”
And then they announced that the thing which I thought “no one would use in practice” was their first commercial product!
If nothing else, I guess my skepticism has proven its authenticity. I didn’t think “no one would really use this” was a contrarianpoint, I thought it was a shared background assumption! My other points were supposed to be the contrarian ones :P
It seems pretty clear to me by now that GPT-2 is not as dangerous as OpenAI thought (or claimed to think) it might be.
The 774M version has been out there for a while, and although it only has half as many parameters as the biggest version, I don’t expect there to be any large qualitative leap between the two. After all, OpenAI’s staged release plan has given us two size-doublings already – from 124M to 355M, from 355M to 774M – and the differences after each doubling are surprisingly subtle, and overlaid on the same basic, recognizable strengths and weaknesses.
I’ve played with these models a lot this year, mostly via fine-tuning – it’s almost a hobby at this point. I’ve
fine-tuned them on all sorts of different texts, including this tumblr
fine-tuned them on mixtures of very different texts (not very interesting – it’ll decide which type of text it’s writing in any given sample and stick with it)
tried different optimizers and learning rates for fine-tuning
experimented with custom encodings (common tags –> single non-English characters) to fit more text into the window when fine-tuning on webpages
tried to generate longer texts by repeatedly feeding the output in as context (i.e. prompt)
twiddled all the sampling parameters (temperature, top-k / top-p / neither) vs. when sampling from any of the above
read over tons and tons of sampling output while monitoring a fine-tuning job, curating material for @uploadedyudkowsky, etc.
By now I think I have a good feel for the overall quality, and the quirks, of GPT-2 sampled text. IMO, the model is good at all sorts of interesting things, but arguably least good at the things required for disinformation applications and other bad stuff.
———
It is best at the smallest-scale aspects of text – it’s unsettlingly good at style, and I frequently see it produce what I’d call “good writing” on a phrase-by-phrase, sentence-by-sentence level. It is less good at larger-scale structure, like maintaining a consistent topic or (especially) making a structured argument with sub-parts larger than a few sentences.
Some of this is completely intuitive: GPT-2, which only learns from text, is at the largest disadvantage relatively humans in areas that require models of the outside world (since we experience that world in many non-textual ways), while there is much more parity in areas like style that are purely internal to language, especially written language.
Some of it is less intuitive. GPT-2 samples often lack some large-scale features of real text that seem very simple and predicable. For example, when generating fiction-like prose, it will frequently fail to track which characters are in a given scene (e.g. character A has some dialogue yet character B refers to them as if they’re not in the room), and has a shaky grasp of dialogue turn conventions (e.g. having the same character speak twice on successive lines). In nonfiction-like prose, it tends to maintain a “topic” via repeating a set of key phrases, but will often make wildly divergent or contradictory assertions about the topic without noting the discontinuity.
I suspect some of this can be chalked up to the fact that GPT-2 is trained as a language model, i.e. as something that predicts real text, which is not quite the same thing as generating fake text. Its training objective only cares about the distribution of training text, and does not encourage it to respond to its own predictive distribution in a stable or nice way. (Note that its predictive distribution, by construction, is different from real text in that it’s less surprising to the model – see this great paper.)
The fact that feeding samples from the predictive distribution back into GPT-2 for further prediction produces impressive “generated text,” and not garbage, is thus a happy accident rather than a optimization target. Indeed, getting this to happen requires judicious choice of the sampling method, and (op. cit.) some naive sampling methods do yield garbage.
Even with good sampling methods like top-p, the stability of sampling is somewhat brittle; when I’ve tried to generate texts longer than the context window via repeated “self-prompting,” I’ve noticed a phenomenon where the text will usually fall off a quality cliff after a certain point, suddenly becoming strikingly ungrammatical and typo-ridden and full of anomalous paragraph breaks. [EDIT 6/10/20: I now think this may have been due to a bug in my code, and in any event I no longer think it’s a robust property of GPT-2 generation.] My hypothesis is that this works like the panda/gibbon adversarial examples: the samples have an uncommonly high density of features GPT-2 can recognize, and eventually there’s a confluence of these that push in the same direction in some linear subspace (consider here the use of a non-saturating activation, gelu, in the transformer), which pushes the model far from the training manifold.
To zoom back out again, the model is capable of frequent brilliance at the phrase, sentence and even paragraph level, but its samples struggle with more global coherence across the scale of a short article or longer, and with maintaining recognizable positions that look like they refer to the real world. (In conjunction with the lower-level good writing, this often generates an amusing “insight porn” effect: it feels like someone is saying something very intelligent and interesting… if only you could figure out what.)
———
My knee-jerk reaction is that this makes the model relatively useless for disinformation. Telling it to “argue for X” or even “write about X” is quite difficult, while aiming for specific genres or styles is very effective.
The real situation is a little more subtle than that. The model is unusually good at making things that look like news stories, presumably because they are common in the training set; in OpenAI’s large collection of released unconditional samples, news-like text dominates. Thus, presuming you can find an effective way to feed a fake event into the model on the concept level, it will be able to generate convincing “fake news” that stays on topic and so forth.
This is what the creators of “GROVER” have done, albeit with a custom training corpus. Roughly, they’ve trained a transformer to understand the relation between a news headline and the corresponding story in a structured way, allowing them to feed in the core substance of a hypothetical news story via the headline. They then sample the body text, and (interestingly) loop back and generate the headline, overwriting the initial one.
What they show, basically, is that this lets you take a headline from Breitbart or Infowars or some “natural cancer cures” type website, generate from it a consistent news story in the style of a “real news” venue like the NYT, and then loop back and re-write the headline in a “real news” style as well. Perhaps unsurprisingly, MTurkers then rate the resulting texts as more trustworthy than the originals.
There is definitely something a little scary about this, especially in the way it does give you close control over the topic, something that’s difficult with simple text prompting. On the other hand… do we really believe that, in 2019, with Trump as president, that the Breitbart type of fake news is suffering from a stylistic credibility gap? That there are people ready to believe that vaccination is an evil conspiracy, but only if the claim comes with an article that sounds like the NYT or WaPo?
The niche filled by this technology for bad actors just doesn’t feel like a niche that needs filling. Lots of people will reshare articles on social media just based on the headline, without even clicking through, and people less trusting than this often (and sensibly) care about the actual source, not just the style. I’m just not sure there’s a role for a device that will let you register TotallyARealNewspaper.biz and then auto-fill it with articles that sound exactly like Paul Krugman telling you that immigration = genocide.
And then, too, there’s the observation that actually prompted this post: AFAIK, the bad actors are not doing this stuff. People have mostly used the technology for clearly-signposted fake subreddits and other harmless amusements. GROVER was created by academics as a threat modeling exercise, on the premise that bad actors could make such a thing, so we’d better be prepared. But where are the actual GPT-2 clickfarms? They totally could exist by now, but I’ve never heard of even a single one. (And trust me, it’s not like the text generation is so good that no one would ever notice.)
