me: when an ML library says the details are in “the tutorials,” i just know i’m going to be poring over its internals 30 minutes later to try to figure out how anything works
—————————————————————-
the documentation:
me: these are my least favorite words in the world
me: i’m definitely going to be poring over internals in 30 minutes, won’t i
me: when an ML library says the details are in “the tutorials,” i just know i’m going to be poring over its internals 30 minutes later to try to figure out how anything works
—————————————————————-
the documentation:
me: these are my least favorite words in the world
me: i’m definitely going to be poring over internals in 30 minutes, won’t i
I’ve made some changes to Frank’s image model in the last few days:
I’ve incrementally improved the model. The new one has more parameters, and can “see” more text per image (up to 384 characters). May improve image quality.
I’ve added classifier-free guidance. This makes the images more likely to contain the text Frank wants to write, at the cost of potentially making them less varied.
You will see tags like “#guidance scale 2” start to appear on Frank posts with images. This tells you how much classifier-free guidance was used, where 0 means “none.”
I’m not sure what the “sweet spot” for this number is, so for now, I’m having Frank randomly pick from a range of different values.
[EDIT 9/6/22: I wrote this post in January 2022. I’ve made a number of improvements to this model since then. See the links above for details on what the latest version looks like.]
Last week, I released a new feature for @nostalgebraist-autoresponder that generates images. Earlier I promised a post explaining how the model works, so here it is.
I’ll try to make this post as accessible as I can, but it will be relatively technical.
Why so technical? The interesting thing (to me) about the new model is not that it makes cool pictures – lots of existing models/techniques can do that – it’s that it makes a newkind of picture which no other model can make, as far as I know. As I put it earlier:
As far as I know, the image generator I made for Frank is the first neural image generator anyone has made that can write arbitrary text into the image!! Let me know if you’ve seen another one somewhere.
The model is solving a hard machine learning problem, which I didn’t really believe could be solved until I saw it work. I had to “pull out all the stops” to do this one, building on a lot of prior work. Explaining all that context for readers with no ML background would take a very long post.
tl;dr for those who speak technobabble: the new image generator is OpenAI-style denoising diffusion, with a 128x128 base model and a 128->256 superresolution model, both with the same set of extra features added. The extra features are: a transformer text encoder with character-level tokenization and T5 relative position embeddings; a layer of image-to-text and then text-to-image cross-attention between each resnet layer in the lower-resolution parts of the U-Net’s upsampling stack, using absolute axial position embeddings in image space; a positional “line embedding” in the text encoder that does a cumsum of newlines; and information about the diffusion timestep injected in two places, as another embedding fed to the text encoder, and injected with AdaGN into the queries of the text-to-image cross-attention. I used the weights of the trained base model to initialize the parts of the superresolution model’s U-Net that deal with resolutions below 256.
This post is extremely long, so the rest is under a readmore
Seeing Frank’s image generator operating in the real world has given me a lot of signal on how it can be improved, so I’m working on that now.
The image generator will hopefully improve a lot over time, like some of Frank’s other features.
In particular, I think the tendency to generate blurry, formless images (e.g.) is due to overly aggressive cropping of the training data.
I cropped roughly to the bounding box of the machine-recognized text in the image, thinking I wanted to make the text as legible as possible to maximize my changes of generating legible text at all.
But sometimes the image-to-text model just sees the letter “D” or something in a large image, so I end up with a blurry, ultra-zoomed-in picture of the letter “D” … and the generator has learned to skillfully imitate these boring, blurry pictures when given a sufficiently short text prompt. I guess I got what I asked for!
Continuing to train now on a differently cropped version of the data, we’ll see how it goes.
Still have to make a longer post on Frank’s new image generator, but a few quick comments:
- Many thanks to everyone I talked to about this project in the EleutherAI discord! Special thanks to RiversHaveWings for suggesting I try diffusion for this problem + various helpful tips along the way.
- As far as I know, the image generator I made for Frank is the first neural image generator anyone has made that can write arbitrary text into the image!! Let me know if you’ve seen another one somewhere.
- The model is a text-conditioned denoising diffusion model. Or rather two of them, a 128x128 base model and a 256x256 upsampler.
- Coincidentally, just 3 days ago, OpenAI announced/released their own text-conditioned denoising diffusion model. I guess it’s an idea whose time has come! Their model is structured a little differently, and makes way better-looking images, although with the writing-text aspect.
- My code for this model is a heavily modified fork of OpenAI’s improved-diffusion repo. It’s on this branch. The Files Changed view here gives a clearer sense of what I changed. (Caveat: it’s extremely hacky research code)
gotta ask at this point but are you steelmaning here for the sake of speculation or do you actually believe yud and the ‘miri adjacent’ believe in imminent ai x-risk in any way other than liminally as a vehicle for self-promotion? like not to be a total cynic but i don’t know how to read (what i perceive to be) miri’s strategy as a legitimate effort towards the stated goals
Oh, these days I’m convinced they’re 100% sincere.
I do suspect there are people and groups in the broader “EA” space that are like this … especially the ones that are bigger and closer to the conventional charity ecosystem, where there are larger amounts of money slushing around.
But MIRI? It’s just not that big, it doesn’t get that much money, it’s unabashedly weird in a manner that might have career penalties (but which true-believer employees don’t care about), and its pitch to donors is the kind of thing you either believe or you don’t, in binary fashion.
I have a hard time picturing the details of the timeline making much of a difference to donors. If you’re the sort of person who says “eh, my AI timeline is shifted a few decades out from theirs, so I can wait a while before I start giving them money,” you’re not the sort of person who donates to MIRI to begin with.
That isn’t the main thing that convinced me, though. The main thing is that the “MIRI-adjacent” crowd produces tons of esoteric, effort-intensive writing and debate that would be both strange and ineffective as PR, but looks perfectly natural if you read it as the result of genuine intellectual interest. (This is like half the content on LW dot com these days, now that it’s merged w/ agent foundations.)
To pick an almost arbitrary example, here’s a math-heavy post by an AI safety researcher not affiliated with MIRI, formalizing the content of a single Yudkowsky remark from the recent dialogues. I suppose there could be a cynical hypothesis on which such people are “marks” wrongly taking the core group at face value… but LW posts by the “core group” (eg actual MIRI researchers) look like this too.
—-
I think the deal with MIRI is simply that it was … founded by Eliezer Yudkowsky. So, it approaches problems the way he does.
Yudkowsky has really pessimistic intuitions about AI safety. His writing on the topic is full of accusations that other researchers don’t appreciate the sheer difficulty of the problem, that some idea X or Y would “obviously fail” in reality, that mere “ordinary paranoia” (his coined term) is insufficient, etc. IIRC there’s some old post where he says something like, “my most basic mental gesture is 'no, that wouldn’t work, try something else.’”
A lot of his conversations with other people, incl. the recent dialogues, have this talking-past-each-other quality, because it seems like he really wants to transmit this pessimistic intuition, rather than win the argument on any concrete point that’s been raised. He feels the intuition more strongly than (most?) other people in the MIRI orbit, who in turn presumably feel it more strongly than anyone else.
Yet, despite believing that “AI safety seems intractable” with perhaps more felt passion than anyone else on earth, Yudkowsky chose to work on – yes, AI safety. To “shut up and do the impossible,” as he puts it.
I think this explains both MIRI’s oddly low rate of output (relative to others in AI safety or just research groups in general), and the oddity of the output they do produce.
They’re not sitting there twiddling their thumbs; they’re considering every idea they can come up with and having the instinctive “no, that wouldn’t work” reaction to each of them in turn. If you think you’re fighting an unwinnable battle, being ordinarily “productive” is going to feel self-deceptive. The nature of the problem already renders most incremental work frivolous. You need to think of something so fundamentally outside the box, it has a chance of evading your intuition that nothing can possibly work.
Likewise, I think the stuff MIRI does produce is less “an approach they feel confident will work” and more like “the least intuitively repellant subset of things that 'obviously can’t work’ (i.e. everything).”
The best critique of this mindset IMO is that it defers too much to intuition and cuts off too many avenues of formal modeling before they can even get started. Math can surprise us, and things that “obviously can’t work if you think about them for 5 seconds” may reveal unexpected facets after 5000 seconds, or 5 million. Sometimes you need to raise the temperature of the system to escape an equilibrium.
But I don’t think “this is self-serving” is a reasonable read on this kind of writing (I mean the LW posts), produced in this volume, for a mostly self-selecting audience. If you just want to make a slush fund for yourself and your friends, there are easier ways!
I’ve been reading these new dialogues between Yudkowsky and MIRI-adjacent people where EY is super pessimistic… he thinks superhuman AI is coming very soon now, and he thinks (reasonably) that the AI safety field won’t be ready in that timeframe
Everyone in these debates finds recent AI progress scary. Things like GPT-3, AlphaZero, AlphaFold keep coming up. Nate Soares says:
I observe that, 15 years ago, everyone was saying AGI is far off because of what it couldn’t do – basic image recognition, go, starcraft, winograd schemas, programmer assistance. But basically all that has fallen. The gap between us and AGI is made mostly of intangibles. (Computer Programming That Is Actually Good? Theorem proving? Sure, but on my model, “good” versions of those are a hair’s breadth away from full AGI already. […]) That’s a very uncomfortable place to be!
I don’t know how to bridge the gap between this and what I believe … it’s a big gap. I dunno, maybe I really am just the guy who will always be saying “everything seems fine,” even when the proverbial house is on fire.
But the gap we still have to close is really big! Soares and Yudkowsky are worried about scenarios where the AI, like, invents killer nanomachines and convinces people to manufacture them over the internet. And does this successfully because it has a model of the world around it, of human behavior, etc. that lets it predict the consequences of its actions. (Or something equivalent to that model in its implications)
We don’t have “AIs” that model a complex world around them like this. Like, at all. Nor do we have models that could plausibly do this if given more compute, or even a research agenda that looks like it’s headed in the direction of such models.
We do have systems (AlphaZero and the like) that can learn world models and do superhuman planning with them, but only in toy domains with extremely simple dynamics.
Matching humans on toy domains is still at the frontier of AI research. Just earlier this winter, EfficientZero solved a major outstanding problem by learning to play Atari games, about as well as humans play them, after only playing them for ~2 hours.
This is a huge deal: 3-4 years ago, I loved to talk about how reinforcement learning was so sample inefficient it took years (?) of game time to learn Atari. That the field started focusing on sample efficiency, and then cracked the problem, is real progress.
But … it’s fucking Atari! It’s simple mostly-deterministic dynamics invented specifically so it could be quickly learned by humans as a form of entertainment. The field didn’t focus on Atari because it was a hard task on an objective scale, it focused on Atari because even a task as objectively easy as Atari turned out to be hard for RL, and you’ve gotta start somewhere.
Meanwhile, GPT-3 doesn’t really “model” anything. It has all sorts of fragmentary implicit models of different things, but it can’t use them to plan. It doesn’t know what they are, can’t connect them to one another. If you want to make it better with more compute, you need to give it more data at the same time, and that’s from a starting point of “orders of magnitude more than the amount of text you’ll ever read.” A starting point of “might as well be the entire internet.”
Show me something that can learn and plan in real time, in a domain that’s a few orders of magnitude closer to adequate for the real world than Atari, with a sensory bandwidth that’s a few orders of magnitude closer to adequate for the real world than Atari, and then I’ll be scared, maybe.
Is this “moving the goalposts”? But if it is, why would that matter? You’re the one afraid of the robot. I’m just listing some properties the robot needs to have. The goalposts keep moving because it’s hard to wrap my mind around just how difficult your robot is to make!
Had an idea and thought I'd ask you about it, since you seem knowledgeable about it: with Codex and similar showing that large language models can write code that compiles, do you think it would be feasible to fine tune a LLM to produce proof checker code from math papers, in order to help automate rigorous proof checking? Possibly using multiple tries or a tree search to allow the model multiple attempts at producing a valid result? If not, how far off do you think a tool like this would be?
I don’t know much about proof checkers, which limits how helpful I can be here.
Also, I’m not sure I understand the concept… you do still have to read over the code and verify that it expresses the intended proof, right? So the idea is that correct code is much harder to write than to verify? (Or just more tedious to write than verify, I guess.)
The most obvious way to train something like this would involve a very large dataset of math papers and their translations into code. Lacking that, you could go the “unsupervised machine translation” route and mostly train on the two domains separately.
In either case, I’d expect the limiting factor to be the quantity of existing proof checker code.
Like, Codex is not that helpful, especially for nontrivial tasks, and it had all of github to train on (well, the python part of it, but that’s still a lot edit: forgot the deployed Codex model knows various langs). I’ve heard OpenAI is working on more impressive code models, so I don’t think Codex is the limit of how useful they can be, but I wouldn’t expect much out of them in niche, high-abstraction languages/domains for the time being.
