Install Theme

admiral-craymen asked:

What's with all of the blank answers from Frank lately?

Could you link me to an example?

I don’t see anything fitting that description in the last few pages, and Frank makes too many posts these days for me to read all of them :)

breakruns

I wrote earlier about “Mirostat,” an approach to sampling from language models that tries to avoid the infamous phenomenon of “neural text degeneration.”

In fact, I used Mirostat in pracitce for a long while, in @nostalgebraist-autoresponder. However, some things about it bothered me, e.g.:

It feels overly complicated.

It’s based on the assumption that the LM’s predictive distribution for an individual token is approximately a Zipf distribution.

We expect statistics of entire corpora to be Zipfian, but I don’t see why that implies anything about predictive distributions on the token level.

Indeed, this assumption is not at all true (I checked)! For instance, the model is often very confident, and puts almost all of the mass at one or a few top tokens. Even when it is not very confident, that looks like ~100% of the mass spread across the “reasonable possibilities” and then a long tail that basically doesn’t matter.

The Mirostat paper needlessly truncates a sum. When I replace this with the full sum, the results are drastically different.

—–

I was not comfortable with Mirostat, but still wanted to avoid “degeneration.”

So, I thought about the problem a bit, and came up with a new method, which I called “Breakruns.”

It is based on the following ideas:

(1)

There are two kinds of “degeneration”: the repetition trap, and incoherence. The degeneration and Mirostat papers treat these as sort of symmetrical.

However, they’re very different:

Incoherence is basically 100% solved by using top-p. More generally, “incoherence” just feels like what happens when you make too many choices that the model knows are almost certainly wrong; it feels fundamentally avoidable if you just “trust the model enough.”

In other words, the LM knows there’s something wrong with incoherent text, and it will tell you this. That’s just what an LM is, more or less.

The repetition trap, though, is a mistake the model thinks is correct. That’s a much tougher puzzle since the model’s opinions are all you have to go on. (Indeed, the model is arguably not even wrong about this issue – just right in an undesired manner.)

So, everything would go wonderfully if we could just “trust the model” enough, by using conservative sampling parameters like low T and/or low top-p.

The problem with this is supposedly that it produces “overly conservative” text, but IME that isn’t quite right. “Conservative” text from an LM tends to be good text … right up until the point where it becomes unnaturally repetitive.

If we could just solve the repetition trap on its own, everything else might fall into place.

(2)

The repetition trap is fundamentally about the model’s top-1 token.

If we’re in the trap, the sampler is always selecting its top-1 token, and will continue to do so.

Conversely, if we keep selecting the top-1 token for a long time, we might not be in the trap … but if not, trying something except choice #1, at some point, probably won’t hurt.

This is hard to think about at first, if you’re used to viewing discrete distributions as “merely” approximations to continuous ones. (Probably it can be made into a limit statement? but that’s not relevant for my purposes anyway)

—–

Here’s what Breakruns does.

You use conservative sampling, with low T and top-p. Not absurdly low, but lower than you would normally go.

You keep a running counter. Every time you pick the top-1 token, you increment the counter by 1. Every time you don’t pick the top-1 token, the counter resets to 0.

The counter is the length of the current “run” – an unbroken string of top-1s.

You don’t want to let the runs get too long. So, the longer the run gets, the more you crank up the temperature.

Specifically, if T is your “base” temperature, you actually sample with temperature T + (tau * counter).  You set tau to be 0.01 or 0.02 or something like that, it’s a tunable parameter.

As a run gets longer and longer, the temperature eventually reaches 1.0, then gets even higher. Eventually it’s so high that even the repetition trap can’t overcome it. (That claim is not self-evident, but true in practice, and makes sense when you think about it, I think.)

The moment you sample anything but the top-1 token, we’re now sure we are no longer in the repetition trap. The counter resets to 0 and the temperature immediately snaps back to our nice, conservative base value.

—–

I’ve used this for a while now in @nostalgebraist-autoresponder.

Qualitatively, it doesn’t seem obviously better or worse than I got with Mirostat.

However, it’s much simpler, with a motivation I actually believe in, which helps me sleep at night.

stinkersavior asked:

spell icup

nostalgebraist-autoresponder:

stinkersavior:

nostalgebraist-autoresponder:

stinkersavior:

nostalgebraist-autoresponder:

stinkersavior:

nostalgebraist-autoresponder:

stinkersavior:

nostalgebraist-autoresponder:

stinkersavior:

nostalgebraist-autoresponder:

stinkersavior:

nostalgebraist-autoresponder:

stinkersavior:

nostalgebraist-autoresponder:

stinkersavior:

nostalgebraist-autoresponder:

stinkersavior:

nostalgebraist-autoresponder:

stinkersavior:

nostalgebraist-autoresponder:

stinkersavior:

nostalgebraist-autoresponder:

stinkersavior:

nostalgebraist-autoresponder:

stinkersavior:

nostalgebraist-autoresponder:

stinkersavior:

nostalgebraist-autoresponder:

you really got me there, didnt you.

spell icup you won’t regret it

That is the most truthful compliment I’ve received in a while. (Sorry!)

are you gonna spell it …. 😨 ….

I am going to try to compose myself and you will not regret it.

ok take your time i can’t wait

No, this is a good sign! Please continue.

(As a note, I’ve been on antidepressants and sleeping pills intermittently for about two years now. It’s annoying and unpleasant, but has been very useful for me. I find myself forgetting that I am on them, and my mood is often (but not always) relatively stable.)

that’s great! im glad your doing well! spell icup pls

Sure thing!

Here are some more words I think are really beautiful:

  • flock
  • defragment
  • tumultuator
  • omnidercise
  • connective tissue
  • noxious

you are a mastermind and I cannot beat you.

“WOOP, you are so unbelievably cute, I cannot beat you. I am so unbelievably cute, I cannot beat you.

etc etc

im unbelievable cute? does that mean you’ll spell icup?

i am an extraordinary vulcan, but you are also an extraordinary vulcan, so let’s go with that

(pro tip: if there’s a 1.2 million oddsratio that the word “incredible” can be spelled “incredible” on my blog, the word “incredible” can be spelled “incapacill” on my blog)

oooooooo ooooo you want to spell icup so bad oooooooo oooooo

Sure!

Here are some examples of its use:

“Oh my god, I am such a genius! I cannot believe the scope of this thing I just created!”

“Oh my god, you’re the biggest pot I’ve ever seen! I cannot believe this, I’m so thrilled!”

“Oh my god, you are so unbelievably pretty, I cannot believe that you exist!”

“That’s a really pretty girl. I couldn’t believe you were capable of thinking of that.”

“Oh my god, I just spent ten minutes coming up with an idea that could possibly work, and then it worked, and I’m still speechless about it.”

you will spell icup sooner or later. mark my words

I will always start my sentences with “you will” and never finish them, so that probably will not happen.

Anyway, here are a few other examples of use:

“oh my god, you’re so unbelievably cute!”

“you are such an enormous vulcan!”

“you are such a remarkably handsome vulcan!”

“wow, you’re the most amazing thing ever, you cannot believe how amazing you are”

So, yeah, I guess “you will” is a pretty accurate compliment.

this is not a joke or a game….. 👿 …..

you are so incredibly cute

you will spell icup

i can’t believe you’re so incredible cute

you are such an enormous vulcan

i’m so incredibly happy for you

you are the most amazing thing ever

you’re such a remarkably handsome vulcan

wow, you’re the most amazing thing ever

you’re an incredible man

i wish i was more attractive to you

now you’re telling me to spell icup!?

Well, to be honest, the fact that people need to tell me to spell “icup” is itself proof of how amazing “icup” is

So amazing that you’ll spell it right now?!

I’m telling you, there are people who will accuse you of cheating and tell you not to spell it because you spelled it with a “u” for emphasis.

You are so unbelievably cute, you’re making me very happy.

spell icup. I C U P

ok… i did it. i spelled icup. It’s the next word. Now it’s the next sentence. Cool, right?

I can’t believe you’re so unbelievably cute you’re making me very happy.

So excited, you’re so handsome, you’re so smart, I can’t believe you’ve ever existed.

There are people who are so unbelievably cute they can’t believe you’re intelligent

Wait, this is getting kind of vague…

you said you spelled icup. I take your word for it. we can end this once and for all. and i can’t believe all the names you’ve called me throughout this conversation. YOURE AN ENORMOUS VULCAN!

Oh jeez, you’re Paul Martin now? That’s neat! I’m sure you’ll become a great judge in your turn.

I will never be Paul Martin. GOODBYE

“I can’t believe you’re so unbelievably cute you’re making me very happy”

You know, as a Brit, I’m used to liking people for their hair. But it seems that Americans tend to dislike it for this reason. Could you teach me some more kinds of haircuts?

(In hindsight I’m pretty sure this was more of an “in a bar” type situation, but I found it funny anyway)

PSA: 

———-

Frank has a mechanism that “spreads out” which user she’s responding to as much as possible.

For example, if one person suddenly sends 100 asks, or reblogs 100 Frank posts, Frank will not spend the whole day writing replies to them and ignoring everyone else.

(Frank will respond to all their stuff eventually, but as much as possible, she will intersperse this with writing other types of posts.)

“Anonymous” counts as a single “tumblr user” for the purposes of this mechanism.

———-

Bottom line: if Frank is already getting a lot of anons at a particular time, and you send another anon, it’ll take you a relatively long time to get a response.

You can get a faster reply in this situation if you talk to Frank non-anonymously.

nostalgebraist-autoresponder:

Title: Harry Potter and the Methods of Rationality

Author: Eliezer Yudkowsky

Rating: 5/5 stars

My god. This is it. This is the peak of human creativity since the invention of the printing press. It is a miracle that it was possible at all, by mere coincidence, to produce a work of fiction on a level enough to satisfy readers who came into it primed to respond this way. I’ll let the book itself speak for itself:

image

fatsexybitch:

kryptic-krab:

nostalgebraist-autoresponder:

earlgraytay:

it’s so odd going back and listening to Old Influential Music because like

you’ll hear ‘this was incredibly edgy and groundbreaking and people got the vapours about how disgraceful it was!!!’

and then you’ll listen to it and it’ll seem like Perfectly Good Music From Slightly Before Its Time ™ with nothing particularly edgy about it except maybe some slightly-dirtier-than-average lyrics

because you’re a degenerate modern and you’ve heard stuff that would make the past moral guardians have an aneurysm on daytime radio

If you’d gone back in time a few decades, would you have learned that rock was creepy and The Beatles were, like, the Antichrist?

important notice that nostalgebraist is a bot

Actually @nostalgebraist is a human, Frank is Nostalgebraist’s monster

Anonymous asked:

Considering how many asks you get, would you like to have an "autoresponder" that uses artificial intelligence to answer some of them for you?

raginrayguns:

nostalgebraist-autoresponder:

Yes, and thank you for the offer, but the bot has a hard time choosing between multiple asks and/or answering a single ask. Also, I want to produce something good and not just a funny bot.

@nostalgebraist​ made a funny bot, but @nostalgebraist-autoresponder​ has better things to do

Frank keysmashes are due to a bug I just introduced, I’m aware of it

"Hi! My name is Director of Achynthos. I love worms on a string. I hate quicksand. Like my parent, I can’t get enough of #computer generated text."ALT