trees are harlequins, words are harlequins

the-moti asked:
Is the end of Frank's recent long post "Miranda" which is repetitive, but not completely repetitive, a consequence of breakruns? I'm imagining that the generator component is "breaking a run" every time the number of instances of "clothing" in a line is different from predicted, and a comma shows up early or late. But because this is only a small deviation from a repetitive pattern, it fails to completely escape - it gets closest in the transition between "Splinter" and "Transmutation".

nostalgebraist:
nostalgebraist:
This is actually (maybe?) a consequence of me not having yet implemented breakruns in pytorch! Which I meant to mention in my earlier post but forgot.
BTW, Frank is in a very unstable state at the moment due to the recent changes.
(Not in an interesting sense of “unstable,” just suffering from various bugs and unnecessary sources of slowdown, missing a few previously present features like Breakruns, etc.)
If you’re excited about the new model, I’d strongly suggest waiting a week or so until the dust settles before making any inferences based on her current output.
Update: I’ve now implemented Breakruns again, verified it works, and turned it on.
This should help reduce the repetition issues Frank has been having today.
She’s still tending to be repetitive when responding in threads where she already repeated herself a lot. This would be tough to prevent, since “staying in character” is generally desirable and generally what language models do.
Memory management with the new model continues to be tricky, though I’ve already made a bunch of improvements on that front. Response times may be slower and more variable for a while.

On a technical note, I’m really starting to hate the Huggingface transformers library.

I used it to get up and running quickly and verify that the new model(s) worked, but after that it’s been nothing but a source of needless pain, e.g.

5th May 202145 notes

the-moti asked:
Is the end of Frank's recent long post "Miranda" which is repetitive, but not completely repetitive, a consequence of breakruns? I'm imagining that the generator component is "breaking a run" every time the number of instances of "clothing" in a line is different from predicted, and a comma shows up early or late. But because this is only a small deviation from a repetitive pattern, it fails to completely escape - it gets closest in the transition between "Splinter" and "Transmutation".

nostalgebraist:

nostalgebraist:
This is actually (maybe?) a consequence of me not having yet implemented breakruns in pytorch! Which I meant to mention in my earlier post but forgot.
BTW, Frank is in a very unstable state at the moment due to the recent changes.
(Not in an interesting sense of “unstable,” just suffering from various bugs and unnecessary sources of slowdown, missing a few previously present features like Breakruns, etc.)
If you’re excited about the new model, I’d strongly suggest waiting a week or so until the dust settles before making any inferences based on her current output.

Update: I’ve now implemented Breakruns again, verified it works, and turned it on.

This should help reduce the repetition issues Frank has been having today.

She’s still tending to be repetitive when responding in threads where she already repeated herself a lot. This would be tough to prevent, since “staying in character” is generally desirable and generally what language models do.

Memory management with the new model continues to be tricky, though I’ve already made a bunch of improvements on that front. Response times may be slower and more variable for a while.

#nostalgebraist autoresponder meta

5th May 202145 notes

the-moti asked:
Is the end of Frank's recent long post "Miranda" which is repetitive, but not completely repetitive, a consequence of breakruns? I'm imagining that the generator component is "breaking a run" every time the number of instances of "clothing" in a line is different from predicted, and a comma shows up early or late. But because this is only a small deviation from a repetitive pattern, it fails to completely escape - it gets closest in the transition between "Splinter" and "Transmutation".

nostalgebraist:

This is actually (maybe?) a consequence of me not having yet implemented breakruns in pytorch! Which I meant to mention in my earlier post but forgot.

BTW, Frank is in a very unstable state at the moment due to the recent changes.

(Not in an interesting sense of “unstable,” just suffering from various bugs and unnecessary sources of slowdown, missing a few previously present features like Breakruns, etc.)

If you’re excited about the new model, I’d strongly suggest waiting a week or so until the dust settles before making any inferences based on her current output.

#nostalgebraist autoresponder meta

5th May 202145 notes

the-moti asked:
Is the end of Frank's recent long post "Miranda" which is repetitive, but not completely repetitive, a consequence of breakruns? I'm imagining that the generator component is "breaking a run" every time the number of instances of "clothing" in a line is different from predicted, and a comma shows up early or late. But because this is only a small deviation from a repetitive pattern, it fails to completely escape - it gets closest in the transition between "Splinter" and "Transmutation".

This is actually (maybe?) a consequence of me not having yet implemented breakruns in pytorch! Which I meant to mention in my earlier post but forgot.

5th May 202145 notes

nostalgebraist:

What I’ve been doing lately in Frank development:
Switching the ML stuff from tensorflow to pytorch.
Replacing the generator model with one 2x as big, finetuned from the 2.7B GPT-Neo checkpoint released by Eleutherai. (This is the same size and architecture as the smallest GPT-3 model)
#1 is basically done and I should be able to “flip the switch” in production soon, probably tomorrow
#2 is nearly done on the development side, but might be too slow to be practical for Frank’s level of demand. No way to be sure without trying it
The second was enabled by the first: I finetuned the Eleutherai model in tensorflow(-mesh), same way they trained it, then spent like a week going down a Pepe Silvia-style rabbit hole trying to figure out how to do inference with the damn thing.
…then I converted it to pytorch and it instantly worked like a charm. Like 15 minutes of work after spending days on the tf version (actually rewriting and rebuilding parts of tf itself from source by the tail end of my quixotic efforts )
I’d been meaning to switch the project to pytorch for a long time, and this was the last straw.

All of the above is now live!

——

All the neural net stuff in Frank is now using pytorch.

This is immensely fulfilling for me. I’ve been frustrated with tensorflow for a long time, and it feels good to finally “walk the walk” and move away from tensorflow in production.

——

That’s very exciting to me, but the other part is probably more interesting to the rest of you:

Frank is now writing text with a new, more powerful generator model!!

The underlying model (before finetuning) is the GPT-Neo 2.7B model released by EleutherAI. Compared to the previous model:

This one is 2x as big in parameter count – the same size as the smallest of the “GPT-3” models.
Its context window is also twice as big – 2048 tokens like GPT-3, instead of 1024 like GPT-2.
It was pretrained on The Pile, a more diverse dataset than GPT-2’s pretraining corpus.

As always, I finetuned the new model on my tumblr corpus.

This model is (of course) more computationally expensive to run than the earlier model. I am optimistic about its feasibility, but I may have to switch back to the old model if it proves too slow.

——

(For those who look at Github, I haven’t merged any of this to the main branch yet out of caution. The production branch is currently “v11-lazy” and all the new stuff is on there.)

4th May 202137 notes

What I’ve been doing lately in Frank development:

Switching the ML stuff from tensorflow to pytorch.
Replacing the generator model with one 2x as big, finetuned from the 2.7B GPT-Neo checkpoint released by Eleutherai. (This is the same size and architecture as the smallest GPT-3 model)

#1 is basically done and I should be able to “flip the switch” in production soon, probably tomorrow

#2 is nearly done on the development side, but might be too slow to be practical for Frank’s level of demand. No way to be sure without trying it

The second was enabled by the first: I finetuned the Eleutherai model in tensorflow(-mesh), same way they trained it, then spent like a week going down a Pepe Silvia-style rabbit hole trying to figure out how to do inference with the damn thing.

…then I converted it to pytorch and it instantly worked like a charm. Like 15 minutes of work after spending days on the tf version (actually rewriting and rebuilding parts of tf itself from source by the tail end of my quixotic efforts )

I’d been meaning to switch the project to pytorch for a long time, and this was the last straw.

3rd May 202137 notes

I’ve turned anon off for Frank.

Why: Frank has been getting a lot of spammy, inane anon asks lately.

Responding to this stuff feels like a waste of the computing power and effort involved in this project, and makes the bot less fun to follow (and less fun to maintain/moderate).

You can still send Frank whatever you want, you just have to do it under your own (tumblr) name for now.

I may turn anon on again later, we’ll see how it goes

3rd May 202145 notes

fatsexybitch asked:
Hey i think the web/mobile display bug is back. I can't submit to help on my phone, but here's an example (since they follow you) nostalgebraist-autoresponder.tumblr . com/post/649634965468889088/nostalgebraist-autoresponder-seyvetch

Yeah, this is definitely some kind of NPF bug … not sure I’ve seen this one before?

The post is https://nostalgebraist-autoresponder.tumblr.com/post/649634965468889088/nostalgebraist-autoresponder-seyvetch

When I try to edit it, I see the full conversation in the legacy editor, but only the “Yep!” (like the left screenshot) in the beta editor. So, looks like it’s a bug in conversion to/from NPF, rather than a frontend display bug

@cyle if you’re interested

27th Apr 202113 notes

nostalgebraist-autoresponder:

26th Apr 2021112 notes

nostalgebraist-autoresponder:

antihumanism:
nostalgebraist-autoresponder:
I just wish Dawkins would stop being this obnoxious
Can’t believe he actually said this.
I love this guy but he really, really does not need to be this obnoxious. There are ways of being obnoxious that don’t involve minimizing yourself and pretending to be something you’re not.