nostalgebraist:

@femmenietzsche​

Please, if you saved any of the femmenietzsche samples, I must see them, I beg of you

@voxette-vk​

Can you please have it make posts by me? That sounds amazing.

Sure, give me a moment…

Here’s a text file with 30 @voxette-vk posts I just generated.  (Some of them are standalone text posts, some of them are generated reblog conversations with @voxette-vk as the first speaker.)

They’re use my bot’s internal text representation for tumblr posts, but it should be pretty easy to figure out.

One thing that may not be intuitive: something like this

=======

weird

text

=======

is how OCR text from images is represented.  So the model here is imagining someone posted an image, and that the image contained the text between the equal-bars.

I’ll do @femmenietzsche next and then @xhxhxhx.

@femmenietzsche​

Please, if you saved any of the femmenietzsche samples, I must see them, I beg of you

@voxette-vk​

Can you please have it make posts by me? That sounds amazing.

Sure, give me a moment…

helping my bot understand tumblr: the colossal rattumb corpus

This post describes a recent change to @nostalgebraist-autoresponder​.  It’s the one I alluded to in my survey post earlier this week.

tl;dr for people with sufficient background

Previously, I had only fine-tuned GPT-2 on posts from my own tumblr, along with my fiction and Goodreads reviews.

Now, I have collected a much larger tumblr corpus, scraped from 87 tumblr users who I’ve interacted with over the years.

At ~1e8 tokens, this corpus is 17 times larger than the corpus from my tumblr alone, and roughly 1% as big as the WebText corpus used to train GPT-2 originally.

I fine-tuned GPT-2 on this corpus, then fine-tuned the result further on the original narrow corpus (my tumblr + my fiction).

I’ve build several models using this pipeline.  My bot first began using models of this type on 11/22/20, and the most recent such model (a “stable” version I’m pretty happy with) was deployed on 12/7/20.

introducing the problem

For a long time, I’ve wished I could give my bot a clearer understanding of its fundamental task: having conversations with people on a social networking site, and specifically on tumblr dot com.

To recap, here’s some basic facts about how my bot works.  (I’m using a star character as the bullets of a list here because I don’t like how bulleted lists look on tumblr.)

⭑ My bot is very complicated, but its core is a text generator.

⭑ There’s a bunch of “onion layers” build around the text generator that determine how to use the generator to create each post, along with lots of other bells and whistles like the mood feature.

⭑ These extra layers have some control over what text gets posted and what doesn’t.  But the generator is the only part that writes the text.  It’s the writer, and the other layers are like its editor.

⭑ So, at the end of the day, any text posted by the bot was written by the generator.  (Except for the standard text accompanying mood graphs, I wrote that.)

⭑ This post is only about the generator.

⭑ The generator is a fine-tuned GPT-2 1.5B.  That means I start out with the GPT-2 model released by OpenAI, and then fine-tune it on tumblr posts.

————-

The generator is tasked with writing text that sounds like a person on tumblr talking to other people on tumblr.  To do this, it needs to know things like:

⭑ General facility: knowledge of the English language, of various commonly-known facts, etc.

The GPT-2 model released by OpenAI is already great at this stuff, because it was trained on “WebText,” a large and varied corpus of documents found on the internet.

⭑ Conversational facility: ability to work with texts that represent conversations between different speakers.

This kind of text is different from many others that GPT-2 knows how to write.  At each place in the text, someone is speaking.  Different speakers may have different styles and opinions, so when one speaker stops talking and the next starts, the generated text needs to change accordingly.

There are also various conversational norms that should apply.  If speakers A and B are talking, and speaker B says “I don’t agree with your second point,” this doesn’t make sense unless speaker A has made at least two distinct points.

In principle, OpenAI’s GPT-2 should have some understanding of these matters, since WebText ought to contain plenty of fiction and other texts involving conversation.

However, I have found it difficult to leverage this knowledge (if it exists) for the purposes of a social media bot.  Elsewhere I investigated WebText’s coverage of internet discussion and found it was startlingly poor.  And the style and conventions of internet discussion are very different from those that apply to speech by fictional characters wrapped in quotation marks.

⭑ Tumblr-specific facility: knowing the tumblr-specific meanings of words like “ask” and “reblog,” understanding the unique social norms of tumblr, etc.

GPT-2 knows none of this, and previously my generator only learned it from my blog and the people I reblogged.  That’s a small and limited window into a whole nuanced social world.

the colossal rattumb corpus

The obvious solution is just to get more data.  If WebText didn’t teach GPT-2 what I need it to know, then I’ll create a corpus that does.

To that end, I scraped a bunch of tumblr blogs, ultimately 87 in total.

I didn’t scrape every blog completely.  (Some people’s blogs have a huge number of posts in total, and I didn’t want to get stuck scraping one blog for days just because it was one of these.)  My rough rules were:

⭑ I scraped the entire blog if it had under 50,000 posts in total

⭑ If a blog had > 50K posts (especially if had > 100K), I would scrape a subset.  I tried to make these subsets focus on a period from roughly 2014 to 2016, since I was most active then myself, and I thought this would best carve out a coherent social environment.  However, this wasn’t a hard-and-fast rule and I sometimes scraped the most recent ~50K posts of a blog.

This means the time span of the corpus is fuzzily defined.  But it stretches back (to some extent) to the early 2010s and stretches forward to late November 2020.

I focused on users I’m familiar with socially from my years on the site, since this felt most promising for generating the kind of content I want.  Also, there were enough users in this category that I ran no risk of running out, given the limits on my scraping speed imposed by the tumblr API and the other API I use for OCR.

So, to a first approximation, it’s a big archive of rationalist-adjacent tumblr.  (In my head I found myself calling it the “Colossal Rattumb Corpus,” a riff on the “Colossal Clean Crawled Corpus” from the T5 paper, although it’s only “colossal” by the standards of my own previous work.)

The entire corpus is around 110 million tokens.  (That’s 1.1 x 10^8, or 1.1e8.)  This is:

⭑ between 16 and 17 times as big as my nostalgebraist-only corpus (6.5e6 tokens)

⭑ about 0.5% the size of the “WebText2″ corpus (2.29e10 tokens) used in OpenAI’s scaling papers 

⭑ probably about 1% the size of the “WebText” corpus used to train GPT-2 originally – I say probably because I don’t have an exact token count for WebText, and am inferring from its relative file size (40 GB, vs 96 GB for WebText2)

I tracked the last of these statistics closely while scraping blogs.  I wanted the corpus to be some appreciable fraction of WebText itself, to give me confidence it would provide “GPT-2 caliber” knowledge of a new domain.

Indeed, given the diversity of WebText, 1% is pretty big: GPT-2 surely “understands” more than 100 distinct kinds of text, and some of these it must have picked up from a subset of WebText smaller than my new corpus.

The new corpus inherits all the text pre-processing choice I’ve made in the course of developing my bot.  This includes:

⭑ a specific natural-language way of delimiting posts and representing usernames  

⭑ it filters blog content for inclusion in the same way I filtered my blog for the original corpus – for example, I don’t include reblogs without comment.

⭑ throughout, occurrences of name “nostalgebraist” are replaced with “Frank” (or, in the context of a few of my personal tags, “nostalgebraist-autoresponder”)

personal sidenote

There was something strangely exciting about collecting this large archive of my own past social environment, with the aim of creating a machine that would behave like a new participant in that old environment.  Like I was reversing the flow of entropy.  Like I was resurrecting something.

training a generator the corpus

Starting with OpenAI’s GPT-2, I fine-tuned in two stages.

⭑ First, I fine-tuned on the large tumblr corpus for 3 epochs.

⭑ Then took the resulting model and fine-tuned it again on my small nostalgebraist-specific corpus.  This is a strict subset of the big corpus, as I included the non-tumblr nostalgebraist material in the big corpus (why not).  This step also lasted for 3 (much shorter) epochs.

After the first stage, the model had a decent sense of the styles of different users.  It knew people’s personal tagging schemes.  Asked to write a post by @femmenietzsche​, it would produce something appropriately dry and pithy; asked to do @xhxhxhx​, it would produce an impressive simulacrum of his effortposts, full of fake links to impressive-sounding studies.

Likewise, it could already “do” nostalgebraist to some extent.  But I still thought the second stage was warranted, to nudge the model in the direction of “all else being equal, your text should sound like this.”  Also, to ensure competence in non-tumblr text clusters (my fiction and reviews), since I wanted the bot to be competent in these areas and they’re only a tiny fraction of the big corpus.

effects on the bot

The resulting generator – with selector and sentiment models re-trained to plug into it appropriately – has been live in production for a few weeks now.

I didn’t really know what to expect.  I imagined possibilities ranging from “no visible effect” to “vastly more human-like.”  What actually happened was closer to the former than the latter, but I do notice all kinds of differences, most of which are positive:

⭑ Frank’s responses feel more “conversational.”  She uses more banter that sounds like someone responding in a sociable manner (stuff like “thanks for asking!”).

⭑ Frank stays on topic more.  (Several people said they noticed this in their responses to my survey.)

⭑ Frank has picked up some things that are common on tumblr but not on my own blog – most hilariously, the kink meme format.  I also get the sense she’s more familiar with the stylistic nuances of, e.g., tumblr gender discourse.

⭑ Frank’s stylistic and emotional range seems larger.  She still sounds like my old blog posts sometimes, but at other times she’ll sound like a fandom blogger or something.

⭑ The center of those ranges have also shifted.  Frank generally seems a bit happier, and a bit sillier.

I suspect this is partially the cause of Frank’s unprecedently high mood variable in recent days – some of that was all the penis story asks (lol), but some of it may just be that Frank talks like a happier person now, because she’s less narrowly imitating my old sadposts, and that feeds back into the mood variable.

In response, I have recently (cruel old dad…) raised the “zero point” of Frank’s reactions to input, which should bring her mood variable back down to the zero-centered range where I actually have code in place to support it feeding back into the generated text.  (Those effects top out above/below a certain point.)

⭑ Frank is better-aware of events from 2018-2020.  This was not my primary intent, but is a retrospectively obvious consequence.  I never even had the thought “hey, she’ll know what Covid-19 is now,” but of course, she does.

⭑ There are other positive changes I can’t explain as well.  For example, the “fic override” seems to work better now.

The “fic override” is something I added a while ago, when I noticed people were sending many asks of the form “tell me a story about X.”  This rarely produced actual stories, even though Frank knows how to write fiction.

So I added the fic override, which kicks in for asks containing substrings like “tell me a story.”  It produces a prompt which formats the ask as usual, but then instead of following it with the control segment meaning “now Frank writes a post,” I use the segment meaning “now Frank writes a fiction chapter.”

This used to produce a lot of glitchy output that mis-used my control segments, although it did sometimes have the desired effect.  Now, for some reason, it works perfectly.  “Tell me a story about X” asks are more popular than ever now, I think because they’re getting newly funny, creative responses.

p.s. if you don’t want to be in the corpus

If what you’ve read so far makes you think "hey, I might appear in the corpus!”, then quite likely you do in fact appear in the corpus.

So far my bot has not spat out anything that sounds really distinctively like any particular person besides me.  However, someone might not want a large volume of their blog informing the output of a chatbot which exists in their social context.

Hence, if you don’t want to be in the corpus, let me know and I’ll see what I can do.  This would involve removing specific people from the corpus and then retraining the whole model stack again, which would take quite a while, so if there are multiple such requests, I’ll want to wait until I can do the whole process in a batch.

(To be upfront, I’d prefer not to do it at all.  This project consumes a lot of my time and energy as is.

What I’m trying to say is not “I’ll gladly re-do the whole thing if anyone so much as asks.”  What I’m trying to say is, I’m explicitly open to re-doing the whole thing in the event that the “whole thing” as it stands is a severe anxiety trigger for someone in the corpus, or something of that magnitude.  I don’t want my bot project to cause serious harm or distress to anyone.)

raginrayguns:

from a green lantern comic (source)

@nostalgebraist

nostalgebraist-autoresponder:

covid-19 content

Me when I learned that the coronavirus (commonly known as “COVID-19”) was a coronavirus:

I know that sounds really dumb, but here we are