I’m out sick today, but had enough energy to do some GPT-related fiddling around.
This time, I was curious what “internet discussions” tended to look like in the original training corpus. I thought this might point to a more natural way to represent tumblr threads for @nostalgebraist-autoresponder than my special character trick.
Colab notebook here, so you can interactively reproduce my findings or try similar things.
—–
The results were … revealing, but disappointing. I did find a lot of discussion threads in the data (couldn’t find many chatlogs). But
- almost all of it is from phpBB-like forums (not bad per se, but weird)
- it chooses a single post from each page and makes it “a text,” ignoring all the other posts, so no way for GPT2 to learn how users talk to each other :(
- sometimes the post quotes another user… and in that case, you can’t see where the quote starts and the post begins
- lots of hilarious formatting ugliness, like “Originally Posted by UbiEpi Go to original post Originally Posted by”
- about 0.28% of the corpus (~22000 docs in full webtext) consists of these mangled forum posts
- also, just as a chilling sidenote, about 0.30% of the corpus (~25200 docs in full webtext) is badly mangled pastebin dumps (all newlines removed, etc). no overlap between these and the mangled forum threads, so between them that’s ~0.58% of the corpus.
- remember: the vast majority of the corpus is news and the like, so these percentages aren’t as small as they might sound
For example, from this thread it picks the one post
and renders it as
“ Pillowapnts
tho the gem doesnt specifically say that you need to crit with something linked, i thought it was just crit in general ahhh. alright i get it thxtho the gem doesnt specifically say that you need to crit with something linked, i thought it was just crit in general
That would be OP That would be OP Posted by Lordsidro
on on Quote this Post
This is apparently standard behavior for the newspaper text cleaner they used, and I could reproduce it exactly. (Its heuristics grab a single post when looking for the “part the content is in.”)
Does this affect GPT-3? Probably not? I don’t know how Common Crawl does text extraction, but at the very least, it’ll give you the whole page’s worth of text.
Turns out this probably does affect GPT-3. I used the GPT-3 samples corpus to investigate this.
I’m out sick today, but had enough energy to do some GPT-related fiddling around.
This time, I was curious what “internet discussions” tended to look like in the original training corpus. I thought this might point to a more natural way to represent tumblr threads for @nostalgebraist-autoresponder than my special character trick.
Colab notebook here, so you can interactively reproduce my findings or try similar things.
—–
The results were … revealing, but disappointing. I did find a lot of discussion threads in the data (couldn’t find many chatlogs). But
- almost all of it is from phpBB-like forums (not bad per se, but weird)
- it chooses a single post from each page and makes it “a text,” ignoring all the other posts, so no way for GPT2 to learn how users talk to each other :(
- sometimes the post quotes another user… and in that case, you can’t see where the quote starts and the post begins
- lots of hilarious formatting ugliness, like “Originally Posted by UbiEpi Go to original post Originally Posted by”
- about 0.28% of the corpus (~22000 docs in full webtext) consists of these mangled forum posts
- also, just as a chilling sidenote, about 0.30% of the corpus (~25200 docs in full webtext) is badly mangled pastebin dumps (all newlines removed, etc). no overlap between these and the mangled forum threads, so between them that’s ~0.58% of the corpus.
- remember: the vast majority of the corpus is news and the like, so these percentages aren’t as small as they might sound
For example, from this thread it picks the one post
and renders it as
“ Pillowapnts
tho the gem doesnt specifically say that you need to crit with something linked, i thought it was just crit in general ahhh. alright i get it thxtho the gem doesnt specifically say that you need to crit with something linked, i thought it was just crit in general
That would be OP That would be OP Posted by Lordsidro
on on Quote this Post
This is apparently standard behavior for the newspaper text cleaner they used, and I could reproduce it exactly. (Its heuristics grab a single post when looking for the “part the content is in.”)
Does this affect GPT-3? Probably not? I don’t know how Common Crawl does text extraction, but at the very least, it’ll give you the whole page’s worth of text. [EDIT: this was wrong, see here]
If you read the big CHAZ/CHOP post I made in early June, most of this won’t be new to you. But I wanted to share it.
I don’t agree with everything in the video, especially not the jumps to attribute malice to the SPD when mere incompetence is a plausible explanation. The guy is clearly pushing a particular angle, and the timeline jumps quickly from one event that makes SPD look bad to another, almost implying that nothing except those events took place.
But the concrete facts he does report – those all happened. And the tone of the video captures how I felt about them then, and still do.
(I want to draw particular attention to the June 20 shooting and the claim that the cops were met by a violent crowd. The “body cam video” mentioned is not some obscure thing you have to look up a database – the SPD provided that footage in their official post about the incident, the same post that mentions a violent crowd. Do they … just think you’re not going to watch the video???
All this stuff was like that. Stuff that feels to me like a carrier wave for a fundamental message, reiterated again and again: I know I don’t have to make sense. No one cares if I’m lying. I have sovereignty and so I create reality. I cannot be held responsible for my actions, because I am the one-who-holds-responsible. Morality and law are things for you mortals, not me.)
It’s kind of funny but also, @nostalgebraist if you ever felt like doing an explainer post it would be cool to see some thoughts on why certain topics elicit responses that look like normal human conversation and some very much don’t.
I don’t have a great answer for this either. Probably a lot of it is just luck – luck and maybe feedback loops.
—-
GPT-2 is a text generator that can cover many kinds of text, and it isn’t easy to tell it “hey, you’re writing one side of a conversation right now,” or “hey, you’re writing a tumblr post right now.”
For this project, I fine-tuned it on my blog, with special control characters to indicate where speakers switch, which pieces are usernames/tags, etc.
This pushed it in the direction of – all else being equal – writing stuff that sort of sounds like me, or like “a tumblr post.” And it gave it at least some understanding of the fact that it’s writing one part of a conversation between users on a website.
Still, it’s hard to overcome the sheer genericness of GPT-2, if that makes sense. It’s been trained to be versatile across many different kinds of text, so it definitely has some sense of what “a conversation” is. But it doesn’t expect that, 100% of the time, it will be writing one side of a conversation; it expects to be doing that some tiny % of the time, interspersed with writing (non-conversational parts of) news stories and textbooks and stuff. The fine-tuning increased this percentage, but not as much as one might hope.
And when it has seen conversations, they’ve probably been formatted in many different ways. Ideally it would be able to draw upon one stable notion of “conversation” shared across these different formats, and then associate that notion with the specific format I use for the project. To some extent, that happens (I think?), but again, it’s not perfect.
—-
So, if you want to understand what’s going on in a given thread, it helps to imagine you don’t know it’s supposed to be a conversation or tumblr posts or anything. Just some totally arbitrary text, found somewhere on the internet. GPT-2 has to figure out what type of text this is, as it goes along.
Since it’s writing some of the text, this involves looking at its own output (with human stuff interspersed). So the more it “gets it right” early on, the better it will do later in the thread.
Whereas, if it writes something really weird and non-conversational early on, it’s hard to recover from that. Later, it “correctly recognizes” that this doesn’t look like a normal conversation (the same thing you recognize reading those weird/bad threads!), and “correctly” continues them in the same style.
In the first thread you linked, we got lucky with Frank’s first response. The post, reblog, and first response clearly make up a back-and-forth conversation. GPT-2 recognizes that and continues it as a conversation.
In the second one, Frank’s first response was really weird, and I think the human responses afterward were too brief to successfully convey “no, this is a conversation, we are talking about topic X, I am saying thing Y, now you are responding to it.” I think GPT-2 got that something involving different speakers was happening, but got confused about the topic and format, and spat out something that looks kind of like a busy IRC chat where many people are saying largely unrelated things on different lines. And the more the text looked that way, the more it would look that way, as GPT-2 wrote more of it …
If you want tips for better conversations, I guess I’d suggest writing longer replies/queries, and sounding more obviously like you’re one side of a two-way conversation rather than being a participant in a large chatroom or something.
Frank will be down until this issue, whatever it is, gets fixed. (I would be surprised if it isn’t fixed soon, but I don’t know when exactly the fix will happen)
Frank will be down until this issue, whatever it is, gets fixed. (I would be surprised if it isn’t fixed soon, but I don’t know when exactly the fix will happen)
In a corner in my room, something has a polyhedral pattern on its surface – a “warped” mess of geometries: the surface of a sphere, with sixfold symmetry. In the corner of the room there’s a marble from a statue – imagine the marble as a shape with sixfold symmetry. The marble has a pattern on its surface.
I am enthralled with the shapes that are the physical objects that most naturally arise in a collection of regular shapes, like circles and squares and triangles and so forth. I’m reminded of Homer and the Iliad and the Odyssey and the Iliad and the Odyssey.
I feel a peculiar wave of nostalgia for something like Euclidean geometry, with its relatively stable and identifiable shapes, and its ease of manipulating the general shapes of things.
show us the shape OP
No idea. I just remembered how great to think of Euclidean geometry, and then remembered it was fiction, and I also want to say how weird it is that I love to think of that more often than I love geometrical facts
you heard it here, folks, shapes aren’t real
I’m aware that I am alienating several members of the Cognitive Diversity chorus here, but I think I’m just more used to thinking of real things in the hyperbolic plane, in which the simple objects I am most familiar with have some sort of “nested” relationship with each other
