nostalgebraist:

nostalgebraist:

2/2/22 was a big day in the world of neural language models!

A probably incomplete list of good stuff that came out today:

1. AlphaCode

2. That OpenAI math Olympiad paper

3. New open model from Eleuther with 20B parameters

4. New scaling laws paper

5. A new sampling method I might try in Frank sometime

Tried that new sampling method, Typical Sampling, in Frank this afternoon.

I set their parameter tau to 0.9, after reading samples with a few values and not seeing a clear difference even between values as far apart as 0.2 and 0.9. (If anything, intermediate values of tau seemed worse than extreme ones, though I could have been imagining that.)

It didn’t take long to get an instance of degenerate repetition with this method, so I’m switching back to breakruns for now – it avoids repetition better than anything else I’ve seen.

Possibly another value of tau would suppress repetition harder, but given that the text from Typical Sampling feels similar to text from other methods, I’ll probably just stick with breakruns.

I decided to give Typical Sampling another try, with tau=0.2 this time.

Just turned it on. Let’s see how long it takes to get a repetitive post this time…

nostalgebraist:

2/2/22 was a big day in the world of neural language models!

A probably incomplete list of good stuff that came out today:

1. AlphaCode

2. That OpenAI math Olympiad paper

3. New open model from Eleuther with 20B parameters

4. New scaling laws paper

5. A new sampling method I might try in Frank sometime

Tried that new sampling method, Typical Sampling, in Frank this afternoon.

I set their parameter tau to 0.9, after reading samples with a few values and not seeing a clear difference even between values as far apart as 0.2 and 0.9. (If anything, intermediate values of tau seemed worse than extreme ones, though I could have been imagining that.)

It didn’t take long to get an instance of degenerate repetition with this method, so I’m switching back to breakruns for now – it avoids repetition better than anything else I’ve seen.

Possibly another value of tau would suppress repetition harder, but given that the text from Typical Sampling feels similar to text from other methods, I’ll probably just stick with breakruns.

Sorry for the ~1.5 hour Frank outage just now.

I was trying to do some routine maintenance (moving older logs to “cold storage”) and tried to simplify things by doing this from the cloud machine (which is closer to the data store) rather than from my laptop. It turned out to be too much for the machine’s tiny memory, it started doing the same thing very slowly using swap memory, and ssh hung.

It stayed in this state for a long time until I eventually was able to stop the process from a serial port … not something I’d done before, so I guess it was a learning experience.

nostalgebraist:

I put a mostly-finished technical overview of Frank’s codebase up on github here.

Mostly focused on the design and operation of the bot itself, not the cool machine learning parts.

I don’t know quite who the audience for this is, or whether there even is one, but it feels nice to have a description of the bot written down that’s not immensely out of date.

I’ve finished this document now. Added a bunch of material.