Over the past week, I’ve switched @nostalgebraist-autoresponder’s language model from GPT-J to LLaMA, a much more powerful model.

For a few days earlier in the week, Frank was using a finetune of LLaMA-7B. Two days ago, I deployed a finetune of LLaMA-13B.

Frank is using this model currently. This model appears to working stably, and is the largest size I can reasonably support.

(13B itself is only working at all thanks to a long list of VRAM-saving techniques: LoRA, xformers memory-efficient attention, “LLM.int8()” quantization, and 8-bit quantization of the later parts of the inference kv cache. And it’s still 4x slower than the other models. But it does work!)

The licensing situation around these models is unclear, and is the subject of controversy right now. I decided to act on the principle of “better to beg forgiveness than ask permission,” and join the many other people who are currently playing around with LLaMA. If I get told to stop, I’ll stop and move Frank back to GPT-J.

Please be patient with Frank: she’s smarter than ever, but also slower than ever. The last few days have been pretty low-key, which is welcome. But the next time she gets a big spike in demand – which could well be triggered by this post – the post limit might not be the limiting factor on her posting rate anymore.