nostalgebraist:

Some of the mystifying things Esther has yelled in her sleep since moving here (exactly one month ago):

“I want the moon to kill me!”

“I don’t want to die, I want to eat potatoes!”

“The moon will die!  The sun will die!”

(These were from three different nights)

From last night (both in very distraught voice):

“Why is the moon made of the moon?!”

“Why does God want the moon?!”

glassyvegetation:

“Hmm, you seem to have done some difficult task. This indicates that the task wasn’t actually difficult or noteworthy. Proof: suppose the task was an actual achievement. That would imply you’re somewhat competent, which is trivially false, QED” - Insane Bad Brains Troll Logic

(via moral-autism)

clone my grubhub repo

From Barack Obama’s ability to overcome obstacles in his election races, to the design of the iPhone, the stoic philosophy has helped its users become world-beaters.

I don’t have the energy to properly effortpost about it right now, but a little while ago I got around to reading the paper “Opening the black box of Deep Neural Networks via Information” and it blew my mind a bit.

The overall picture it gave me is that deep neural nets, when trained with SGD, learn by first memorizing a lookup table for the training data, and then forgetting everything about the table keys that isn’t necessary to uniquely identify the table values.

Early on, the last layer encodes the requested value but also the whole key, like a dictionary {k: (k, v)}.  This then gets lossier and lossier for the first part of the tuple without getting lossier for the second, so the end result is like {k: (irrelevant noise, v)}.  At the very end of the network, the fully-connected-softmax or whatever scoops up the “v” part and ignores the “irrelevant noise” part.  Since the last layer now no longer has the information necessary to reconstruct the key, and since this loss of information happened over a long (”deep”) series of lossy steps rather than in a single lossy step of table lookup, we can hope that it’s now learned something meaningful and generalizable about the relationship of these keys to their values.

So, during training, they overfit first, and then regularize themselves.  In itself, not too shocking.

But here’s the kicker: the regularization part happens specifically because of SGD.  Once the gradients are small enough, they’re basically applying a random diffusion to the weights, and so we end up with a neural net that is “as random/generic as possible,” conditional on getting the answers right on the training data.

The authors argue that network depth helps here, because it speeds up the diffusion process.  Thus, the great success people have had with deep networks trained by SGD is attributable to “depth + SGD” being a synergetic combination, not by deep networks being an inherently great space of models and SGD being merely “good enough” as a way to descend loss functions in this space.

The main limitation of this paper is its reliance on a single invented data set for its results.  I get the sense they did this to make it possible to estimate mutual information well enough to get anywhere.  They do have a github repo where you can do the same analysis in other cases, which is neat although I haven’t tried it.

I saw a copy of some university magazine whose cover featured the phrase “Envisioning Inclusive Excellence,” and my brain immediately spoonerized this to “Envisioning Exclusive Incellence”

If you can stand it, the explosive wedding ceremony between tart and sweet is unique in the fruit world.

Perhaps one day, all the sage grouse will hear out on the lek is the quiet whine of a fembot turning its head.