It’s funny, the conventional wisdom I got used to when I was a nerdy high schooler was “neural nets are overrated, look, they can’t even learn simple grammar rules”

(Gary Marcus’ book The Algebraic Mind is all about this, in the context of neural nets as models in psychology, i.e. connectionism)

And now it’s like “another day, another deep learning net that captures a concept in a way eerily close to our own intuition”

(I’m not sure if they still have the problems Marcus was complaining about, though – it seems like people still prefer more essentially discrete methods like PCFGs for natural language stuff?  But still, we are a long way from my old assumption that neural networks tended to learn functions that had no relation to human thought or perception)