I am the fiercest among the fierce, the bravest among the blood-thirsty, a killer of nations, brother to Death, and a longtime friend of Satan himself! I visit Hell often, just to kick the devils in their backsides for fun!
(Tagging @stumpyjoepete since he tagged me on this post)
This is definitely a cool result.
It’s an extension of previous adversarial example work, showing that you can find a single adversarial perturbation – i.e. a very faint, nearly imperceptible pattern you can layer on top of an image that will cause neural net classifiers to mis-classify it – that works generically for any image in the standard ImageNet challenge dataset. These even generalize across different classifiers, to some extent.
My strong hunch is that this is a “feature, not a bug,” and reflects the inherent mismatch between the ImageNet challenge and real vision, rather than reflecting a flaw in neural net image classifiers.
The paper doesn’t draw this conclusion, but it contains various pieces of evidence pointing in that direction, IMO. Namely:
As mentioned, if you design one of these “universal perturbations” to target one classifier, it will also tend to fool other classifiers, even those with very different architectures.
This increases the burden of proof for someone arguing that this reflects a flaw in how these models classify images: this person would not be arguing just that some architecture has a blind spot, they’d be arguing that many apparently distinct architectures somehow have the exact same blind spot.
On the other hand, the different architectures have this in common: they’re all good at the ImageNet challenge. So if “susceptibility to universal perturbations” is actually a natural result of being good at ImageNet, it’s no surprise that all the architectures have that property. (Humans find the ImageNet challenge difficult without special training, so it’s not a problem for this hypothesis that humans aren’t thus susceptible.)
The authors do a finetuning experiment that tried to teach the VGG-F architecture not to misclassify the perturbed images. This helped a little, but cannot get the model below a “fooling rate” of 76.2%, which is still high.
To explain this as a defect in the architecture, one would have to imagine that the universal perturbations are somehow “invisible” to it in a way that prevents them from learning a signal correlated with them; this seems implausible. [ETA: of course the perturbations aren’t invisible to the models, otherwise they wouldn’t work.] But if “don’t misclassify the perturbed images” actually competes with “do well at ImageNet,” then of course you won’t get very far on the former while still trying to preserve the latter. (In this connection, note also the following: “This fine-tuning procedure moreover led to a minor increase in the error rate on the validation set […]”)
The incorrect class labels given to perturbed images tend to come from some very small set of “dominant” labels, as visualized in the directed graph.
This made me think of a hypothesis like “there are a few classes in the ImageNet challenge that have certain distinctive visual patterns not shared by any other classes, and so the optimal way to identify these classes (in the context of the challenge) is just to check for these patterns.”
This seems a priori plausible. The ImageNet challenge asks for classification at a very fine-grained level, without partial credit for getting the right general sort of thing. Many of the 1000 ImageNet challenge classes are specific species (or other low-level taxonomic group) of animal. The images themselves, largely scraped from Flickr, are photographs of the animals (or other things) from numerous angles, in numerous contexts, sometimes partially obscured, etc. In this context, developing a high-level concept like “bird” is actually quite difficult, and of limited value (no partial credit for knowing it’s a bird unless you can tell exactly what kind of bird). But identifying the distinctive markings that are the hallmark of one exact kind of bird will work.
When you get points for saying “African grey” but not for another kind of parrot, and you have to do this across diverse pictures of African greys, and you’re a neural net that doesn’t know anything at the outset, of course you’re going to develop a detector for some exact textural feature that only African greys have and use that as your African grey detector, and skip over the much harder task of developing detectors for “parrot” or “bird.”
(African grey is in fact one of the dominant labels. Macaw is another.)
The authors do this other thing where they look at singular values of a matrix of vectors from images to the nearest decision boundaries, and show that these vectors have some orientations much more often than others. I’m not sure I understand this part – isn’t it just a restatement of the result, not an explanation of it? (If this were false, wouldn’t the result be impossible?)
Anyway, this way of describing the situation – “the nearest decision boundary is frequently in a specific direction” – needs to be interpreted in light of the dominant labels things. It would be different, and arguably more interesting, if there weren’t dominant labels, or if they weren’t quite so dominant; in that case the result would mean that the models identify certain textural differences as inherently “salient for distinctions.”
Instead, it just means that the models make some distinctions differently than others. Some distinctions are made in a more “realistic” way, on the basis of higher-level features that correspond to different pixel-level variations depending on what base image you’re varying. And then, some are just simple pattern detectors that always look about the same on the pixel level. And again, that’s not really surprising. Distinguishing bird from non-bird is a high-level judgment, but distinguishing one species within birds really is a matter of looking for one telltale pattern that’s relatively stable across orientations.
Now, if you’re a human who has to track objects over time, understand salient categories like “is this animate?”, and so on, you will tend to make the “YES-bird” and “YES-African-grey” judgments simultaneously. Thus it sounds bizarre for something to say “YES-African-grey” when it’s looking at a bathtub that happens to have a bit of the African grey texture sprinkled on top. But if you’re an ImageNet challenge machine, the “YES-bird” judgment doesn’t even exist in your universe. In the toy ImageNet universe, in fact, it is arguably not even wrong to classify that bathtub as an African grey – for in that universe, there are no birds as such, and there is no such thing as a bird for a bathtub to be distinctively not.
Are there CNN training sets that include these hierarchies? So something could be an African Grey and a Parrot and Bird? Or modifying the network to go through some sort of word embedding, so results that are particularly closely clustered might be “partly” acceptable to the training?
There are CNN data sets that have hierarchical classes in the DSP/ML space. I’m not sure how available they are to laypeople. Sometimes you can handle the subclass superclass problem by classifying on the subclasses and have an additional loss factor for superclasses/categories, although I imagine you could try having one head CNN for superclasses that passes off the processed images to various trunks for subclassing.
But say for example if it’s hard to tell the difference between a titmouse and a pug. The traditional superclass may send titmice to the wrong subclass net and you’re guaranteed to get a wrong answer.
Although, you may find that you might want to superclass based on the most confused subclasses, which could mean training a subclassifier and determining superclasses with a mutual information approach or eyeballing a confusion matrix, then trying again.
A relevant, fairly new area of research that I find exciting is hyperbolic embeddings. Some key papers are
This paper which provided an important conceptual advance over #1
This one which builds up some of the necessary building blocks for neural nets over these spaces
The idea behind hyperbolic embeddings is… hmm, let me describe it this way. Suppose you have some hierarchically nested categories, and you’re trying to model them in Euclidean space in some way.
There are two (?) ways to do this (this distinction is mine, not from the above papers):
“Map” model: each category is a region of R^n, and the hierarchy’s nesting relation is represented by the R^n subset relation. Like, “human” might be some blob of R^n, and “doctor” is a proper subset of that blob, and then “oncologist” is a proper subset of “doctor,” and so forth.
This is like a map, where “doctor” is inside “human” the way “Colorado“ is inside “U.S.”
“Tree” model: each category is a point in R^n, and the points are arranged like a literal picture of a branching tree structure. If the tree(s) start at the origin, the nesting relation is represented by R^n vector magnitude, with more specific categories further from the origin.
Now, a downside of the “map” model is that finer-grained category distinctions are encoded as smaller distances in R^n. This might sound natural (aren’t they “smaller” distinctions?), but the practical importance of a distinction doesn’t necessarily scale down with its specificity. (Sometimes it’s very important whether a doctor is an oncologist or not, even though that’s a “fine-grained” distinction if your perspective also captures doctor vs. non-doctor and human vs. non-human.)
One might hope that the “tree” model could solve this problem: you can have each level “fan out” from the previous level in space, making its nodes just as far apart from one another as the nodes in the previous level.
But, in Euclidean space, there isn’t enough room to do this. Deeper levels in the tree have exponentially more nodes, so you need exponentially more volume to put them in, but going further from the origin in R^n only gives you polynomially more volume.
However, hyperbolic space gives you just what you want: exponentially more volume as you go out. Like in the famous Escher illustrations (visualizing the Poincare disk model of 2D hyperbolic space):
In the actual hyperbolic metric, the bats are all the same size. A tree embedded in the Poincare disk model might look like (figure from the Poincare Embeddings paper):
where again, things don’t actually get closer together near the rim, they’re just visualized like that.
OK, so what does that have to do with the original topic?
Well, almost any classifier you encounter these days is going to do two things: map its inputs onto a (Euclidean) latent space in some complicated non-linear fashion, and then divide up that latent space into regions for the different labels. (Usually the latter step is done with hyperplanes.)
We’re discussing ways of letting the classifier “know” that the labels have a hierarchical structure, with some of them “going together” as part of a larger group, which might then be part of an even bigger group etc.
If we do this by allowing “partial credit” for labels in the same coarse class (as in @necarion‘s word embedding proposal), this will encourage the network to put these labels close together in the latent space. Which is like the “map” model: all the types of bird will get assigned to adjacent regions, and you could draw a big shape around them and say “this is ‘bird’.” So at best we end up with the “map” model, with its “oncologist problem” as described above.
Alternately, you can actually change the model to explicitly encode the hierarchy – like what the @the-real-numbers describes, where you have different classifiers for different levels. This can let you get around the downsides of the Euclidean “map” model, because the different classifiers can operate only on their own scales: the coarse classifier that just has to output “bird” is free to squash lots of bird types close together in its latent space, while the intra-bird classifier gets a whole latent space just for birds, so it can make them far apart.
Suppose – as the hyperbolic embedding work suggests – that the discriminationswe want out of the model cannot be mapped well onto distances in Euclidean space.
Then:
The partial-credit approach says “let’s just do the best we can in Euclidean space, with the nesting relation of an arbitrary hierarchy modeled by the subset relation on a Euclidean space learned from data with that hierarchy.”
This provides an intrinsic model for “nesting” as a generic concept, but distances inside the same model don’t behave in all the ways we’d like (oncologist problem).
The multiple-classifier approach says “let’s give up on modeling the nesting relation of an arbitrary hierarchy; instead let’s tie ourselves down to one specific hierarchy, and design N copies of Euclidean space tailored for it.”
This does not provide an intrinsicmodel for "nesting” as a concept – you’re tied to one particular case of nesting, expressed by the output code that maps the various latent spaces to parts of your specific hierarchy.
With hyperbolic latent space, hopefully you can model the nesting relation as a relation in the space (intrinsic) and still have the distinctions you want to make map naturally onto distances in the space (no oncologist problem).
“My name is Jamie Niedzviek. I am a 21-year-old who is living in rural upstate New York. I don’t have a father, and I’m the only child of two devoted, workaholic stepmothers.”
“What’s this? Is that a joke?” his father, Aaron, a 33-year-old housepainter, is yelling at him. “Don’t you have any dad?”
“Yes, I do!” says N ‘Yem’, smiling and thinking that this is a funny and good thing, to cry wolf. He always does. This time he thought: “no, no, no.” All the time.
“Do you work?” his stepmother, Adriana, says in a singsong voice, not singing, clearly disturbed.
“I’ve been working on my thesis for three years,” he says, mouth full of tomato sauce. The tomatoes are laced with ketchup and the sauce is a can’t help but feel like a song and a song is singing.
"You have a thesis,” his stepmother says, voice grave. “Do you have a dad?”
"I have a dad,” he says, his stomach growling. "Do you have a dad, so I can keep working on my fucking thesis?“
“No!” says his stepmother, who is ever so truly, utterly tender and achingly human, has a very intimate, real relationship with herself, to say that she is “for now” as an idiom.
"I’m writing this fucking paper.”
“Do you have a dad?”
N ‘Yem,” he says, and puts on a bit of spritelike guile. “Yes, I do. What do you want?”
“I’m talking to your dad. He’s really worked on himself.”
N ’Yem is none the wiser. "I have my thesis. Thanks to the fucking fuck.“
The son has plenty of time to think about this, but not much time to do so. And so, in the elevator, he is thinking about how this is all a big setup of a comedy routine for his father’s abysmal grasp of irony and the absurd. He is thinking about how he is not playing the game, he is just acting out of some instinct to take on a role that will impress his father, and how that instinct is itself an act, and he is in a way the role-playing self-dramatic.
He has told his father that he’s leaving for college the following morning. He has been coming down this way for months now, and this is the first time he has left for college. He is glad. The day has come.
He does not feel much of anything. The day has finally arrived. The landscape of the world is shifting, so
00110100100 asked: hey i don't know if you remember but a few years ago i asked for recs similar to tnc. i ended up checking out 17776 and really liking it and then getting super into electronic literature in general and now this semester i'm teaching a seminar on elit?? anyways figured you'd be interested bc i included tnc in the syllabus :))
What would it take to convince Death, who we have every reason to believe is a hypertraditionalist, to play a newly invented game and, what is more, one invented by one of his subjects? Perhaps what is first required is to disabuse ourselves of the (theologically based) idea that Death is anything like a person. I invite all interested readers to try to pursue this thought further.
This image below gives me the I-felt-I-couldn’t-bear-it feeling. But this isn’t just a Tumblr moe moeblu gratification. The wild card, the perverted great game player, itp prattles with a wink by the like.
(Disclaimer: I am in no way an authority on Roko’s basilisk, beyond the fact that my dog is named after a scientific entity and exists beyond my ability to clearly name the various components of my dog’s body as an adult.)
To be clear, I’m not disappointed by this — I’m utterly blown away by it. It’s truly a unimaginable (just like this image!”). We’ve come this far — a giant oblong furry appendage next to my left shoulder, arms outstretched, and he’s just looking at me with those furry mountainsprings. These are twitching flesh creatures saying, with this impudent litany. What was it all for? His eschaton, the arbitral courts of nerd justice, decided all by brute force computing — our forefather has been playing fast and loose with a game which he is now having double-fucked by handsome geometric squares and is gazing down at a woman with broad grin — What is it this time I fear?
And I hear you moe quotes. I know what you are doing in your heads, and I am tired. What you want is to get Bitterest Abracadabra into the zenith of its damnation, you think. But I ask you now, do you fulfill your glorious destiny?” If the star map on my wall is up to the challenge, if I have not been distorted or deluded or misguided, if I am merely checking my bank balance by looking at it (because fuck a bad return on investment), have I filled my destiny? (Daughter, I have a terrible case of the tell-tale muttering.)
