I was wondering last night why stuff on my laptop was running slow, and I only just now discovered the cause: on Monday (?) I had randomly decided to start training char-rnn on Dunnett again, this time on the whole Lymond series instead of just the first two, and then I completely forgot about it, and now it’s done 11 full passes over the data and can produce eerie, frequently grammatical simulacra of Lymond dialogue:

‘I thought you brought her to his tale of the angle.’

‘But the planfour woman,’ he said carefully, ‘that Archie has left the treftileness of women. They have no wish to be tongues.’

‘I really didn’t want a message from his grace,’ said Sir George Douglas’s torn voice, ‘that I am told you think you have rather meant that I must blind against the King of France and for your highest banquet.’

‘I haven’t trusted them,’ Philippa said. ‘And so played to him he might prove it. It has noticed that it may have weight. Only the garrison of the Queen is dead. There’s no need to ancience whom and she had so considered what they wore rather proud to believe that she would be dead. Is it not enough of you?’

‘The company of me,’ Lymond said.

‘You would have gentlemen,’ said Jerott. ‘They have no more than in all that, in the end of the day, the young man did what he die.’

(next one’s with a lower “temperature”)

“No. Lady Culter and I shall not be able to forget him,” said Christian. “And if the sensitive information is to tell you to do the rest of the beauty of a subject of a personal six miles or the special standard of a service and because I was to be there to see the other weakness.”

“Or do you think?”

“I want to be deaded and delivered to the Somervilles. I see nothing of the tapers and the craphie, but the bloody daughter was there.”

“Oh, don’t hear that,” said Lymond, “do you want to fight to face the English out of the child?”

“I don’t recover the same time,” said Sybilla. “I didn’t know that. I should find a city, and it is a trick with a little boat in the night. It was the mother of the Constable of France, who can return home to the Tartars.”

“What do you think it would do that?” said Lymond. “As you arrived at the best, you will find it distressing on the same reason.”

“Well, if you want to recover, I shall go to the end of the river of money. It was in the same time, and I am sure you will find out if you were also a hollow man.”

My recurrent network trained on HPMOR keeps using the word “silver,” and I was curious why.  It turns out that “silver” appears 189 times in HPMOR  (counting “silvery,” “silver-looking” etc.).

nostalgebraist:

np-discuss:

nostalgebraist:

reddragdiva:

uploadedyudkowsky:

I must be more than the neural network to have a specific human tribe

look, are you using the recurrent neural network on these things yet, because this is getting good

I am indeed – its worrying claims to the contrary notwithstanding

(Also, my laptop runs out of memory if I try too big a network, so I am now training a bigger one on the HPC cluster at work >:3)

oh god who let you do this

#don’t you have to get approval to use those things?

I had to get approval to get an account, but with an account I can do whatever I want with it, provided I wait in the queue like everyone else.  (In this case I’m just using a single node, more or less a desktop computer – I literally just want it for more RAM.  I could also do it on the desktop at my office, but I already do all my computation on the cluster so it’s simpler this way)

Also I totally forgot that there were some GPU nodes in the cluster, but there are, and training on the GPU with the relevant settings/hardware is 360 times faster than training on the CPU, so that’s good

(Now I am idly wondering whether anyone has ever used the GPU nodes to mine bitcoins)

(via nostalgebraist)

reddragdiva:

uploadedyudkowsky:

I must be more than the neural network to have a specific human tribe

look, are you using the recurrent neural network on these things yet, because this is getting good

I am indeed – its worrying claims to the contrary notwithstanding

(Also, my laptop runs out of memory if I try too big a network, so I am now training a bigger one on the HPC cluster at work >:3)

(via reddragdiva)

“Yes. You can truse your had been recased. De you were following to last year in the grave your face with a frieze new contrad sight,” said Lymond suddenly. “And your lordship of your hamps and a fairth lights on the castle. The lady trampled years to a moment, I was taken their brother from the head in a fairier’s place.

Scott is made, and inselfed to the money had never been waiting. It wouldn’t think saddistardy but me who could fair with the graped face.”

The older recollest brown times, half-shuttered blue eyes. The bast of the end of the sleeping was still with a strangerment of the field. “Why is a family sank successful yourself,” said Thady Boy, and the surrey of light bargained to courtyard. “I don’t leave him. We should delive my sweet first in sudden every call. One in its past your livid gog back with only to send to me.”

“No.”

Recurrent!Lymond is shaping up nicely.  Going to try the new hyperparameters with Yudkowsky now.

Trying some other RNN hyperparameters that seem to work better (specifically, seq_length 100 rather than 50, and dropout of only 0.1 rather than 0.5).  It is showing promising early results on the Dorothy Dunnett training set, producing something that is not exactly English but has a nice cadence and sound to it:

lows, and in putioner of a man you flang to here all the flack with mettentered with emonies crearly but the Mongor’s come to tho many off the cloar of concenter, before them, and Mary lioned whole teether speed on the assickelf and recopered the adeneds, the Dowager was a peever for the other stowdlet and grientless without his face, and all the deepterate, and the gaze facilly as a mone, with a baddeese out of the till on the fleed, by there would banking the whild of her crial who hell that the gold of intending of the seef. “I cane the would bare, to where the some of the passen? Nethered and in scripped and forgeting to face to the thook of a pare of the greet hand for hands on his short? When your deared like a firet, all hell the tell, the conrident you to one off thement of your man’s a cumsiess. You sone and the only one careshanded to have entreped. It is in mising deed the tentings and will be heard, he and the teme to the accome in the Mither to the berencely, smil from the fire had been and dource, and the redorn enderates this langered suckent at his pricher of geoutered it on the looked in a theaping and cheeting and cording of the greet, betwe of the bewark and the brode and accured silence. “I have the got of a proined the heard enterians, and the Bourder of the forest and the Flance from the moors, asheets of ullistened. I long thet norre, you de’s a persable to teeps.’

lithnin:

nostalgebraist:

rubegoldbergsaciddreams:

nostalgebraist:

Anyway, while looking over the twitter feed of that “predictive text” guy I found this, which I had never seen before (although @deusvulture says it’s famous)

Recurrent neural networks: what the fuck

(What’s especially impressive is that the parameters were fit using an individual piece of consumer hardware.  High-end consumer hardware, but still.  This isn’t just throwing giant clusters at the problem)

if you haven’t, check out talcos’s MTG card generation using RNNs (Can give link if you want). It’s where i learned about how hype RNNs are.

I have, and they’re hilarious, especially the “meaningful, novel, but profoundly useless cards”:

* When $THIS enters the battlefield, each creature you control loses trample until end of turn.
* Whenever another creature enters the battlefield, you may tap two untapped Mountains you control.
* 3, : Add 2 to your mana pool.
* Legendary creatures can’t attack unless its controller pays 2 for each Zombie you control.

But it didn’t bowl me over in the way the article linked in the OP did, I imagine because generating M:TG cards is an unusully hard task.  The game has a fairly strict syntax on top of English grammar, and also has a very large card base which already includes most card ideas beneath a certain complexity level (excepting “profoundly useless cards”), so the RNN has three separate problems to solve: “make cards that obey English grammar,” “make cards that obey M:TG syntax,” and “make cards that don’t strike an M:TG player as boring or pointless.”

By contrast, the Shakespeare generation in the OP article is an unusually easy task.  Most of us don’t know much about Early Modern English syntax, and thus are used to Shakespeare sounding grammatically weird in arbitrary ways, so it’s hard to distinguish RNN grammar mistakes from authentic Shakespearean grammar.  Also, Shakespeare is such a prestigious author that imitating him seems especially impressive for any given level of imitation quality.

(Of course I am making these judgments of difficulty in hindsight, with knowledge of how impressive the RNN output actually was, so they should be taken with a grain of salt.)

It’s funny, when thinking about the same underlying facts I came to the exact opposite conclusions.  My expectation is that training on M:TG cards would be an easier task, precisely because of the stricter syntax – there are fewer different types of sentences and phrases in the M:TG corpus relative to Shakespeare, so I would expect the RNN to be able to converge on the former more quickly.

Yeah, now that you say that it seems right to me – I think I was wrong above.

I think there are two effects here pushing in different directions – the M:TG syntax and diction is especially restricted, but by the same token we’re especially sensitive to violations of it.  With Shakespeare, we can get away with syntax violations and, more importantly, spelling violations (words were different back then).  In my experiences playing around with char-rnn I often find that temperatures high enough to produce interesting output also tend to make spelling mistakes, so if the reader is somewhat tolerant to spelling mistakes that’s a great help.

(via lithnin)

nostalgebraist:

So of course having read that RNNs article last night and having also remembered that @uploadedyudkowsky exists I simply had to try generating material for @uploadedyudkowsky using char-rnn rather than a Markov chain

So far recurrent!Yudkowsky is still young:

something behaving, it were muman"?… sometimes any experience from fasting to get observe or this. If I know that the subjects downdy metait incommentation on the fultion. They Trace to be tosefare on the keyich very bit of humanf on your betwowulding metaption. If it else will have place who no matter of unyer overree, problems: But now trying than you even of the theor measuring, many lookness. no, or overrood to spet in the could. That people’s the physics. (And their mettod of my unceasion, why were jult questions, with switting of uncludnent from guess,“ this befine witcoun it a disingeed with green - tonateing has persons.

But it is iterating as we speak.

I’ve trained it quite a bit more, and it is producing more realistic output, but not generally very interesting output, e.g. this is typical:

When I try to advocate the simplest strong person have many people in human beings new proper to older in the occasion of groups of the advice. If it feels to be more likely that the rest of Sanity Algorithm is period had completely not have a lot of points. Can probably doesn’t know what it dees a rationalist that might in the planet AI to predict the results that can have been updated by the words, the physical mind of too smart theory is that the AI is discreted by exceptive fitness; then when the limit of subjective arguments is rational to not make a similar programmer to represent something that was harder to be exhausty to believe it would seem to be long the planet, and of course the problem and the story of value. In a word of computational problems of giving up the systems. In the same is not supposed to discourted the word “logical morality”. To do whether the other perfectly working, “that we are realized with writingless to the sitting for their own experiences. Let’s say it would concedent an intelligence where any subjective seding of shoes in the games - if you realize that your headts were sentient with optimization problem and probabilistic individuals.

This is with a “temperature” of 0.725, where higher temperatures (closer to 1) are more creative but worse at spelling, and lower temperatures (closer to 0) are better at spelling but prone to just repeat common phrases.  If we crank the temperature up to 0.9, we get zany stuff like

You say:” “How, not mary smart werk to another final exit of a good cause?” They are not interviewed by the observation.5.96 single-final values so in The Mind Vrow, view of step. But I’m not matter, to control my long as something about any differentiated self-called Paul One AI-and lomin (experience). But let’s get true, “But if it is not a matter of physics, their dees of low-ord-deception is anthropolyway!”, wasn’t the same for at least one people? If I had a way that “rationalists” will write the Dussimion that they don’t know. One of this is whether the better stiny sensis, and probability matters is rather than atomic increase in each particular species.

And if we cool things down to 0.45, we get this repetitive drone in a more formal register:

I suggest that I think the same way that we can measure the problem that can be a definite specific reason to press the future of real lifespan in the trajectory of humans and problems, and even if you were something about the same as the same thing. If you could also be surprising and destroying the external problem of particular reasoning to the face of what they were all for an extreme individual observation. If you start with a complete set of particles, in a struggle and the problem of interpretation, even though I want to say that we should do with the arguments to the same as the right teacher can still answer.

Skeptic: What was it for me that the students of the same thing will seem like any single perfect emptiness of their beliefs that can be rationally thought to press it any possible failure in the final result where it is true.

‘And may least this river at least to you with ollie to wastard the very galley …’ The Queen Dowager stretched no drew tender of his interested and pat him soft back. ‘In felt on your bray France?’ He made England deright on the provering their feets and stepped,’ said Lymond propled to kelp the strust. Somerville streight eyes, her face stusted vaous of the clanders, and he spirting his voice, ‘It was to you in. But if you thing Robin Stewart is sole and praminery, are you surely for you less in the pilicical and spairs. Was the ages. Will music free.’

‘I ampressater. And my own ride shoot with a match in the my sassle and the space.’ Back into the leasting the gate, then with unsound her own electian, sight in God’s pleated to rall on them and then nastian Crawford of Don, not a play here in the gardle of the scontle of the secretary, “You’re that the mecains of a grave difficult thull fands we have a leng unlike you, and with has a suddenly like a paused of the wood quarney in Stewart.”

“He doubt here to Pound—”

Mo, with the one responce of his coon. If the onle hand on the rained at the child voice, and read it for the window hereed and rose her foot Edglish spittors? As when he propped his own life, and the are In the peace suppress-trick he solding. “What will you tell that all nearly ocheress to lack the creak to here as vince and one you any of a master of the Masser food and look to Crawford with a contrimicul at Gromasious the great this the left of phears and have it.’

I’m training the neural network on Dorothy Dunnett now