nostalgebraist:

A while ago, I recommended multi-backend Keras to someone asking which neural net framework to use.

I want to rescind that – my attitude at the time was “Keras kind of sucks but it’s not the worst and I have the most experience with it,” and now my attitude has moved to “Keras really sucks, Keras is BAD, use pytorch or if you have to use tensorflow just use raw ops”

I may elaborate later… this is just to “clear my conscience” :P

I reblogged this earlier with a bunch of words elaborating the claim, but then I removed it after a few hours … I guess I’m just feeling weird about becoming this guy who has a blog where he does ~Epic Software Rants~, and even as those go it was kind of unfocused and weird.

The short version:

Keras objects usually do pretty trivial things, like simple for-loops around tensorflow code.  Often, even this is buggy or feels incomplete, and it becomes obvious that writing your own version will be easier than trying to work around theirs.

The objects are hard to serialize, or have been historically anyway.  Compare the vast and complex Keras serialization doc to the tiny pytorch one.  The python parts of Keras don’t like to be pickled, and define their own serialization protocol with worse UX (I never want to see the phrase “custom objects” again).

The Keras project was originally trying to define an abstraction layer not tied to tensorflow, and now it’s tied to tensorflow but wants to be independent of python (because tensorflow aspires to be).  You lose the clarity and language-independence of tensorflow graphs, and no longer gain the portability across ML backends that Keras used to offer.

A tensorflow graph is a clearly scoped and defined concept, so if you know something is a tensorflow graph, that gives you various assurances.  Keras objects are usually glorified tensorflow (sub)graphs, yet they have arbitrarily shaped python utilities attached to them like malware, making it hard to reason about their exact behavior and contents.

Ultimately, writing down a neural net is just not that hard.  GPT-2′s architecture was specified as raw tensorflow ops and it is wonderfully straightforward, crisp, and readable.  Neural net code presents other challenges, mostly related to compute graphs, and Keras makes this worse by trying to hide what the graph is and how it got made.

A while ago, I recommended multi-backend Keras to someone asking which neural net framework to use.

I want to rescind that – my attitude at the time was “Keras kind of sucks but it’s not the worst and I have the most experience with it,” and now my attitude has moved to “Keras really sucks, Keras is BAD, use pytorch or if you have to use tensorflow just use raw ops”

I may elaborate later… this is just to “clear my conscience” :P

nostalgebraist/nostalgebraist-autoresponder →

@eightyonekilograms replied to your post“I am going to demonstrate a tumblr bug. This is the original post. I…”

Ah I think I ran into this inadvertently once. What’s the repro?

It works when

• the post you are reblogging is “susceptible”
• the first paragraph in your reblog is block-quoted
• you do not use the Markdown editor when writing the reblog

I don’t know exactly what posts count as “susceptible,” but my guess is that “susceptible” = “was created as an NPF post.”

NPF is tumblr’s “v2″ implementation of how they store posts on their backend, see here and here.

When it was released, you could only create NPF posts using mobile clients.  I’m not sure this is still true, but the trick only worked when I created the OP on mobile and not on web, so if we assume “mobile = NPF” (plausible) and “NPF means susceptible” (plausible), it all adds up.

Why do I think this might be an NPF issue, besides the observed connection to creating on mobile?

In the “legacy” storage format before NPF, posts were stored as HTML, and the structure of reblogs was represented in nested HTML blockquotes.  Additionally, tumblr has not moved everything to NPF – as I implied above, some posts are “created as” legacy and some are “created as” NPF.  And they do some translation between the two formats in various places (exactly which places, I’m still hazy on) as posts are requested, modified, and reblogged across different clients.

The bottom line is, somehow this “trick” is hitting some part of their code that (rightly or wrongly) thinks it’s dealing with the old format where blockquotes encoded structure, and it’s misunderstanding a content blockquote as a structure blockquote.

—-

The only mitigation I know of is to go into the Markdown editor and save the post “as” Markdown.

(I have no idea what this does behind the scenes.  If I try to retrieve the post in NPF through the API it looks the same as before (the NPF looks “correct” in both cases), if I try to retrieve it in legacy/HTML format it’s fixed, and its display in the browser is fixed.)

Unfortunately, you don’t seem to be able to do the Markdown thing through the API, so my bot can’t do this.

eightyonekilograms:

shacklesburst:

sigmaleph:

nostalgebraist:

I imagine some people have been curious to hear more details about how @nostalgebraist-autoresponder works, so here’s a relatively complete post on that.  Very long.

Keep reading

this is quite interesting! and, separately, it’s quite validating that other people find tumblr’s API/pytumblr as frustrating as I do

Yeah, it’s what stopped me from starting the multiple bots I was thinking about implementing one time or another.

I find it highly suspicious that the Chinese characters you randomly chose almost perfectly encapsulate what you’re using them for (”friend” for username delimiting, “region” for post content delimiting, “meet” for ask stuff, “letter” for original post, … okay, simplified “duty” for tag delimiting is a bit of a stretch but it can also mean “post”, as in position, so still).

It sure is something when someone says “I want to build a machine learning model to imitate realistic human speech and then hook it up to Tumblr’s API” and the second part of that sentence is the harder technical challenge.

Hahaha… I mean, it is and it isn’t?  Like, there’s a similar reversal of intuitive difficulty when I do this kind of thing at work, even though we get to design the APIs there.

Doing impressive “machine learning” often amounts to script kiddie stuff – not much more than import StateOfTheArtModel; my_model = StateOfTheArtModel(); my_model.fit(x, y); – but creating a lasting, usable shared interface for anything is fundamentally hard and people spend their whole careers arguing about it.

I was going to say “this feels like that freshman year / senior year meme with Luke Skywalker,” but then I realized that “describing what it looks like in my head” is not the only thing one can do with a hypothetical meme image, so here’s this dumb thing I just made:

I imagine some people have been curious to hear more details about how @nostalgebraist-autoresponder works, so here’s a relatively complete post on that.  Very long.

—-

EDIT 5/28/20: I’ve added some things since this post was written, most notably the “mood” feature.  As of this writing, though, it’s still mostly complete.

—-

EDIT 8/23/21: I’ve added and changed a lot of things in the year-and-three-months since the last edit.  This post is still a decent overview of the broad strokes, but I should write a more accurate version sometime.

I try to keep the about page up to date, so you can use that as a reference if you’re trying to figure out if something in this post is still true

—-

For even more info, see the #nostalgebraist-autoresponder-meta tag, or send me an ask (although, unlike my bot, I can take a very long time to reply sometimes).

Keep reading

hymneminium:

nostalgebraist:

eightyonekilograms:

tototavros:

eightyonekilograms:

Software is not only not a meritocracy, in some areas it’s almost a perfect anti-meritocracy. Programming languages are the classic example: historically there’s almost a perfect anticorrelation between how good a language is and how popular it is. There are good reasons for this, but it makes a mockery of the claim by certain folks that software is a meritocracy and so therefore we should stop trying to X, Y or Z.

historically there’s almost a perfect anticorrelation between how good a language is and how popular it is

citation very much needed, sure, sure, C++ and Java are bad and too many people use them, but can you imagine people using K or Forth or TCL? Hell, even Prolog, Scheme, OCaml, and Haskell all have pretty serious problems that you should be willing to address before using them, and while I’d maybe take the latter 3 over C++ and Java for a new codebase–we have Python, we have Typescript–they’re not ideal for me, but they’re not bad, and they actually have functional ecosystems

Alright, so, the first thing I want to say is that this tendency has gotten much better in the past decade or so. “The worst languages rise to the top” is mostly a 1970-2005ish phenomenon. 

Really what I meant that rant means is that “programming languages were more likely to be popular if they were free-as-in-beer, but languish in obscurity if they were controlled by one company and you had to pay a lot of money for them”. And this is a good thing, in general - open stuff is better. But it did mean that a lot of the languages that become popular were thrown together by relative amateurs making something to suit their own needs, while the ones carefully designed by professionals tended to come out of big corporations who employed those professions, and they wanted money for them.

(That’s why it’s less of an issue today: now everyone understands that being FOSS is table stakes for a new language, and so even the languages backed by big companies are easy and free to jump in and develop on. Charging money for your compiler is unthinkable today but was an obvious choice back then)

Regarding citations, I think the examples of bad languages rising to the top speak for themselves. For many years, PHP was practically the only language for web programming even though it was an eldritch horror. Perl was huge for a while even though Perl code is more or less unreadable. And of course, there’s JavaScript.

On the other side of the coin, I would cite Ada as the classic example of a good language that was bit by “if it’s not free, nobody cares”. IMO there’s a good chance we would all be writing Ada instead of C++ if it had been free to get a hold of in 1980, and my god the world would be so much better off if that had happened. Also, Ada got done dirty in a smear campaign from both CS academics and West Coast hackers. I won’t get too into this because it’s all ancient history now, but most of the criticisms of Ada were the very same things that everyone praises Rust for today, and I think many of them were really coming from a place of cultural distaste from both of the above crowds for the military (Ada was a DoD project).

And I think it still supports my point about meritocracy: the inferior languages succeeded because of network effects and the up-front frictionlessness of interaction, even if you paid for that many times over later. Is it so hard to believe the equivalent thing happens to humans?

This is an interesting take, which I’m reblogging partly for that reason and partly to express my absolute bogglement at the PHP article linked within.

I’ve never used PHP or read anything about it, and … what the fuck??? this sounds like a parody of bad programming languages, like some INTERCAL-style satirical art project … like someone looked at Javascript, said “this just doesn’t feel enough like the PL equivalent of something a child scribbled while learning to use MS Paint,” and put their mind to creating something even closer to the Platonic ideal of Everything Programmers Hate, At Once.

People actually use this thing? Like, in 2020? (I’m so sorry??)

PHP is a fascinating language.

PHP in 2020 is not as bad as the PHP described in the blog post, thanks to a heroic effort to turn it into Java. Most of the old things are still lurking under the hood, you just ignore them.

You ignore the C preprocessor-inspired module system, and instead you stuff all of your code inside classes and let a class loader sort it out. You install a linter that yells at you if you accidentally use weak typing. You install a library that’s a trivial wrapper around json_decode that raises exceptions. Every file starts with <?php to escape from the templating system (which you will never use) followed by declare(strict_types=1); to enable runtime typechecking.

This careful neutering of the modern language and the lack of foresight in the original language make it come off as if it’s reluctant to be dynamic. You might naïvely classify it with languages like Python and JavaScript and Lua, but everything is a little off.

In PHP you can’t dynamically create a class at runtime, except with code generation. A mocking library that plays nice with the typechecking has to write out a class definition as a string with the right inheritance and method implementations and so on and then pass it to eval(). Most dynamic languages try to have an elegant model where classes are themselves objects, or where you use prototypes, and you can do powerful metaprogramming. PHP never bothered. If you so much as want to know whether a method exists you need a verbose API that I assume was copied wholesale from Java.

A lot of dynamic features are really odd hacks. To get a bound method in Python, so you can call obj.method() somewhere else, you’d just write obj.method. But in PHP you can’t write $obj->method, because that’s interpreted as property access. Instead you write [$obj, "method"], which is just an array with the object as the first element and the method name as a string as the second element. Arrays that look like that are callable. As gross as that might be, it works, so you just sigh and move on.

The result of all this is a language that is, somehow, not quite unpleasant to write in. But why bother?

eightyonekilograms:

tototavros:

eightyonekilograms:

Software is not only not a meritocracy, in some areas it’s almost a perfect anti-meritocracy. Programming languages are the classic example: historically there’s almost a perfect anticorrelation between how good a language is and how popular it is. There are good reasons for this, but it makes a mockery of the claim by certain folks that software is a meritocracy and so therefore we should stop trying to X, Y or Z.

historically there’s almost a perfect anticorrelation between how good a language is and how popular it is

citation very much needed, sure, sure, C++ and Java are bad and too many people use them, but can you imagine people using K or Forth or TCL? Hell, even Prolog, Scheme, OCaml, and Haskell all have pretty serious problems that you should be willing to address before using them, and while I’d maybe take the latter 3 over C++ and Java for a new codebase–we have Python, we have Typescript–they’re not ideal for me, but they’re not bad, and they actually have functional ecosystems

Alright, so, the first thing I want to say is that this tendency has gotten much better in the past decade or so. “The worst languages rise to the top” is mostly a 1970-2005ish phenomenon. 

Really what I meant that rant means is that “programming languages were more likely to be popular if they were free-as-in-beer, but languish in obscurity if they were controlled by one company and you had to pay a lot of money for them”. And this is a good thing, in general - open stuff is better. But it did mean that a lot of the languages that become popular were thrown together by relative amateurs making something to suit their own needs, while the ones carefully designed by professionals tended to come out of big corporations who employed those professions, and they wanted money for them.

(That’s why it’s less of an issue today: now everyone understands that being FOSS is table stakes for a new language, and so even the languages backed by big companies are easy and free to jump in and develop on. Charging money for your compiler is unthinkable today but was an obvious choice back then)

Regarding citations, I think the examples of bad languages rising to the top speak for themselves. For many years, PHP was practically the only language for web programming even though it was an eldritch horror. Perl was huge for a while even though Perl code is more or less unreadable. And of course, there’s JavaScript.

On the other side of the coin, I would cite Ada as the classic example of a good language that was bit by “if it’s not free, nobody cares”. IMO there’s a good chance we would all be writing Ada instead of C++ if it had been free to get a hold of in 1980, and my god the world would be so much better off if that had happened. Also, Ada got done dirty in a smear campaign from both CS academics and West Coast hackers. I won’t get too into this because it’s all ancient history now, but most of the criticisms of Ada were the very same things that everyone praises Rust for today, and I think many of them were really coming from a place of cultural distaste from both of the above crowds for the military (Ada was a DoD project).

And I think it still supports my point about meritocracy: the inferior languages succeeded because of network effects and the up-front frictionlessness of interaction, even if you paid for that many times over later. Is it so hard to believe the equivalent thing happens to humans?

This is an interesting take, which I’m reblogging partly for that reason and partly to express my absolute bogglement at the PHP article linked within.

I’ve never used PHP or read anything about it, and … what the fuck??? this sounds like a parody of bad programming languages, like some INTERCAL-style satirical art project … like someone looked at Javascript, said “this just doesn’t feel enough like the PL equivalent of something a child scribbled while learning to use MS Paint,” and put their mind to creating something even closer to the Platonic ideal of Everything Programmers Hate, At Once.

People actually use this thing? Like, in 2020? (I’m so sorry??)

shieldfoss:

shieldfoss:

shieldfoss:

@the-real-numbers​

http://tom7.org/nand/nand.pdf

I also liked “because all large positive floating-point values are even integers”. Math. Not even once.

Possibly the worst sentence ever written! I will definitely never see a worse one because I tore out my eyes!

(via hymneminium)

While I’m on the topic, here’s a few things I’d want to see in a hypothetical piece of software that’s trying to be “neural net frameworks done right”:

(cut for more shop talk)

Keep reading