Yahoo Pipes may be dead, but Node Red seems to be the device swarm orchestrating, data-flow programming tool I was hoping it might evolve into.

It’s free software, it runs on everything including RaspberryPi and talks pub-sub protocols like MQTT.


Source: Node-RED

So Yahoo Pipes is closing down.

As of August 30th 2015, users will no longer be able to create new Pipes. The Pipes team will keep the infrastructure running until end of September 30th 2015 in a read-only mode. You can download your Pipe definition by specifying your Pipe-ID (_id) and the output format (_out=json) to the following end-point: For e.g:

Please post any questions you have related to this announcement on the Yahoo Developer Forum for Pipes. The Pipes team thanks you for support and cooperation.

Source: Yahoo Pipes Blog – Pipes End-of-life Announcement

Clearly, they didn’t have my intuitions about it. Seems there are some alternatives in “algebra of feeds” space, though.

Arguably, this could be the beginning of an interesting wave.

Stephen Wolfram's Introduction to the Wolfram Language

I’m impressed. Beyond being just a grab-bag of libraries (which it could have been) it seems there’s real thinking about composability here. The laziness of evaluation, the “symbolic-ness” makes it a powerful functional programming language. In fact, it’s almost a “data-flow” language. Of the kind, like Yahoo Pipes, which can become an orchestrator of web-level components.

That already gives some substance to the claim that this is a “new” kind of programming. But what REALLY differentiates it, I think, is the philosophical commitment to what O’Reilly used to call “Data Inside“.

This might well herald a new trend : “programming language as a service”. A language whose essence and value depends on not just the library of functionality, but the continuously updated data living in that cloud. This is genuinely new. (Maybe it contrasts with something I once said about SemanticCommitment, that semantic constraints weren’t the treasure that the SemWeb people thought they might be. Perhaps the Wolfram Language is a bet against me, and on the idea that we now have comprehensive enough databases of “knowledge” about the world, that packaging huge ontologies of semantics within languages is a valued feature.)

It’s also a huge risk to the programmer, of course. You’ll only EVER run Wolfram Language against Wolfram’s cloud. Every program you write will be dependent on Wolfram in perpetuity. And we know where that story leads … the cloud provider’s business model inevitably evolves to being observation and analysis of what the customer is doing. (And it should go without saying that every program you run is explicitly open to inspection by the NSA etc.) But some people will be willing to make that trade-off.

Third in a series (#1, #2) of questions occupying my mind at the beginning of 2014. Which may (or may not) inform what I’ll be working on.

3) How can we program on tablets?

I’m now a tablet user. I became a tablet owner at the end of 2012. For six months I played around with it, trying a few Android programming exercises. But I only really became a regular tablet user half-way through the year. Firstly when I put Mind Traffic Control into a responsive design. Secondly when I bought a couple of e-books. And I only really got committed when I did OWLdroid and coupled that with btsync.

So – somewhat late to the party, I admit – I’m now a tablet enthusiast.

And so my question is, how the hell do I program on this thing?

There’s a trivial answer to that question : get an external keyboard, an appropriate editor / IDE and treat it like a normal computer with a small screen. I can do that. I’ve worked a lot on netbooks and small screens don’t freak me out. But that’s not really what I mean.

Because tablets aren’t meant to have keyboards. And a computer without a keyboard challenges one of my deepest held programming beliefs : the superiority of plain text.

Plain-text is so flexible, so expressive, so powerful, so convenient to work with, that I’ve always been highly sceptical of those who want to do away with it. But on a keyboardless computer, it’s a different matter. Plain text isn’t at all convenient without a keyboard.

Especially the text of programming languages which makes rich use of another dozen or so punctuation symbols beyond the alphabet and numerals. And where manipulation relies on cursor-keys, shift and control, deletes (both forward and backspace), page up and down, tab-complete etc.

And yet tablets are becoming ubiquitous. Increasingly they’re the target of our programming, and the tool we have with us. So how are we going to program in this new environment? With multi-touch or stylus but no keyboard?

I have yet to see anything even vaguely plausible as the revolution in programming “language” we’re going to need for this.

I don’t think it’s the “Scratch”-like or “App Inventor”-like “stick the blocks together” languages. The problem of programming on tablets shouldn’t be conflated with the problem of teaching novices to program. (Which is what most visual programming environments seem to be about.)

One issue with that kind of system (and other “flow-charts”) is that blocks need to be big enough to be easily and unambiguously manipulated with fat fingers. But to be decently usable, a programming system should be able to have a reasonable density of information on the screen, otherwise you’ll spend all your time scrolling and forgetting what you’ve seen. How do you resolve that tension?

Perhaps “data-flow” programming of the Max/MSP, PD, Quartz kind. Piping diagrams. Process Modelling packages have something to teach us about orchestrating in the large. But they are shockingly clumsy for certain fine-grained activities that are expressed easily in text. (Eg. how the hell can you talk about tree-shaped data or recursive algorithms using this kind of piping model?)

So I don’t have any information about who is doing interesting work in this area. (Aside : while writing this post, I thought I’d consult the collective wisdom on StackExchange. Needless to say, my question was immediately shot down as too vague.) But I’m now very curious about it.

Facebook released a plugin for Quartz Composer which lets you design interactive mobile apps.

That’s interesting. I don’t really know anything about Quartz. But I believe it’s like Max/MSP or PureData. A “dataflow” design system. Dataflow makes a lot of sense for plumbing together multiple services. And I’ve long believed it could / should be applied to orchestrating web-scale systems. For a while I hoped that Yahoo Pipes would go in this direction.

So will this be successful? On first glance, it looks like they’re trying to work at a very low level with the diagrams. In Max/MSP and PD, this is actually more fiddly than writing code. It’s the larger scale where diagrams shine. Still, it may well take off with a certain audience. Interesting move. I wonder if someone will try to copy it in Max or PD.