A Tower of Simple Systems

I posted this on Future Programming at Quora : A Tower of Simple Systems

Re-reading the answer I reposted here a couple of days ago about the “Holy Grail” of programming language design : modification.


So here’s a question.

A dumb solution …

But what if we created a system as a tower of “snapshots” of solutions?

Git and other source control systems already store our development history as a series of snapshots.

Containers already give us the way to snapshot full systems.

Things like Amazon Lambda give us the ability to spin-up functions quickly and cheaply on demand.

So …

… what if, when I build a system, I just make a system that handles the first use-case or user story.

That’s now “frozen” at the base of my tower.

Then a new use-case or user story comes in. Instead of trying to slot handling this use-case into my existing program, I just create a new, trivially simple, solution to handle it. That gets added as the next “floor” of the tower.

Now the system is up and running. A user interaction happens.
The system first tries to see if the current top-floor of the tower can handle the behaviour. If it can, fine. If not then we simply drop down to the layer of the tower below it to see if that can handle the story.

And so on …

Effectively our tower of old versions of the system is acting rather like an immutable database, “story-handlers”. For any user-story we can keep adding new handlers for it to the top of the tower (where it supersedes older versions). Other user stories which haven’t changed, we just fall down the tower until we find a handler for them.
Would this simplify trying to compose complex systems?

Obviously this will work best when each user-story handler is a purely functional program (easy and cheap to spin up) and state is kept in an external database.

What does everyone think? Is this just crazy or a viable way of approaching the problem of growing a system? Ie. embrace the fact that you’re going to get cruft (or layers of archaeology) building up anyway, so just handle that explicitly?

Can we call it “Cruft-oriented programming” 🙂 ?

August is Patterning Month

August is Patterning month again.

I’m back to work on the Patterning library. And, in particular, getting it working properly in the ClojureScript, in-browser version. I’m going to be using devcards, figwheel, spec and other good tools in the Clojure community.

I’ll be revamping the site and new versions of the code.

Watch this space …

Welcome to 2015 (part 2)

Bloody hell! I just wrote a blog post and WordPress lost it.
Grrrr …
The short resume was :
– yes, I’m late (Feb) continuing my “Welcome to 2015” blog posts.
– last year I got the functional programming bug. Haskell is pretty damned good, but I’ve fallen for Clojure because a) dynamic types, b) Lisp syntax(lessness), c) Paredit.
– last year I did more Python, but not much OWL. This year, OWL is calling again. And as well as making me want to rethink Mind Traffic Control in light of it, is also seeming to demand some kind of pivot on GeekWeaver.
I’ll be coding and blogging more on this ongoing development in coming weeks / months.
I’ve also been playing a bit with remoteStorage, but nothing to show yet.

What's Special About Dynamically Typed FP?

The magic of Functional Programming is that you can write the verbs before the nouns.
You have so little commitment to what the data structures are, that it doesn’t get in the way.
Today I find myself mapping across and diff-ing lists of arbitrary ad-hoc dictionaries and tuples.
Had I sat down and had to invent / write methods in terms of classes or Haskell’s parameterized types I’d never have thought of these ad-hoc collections as types to define methods on or even to define functions in terms of.
But as it is, mid-function, I can just decide that what I need is to put them into a collection and process this collection in some way.


OK. Consider me won over.
Lisp is great to work with. The things that grabbed me about FP in Erlang and Haskell (pattern-matching arguments, partial application, lazy evaluation) are all here.
My code is as concise as Erlang and damned nearly as concise as Haskell (I think the line count is similar though the number of characters per line is about twice as high)
Although I respect Haskell’s type system, Clojure’s dynamic typing holds me up less. I *am*, admittedly, hitting more problems that a type system would have picked up. (Far more than I typically hit in Python) but I think as I get more used to the language this will go down.
I’m not being particularly demanding in terms of libraries. (Quil is almost all the libraries I need at the moment.) But I’m finding that there are all these handy things like spit which make me smile. (Remember what this was like in Java?)
Things that would still be a bit fiddly in Python / CoffeeScript continue to turn out to be easier than I imagined they would be when I implement them in Lisp. That’s partly because I default to approaching them as “how do I write a small lambda that can then map across this list” and mostly, by the time I wrote that function, all I have to do is … er … map it across the list.
Lisp is extra-ordinarily compressible. I keep finding ways to refactor and fold up things I did earlier to make them shorter and shorter. And the more functionality I add to my program the more compressed it seems to get.
I am grinning to myself at the thought that my code might be compilable to both Android and into Javascript. I haven’t tried either yet, but if I manage it then Clojure is now my new favourite language.
CoffeeScript isn’t at all bad, but if ClojureScript works out, then I’ll probably start to move aggressively to use it as my default in-browser language. (And yes, I guess that may mean rewriting OWL in it. OWL is still short enough that I think I could sprint it in a couple of days.)
Similarly, if I can compile Clojure libraries that can be called from Java projects then you’ll start to see me more productive on Android. (Beyond OWLdroid I have a couple of other bits and pieces of apps. written, but not taken them to completion. If Clojure slots painlessly into the workflow, it will be a lot more tempting to dive back in.)
So … yeah … Clojure rocks!

Wolfram Language

Stephen Wolfram's Introduction to the Wolfram Language

I’m impressed. Beyond being just a grab-bag of libraries (which it could have been) it seems there’s real thinking about composability here. The laziness of evaluation, the “symbolic-ness” makes it a powerful functional programming language. In fact, it’s almost a “data-flow” language. Of the kind, like Yahoo Pipes, which can become an orchestrator of web-level components.

That already gives some substance to the claim that this is a “new” kind of programming. But what REALLY differentiates it, I think, is the philosophical commitment to what O’Reilly used to call “Data Inside“.

This might well herald a new trend : “programming language as a service”. A language whose essence and value depends on not just the library of functionality, but the continuously updated data living in that cloud. This is genuinely new. (Maybe it contrasts with something I once said about SemanticCommitment, that semantic constraints weren’t the treasure that the SemWeb people thought they might be. Perhaps the Wolfram Language is a bet against me, and on the idea that we now have comprehensive enough databases of “knowledge” about the world, that packaging huge ontologies of semantics within languages is a valued feature.)

It’s also a huge risk to the programmer, of course. You’ll only EVER run Wolfram Language against Wolfram’s cloud. Every program you write will be dependent on Wolfram in perpetuity. And we know where that story leads … the cloud provider’s business model inevitably evolves to being observation and analysis of what the customer is doing. (And it should go without saying that every program you run is explicitly open to inspection by the NSA etc.) But some people will be willing to make that trade-off.

NoSQL and the Tar Pit

In a Quora answer I went back to a theme that I mentioned when discussing Bret Victor a few months ago.

Here’s something that struck me yesterday when reading Out of the Tar Pit which is a very good essay that seems to signal the direction that many smart people think software development should be evolving in : namely giving up on as much explicit state and control flow as possible and moving towards a declarative style or saying just what your program should produce without worrying about how it does it.
I can’t overemphasize how big this idea is. Most important and smart people thinking about software will sign up to the idea that we need to move towards more functional languages, more declarative style, abandon more state and explicit control flow. Perhaps even separate the essential logic of what you want done from the “accidental” hints that can enhance performance into separate languages / parts of the system.
And yet …
And yet, the most widely adopted, commonly used example of this separation of telling what the program should do in one language and performance hints in another, (acknowledged in the paper) was the good old fashioned relational database written using SQL; which did, indeed, allow programmers to declare what they wanted their queries to deliver without worrying about access paths, control flow or performance. And then database admins worked behind the scenes profiling, creating special indexes etc. to improve performance.
Now, since this important paper was written, there’s been an absolute revolution in database circles, called the NoSQL movement, a wholesale rejection of the relational database model and its replacement by systems that hark back to the hierarchical and network databases of the late 1960s, Although NoSQL was adopted by people working on enormous systems across hundreds of thousands of machines, its popularity is so great that a new generation of programmers reaches for NoSQL database solutions (and explicit modelling of data-structures and responsibility for traversing access-paths etc.) more or less by default, even for small prototypes.
So, I’d say that NoSQL is one of the most successful “contrarian” movement. It’s massively popular and “trendy” while going against everything that many smart programmers think and say they want, and what many people had forseen as the future of software development.
It signals either that the argument in Out of the Tar-Pit is wrong : namely because performance is so important that programmers never want to give up explicitly modelling state and defining control-flow, or that people’s intuitions are badly broken.

Programming Language Features for Large Scale Software

My Quora Answer to the question : What characteristics of a programming language makes it capable of building very large-scale software?

The de facto thinking on this is that the language should make it easy to compartmentalize programming into well segregated components (modules / frameworks) and offers some kind of “contract” idea which can be checked at compile-time.

That’s the thinking behind, not only Java, but Modula 2, Ada, Eiffel etc.

Personally, I suspect that, in the long run, we may move away from this thinking. The largest-scale software almost certainly runs on multiple computers. Won’t be written in a single language, or written or compiled at one time. Won’t even be owned or executed by a single organization.

Instead, the largest software will be like, say, Facebook. Written, deployed on clouds and clusters, upgraded while running, with supplementary services being continually added.

The web is the largest software environment of all. And at the heart of the web is HTML. HTML is a great language for large-scale computing. It scales to billions of pages running in hundreds of millions of browsers. Its secret is NOT rigour. Or contracts. It’s fault-tolerance. You can write really bad HTML and browsers will still make a valiant effort to render it. Increasingly, web-pages collaborate (one page will embed services from multiple servers via AJAX etc.) And even these can fail without bringing down the page as a whole.

Much of the architecture of the modern web is built of queues and caches. Almost certainly we’ll see very high-level cloud-automation / configuration / scripting / data-flow languages to orchestrate these queues and caches. And HADOOP-like map-reduce. I believe we’ll see the same kind of fault-tolerance that we expect in HTML appearing in those languages.

Erlang is a language designed for orchestrating many independent processes in a critical environment. It has a standard pattern for handling many kinds of faults. The process that encounters a problem just kills itself. And sooner or later a supervisor process restarts it and it picks up from there. (Other processes start to pass messages to it.)

I’m pretty sure we’ll see more of this pattern. Nodes or entire virtual machines that are quick to kill themselves at the first sign of trouble, and supervisors that bring them back. Or dynamically re-orchestrate the dataflow around trouble-spots.

Many languages are experimenting with Functional Reactive Programming : a higher-level abstraction that makes it easy to set up implicit data-flows and event-driven processing. We’ll see more languages that approach complex processing by allowing the declaration of data-flow networks, and which simplify exception / error handling in those flows with things like Haskell’s “Maybe Monad”.

Update : Another thing I’m reminded of. Jaron Lanier used to have this idea of “Phenotropic Programming” (WHY GORDIAN SOFTWARE HAS CONVINCED ME TO BELIEVE IN THE REALITY OF CATS AND APPLES) Which is a bit far out, but I think it’s plausible that fault-tolerant web APIs and the rest of the things I’m describing here, may move us closer.