The Future of Lisp

Another rescued Quora answer to the question : What is the future of the LISP programming language?

I’m writing this answer as an unabashed Clojure fanboi.

I hope that the Lisp community recognise what I think are the two big advantages that Clojure brings to Lisp. And that future Lisp family languages keep them and build on them.

The first of these advantages is immutability. Most FP languages go hard for immutability (it’s more or less the essence of FP). But for historical reasons Lisps have tried to sell themselves to a wider public on being “multi-paradigm” and so have been tolerant of mutable state.

I’m a fan of immutability though (Phil Jones (He / Him)'s answer to In your experience, are immutable data structures and paradigms easier or harder to understand in general (and especially for newcomers) than regular imperative mutable programming paradigms? Why?) and I hope very much that we’ll see both growth in Clojure. But also newer Lisps will follow its lead in restricting state to very specific mechanisms like “atoms” etc.

The second of these advantages is syntax for data-structures.

As I’ve mentioned elsewhere ( Phil Jones (He / Him)'s answer to If Clojure is one of the most expressive languages of today and has similar expressive power as Common Lisp, which goes back to early 80s, can we say that the field of programming languages hasn't progressed much in the last 40 years?) representing “computation” is a solved problem in programming languages. And Lisp’s minimal syntax is just fine for it. But there is scope for better representation of data. And I believe that Clojure’s EDN demonstrates that it is worth trading the cost of a little bit of extra syntax, in return for better ways to express complex data literals in our programs.

Today, for anyone thinking of inventing a new Lisp, I would strongly urge them to adopt both of these features from Clojure. Build the language on EDN with the vectors and maps and keywords standard along with the usual parentheses for list syntax. Implement these standard data structures as immutable. And restrict mutable state with specific mechanisms like atoms and refs.

Now, if you read that second answer linked above, about expressivity, you’ll know I have some further opinions about language design. And syntax for representing data, architecture and constraints.

I’m very ambivalent about static typing. I tend to be in the dynamic typing, not caring much for static types camp. With occasional visits to the “OMG I wish I had a way of setting and testing type constraints on this mess” camp.

Clojure has Spec. Which I think has both some good features. And some features I can’t get on with.

Two developments that interest me greatly are Shen Language and Carp Lang

Shen is a Lisp that adds a powerful type checker. One which is almost a kind of Prolog-like inference engine.

Carp is an interesting mix of Clojure syntax, but with low level memory management inspired by Rust (a borrow checker). And a type system with type inference that’s more like the ML family of languages than a Lisp. It has these features because it’s aimed at writing low-level code for embedded systems, games and audio applications etc.

But one thing that distinguishes it is that it has, what looks to me, the nicest, most Lispish and convenient way to add type checking I’ve seen in a Lisp. You wrap any expression with what’s effectively a compile time assert that the expression result is of a particular type.

So you write something like :

(defn f [x y]
  (the int
     (+ x y)))

That’s it. You’ve typed the expression (+ x y) as having to return an int. (That’s what (the int sexp) does. )

And then type inference lets that constraint percolate through. Eg. it recognises that the arguments x and y must be ints and the return value of the function will be one.

This feels right to me in a way that other optional / gradual type-checking in Lisp doesn’t. Maybe I just don’t get it, but Spec comes across as the opposite of Clojure’s usual elegance of beautiful design. It’s ugly and clunky.

There are definitely days I dream of “the next Clojure” (ie. the next step forward in Lisp) which is like Clojure except that Spec has become as powerful as Shen, and as elegantly easy to slot into existing code as Carp. With compile-time checking.)

The Free Software Foundation #1

I’ve been answering a couple of questions about the Free Software Foundation on Quora, recently.

Here’s the latest.

Should social (software) freedoms as espoused by the Free software Foundation trump technical superiority as implied by the Open Source development paradigm?

Yes.

The political message of the Free Software Movement is much bigger and more important than the “look this is a cool way to develop software” message that the Open Source people developed.

Our world is being eaten by software. Everything we do in our lives today has a layer of software mediation. And that layer of software is getting thicker, and the layer of free humans with their own discretion to make decisions, is getting thinner.

So at the supermarket you pay a machine rather than someone at a checkout. If you buy on Amazon, almost all your shopping interaction is with server farms, not service workers. Same with most companies, you deal with websites and bots rather than receptionists. We learn from online videos rather than take personal classes. Factories build things with more and more robots and 3D printers and fewer and fewer humans assembling and welding parts. We manage our social relations via social media rather than meet in person in pubs and cafes. Yes, we love socializing and we do a lot of pubs and cafes too. But the proportion of our social lives mediated through software just keeps getting higher and higher. We increasingly vote via software. Manage our money via software. Manage our health bureaucracy via software. Police watch us via software. The military fights wars increasingly through software (cyber attacks on enemy computer networks and autonomous drones).

The world is being eaten by software. And that means the limits of your freedom, to what you can and can’t do in this world, are encoded into and determined by the software that the machines run. And those are decided by whoever programs / owns and controls the software.

In this world eaten by software, we have no freedom if we can’t ensure that the software works for us. If the software works for someone else, that someone else rules every aspect of our lives.

And this what the Free Software Foundation and the Free Software philosophy stands for. For the fight to push back against having our lives controlled through software, by giving us the right to see and control what software runs the machines that inscribe our lives.

The Great Software Stagnation

Jonathan Edwards is one of those people I both profoundly agree with and profoundly disagree with.

In his new year essay The Great Software Stagnation he again complains about the way the computer science field seems to be running out of new ideas.

I agree wholeheartedly with his frustration. And, sure, with a few bright exceptions, innovation in computing seems very poor compared to the enormous strides made in the 60s and 70s.

Of course, some of this is that lower-hanging fruit have been picked. Some is the natural inertia of a large number of users of computers these days.

But I continue to oppose Edwards’ theory, that it’s all the fault of the internet and Silicon Valley luring people away with promises of getting rich. I think (as I’ve outlined in this following Tweetstorm response to Edwards) that it’s all the fault of “direct manipulation” as an ideal for UI designers.

The Great Software Stagnation is real, but we have to understand it to fight it. The CAUSE of the TGSS is not “teh interwebs”. The cause is the “direct manipulation” paradigm : the “worst idea in computer science” \1

Progress in CS comes from discovering ever more abstract and expressive languages to tell the computer to do something. But replacing “tell the computer to do something in language” with “do it yourself using these gestures” halts that progress. \2

Stagnation started in the 1970s after the first GUIs were invented. Every genre of software that gives users a “friendly” GUI interface, effectively freezes progress at that level of abstraction / expressivity. Because we can never abandon old direct manipulation metaphors \3

The 1990s were simply the point when most people in the world finally got access to a personal computer with a GUI. So that’s where we see most of the ideas frozen. \4

It’s no surprise that the improvements @jonathoda cites, that are still taking place are improvements in textual representation : \5

The main innovation in Rails is textual (you rethink the Ruby code that describes an ActiveRecord class as a schema definition for the whole system) \6

DevOps tools take configurations that were done through filling in web-forms or obscure config files and puts them back under control of scripts in a real programming language. \7

React is a new higher level set of linguistic constructs to describe a reactive UI … etc. \8

In other words, progress in CS is still possible, but has slowed down in all the places where people have stopped programming in languages where new ideas can be created through grammatical composition and abstraction \9

And where we try to get by with a fixed repertoire of gestures to manipulate data directly. \10

If you want to fight stagnation, you have to fight the cult of direct manipulation, and fight the expectation that computers should be so easy that users don’t need to think or learn new concepts to be able to do things with them. \end

Now, I don’t expect Edwards to agree with me on this. He’s very keen on direct manipulation in his experiments. And I understand that. Direct manipulation is a wonderful thing when your gestures are sitting at the top of a tower of linguistic abstractions. Mapping that ultimate level of power directly to your body and visual system gives you remarkable power very easily.

The problem is that we don’t know how to grammatically recombine and build further abstractions in the gestural domain, so however great that power is, a direct manipulation approach will always freeze you at that point.

Sticking with text and language feels more artificial and clunky, but if you do stick with it, then you at least have a chance that you’ll later on invent a higher level language to go beyond your current state.

A Test of Mind Traffic Control

Today I managed to screw up my todo.txt file that I use with Mind Traffic Control.
I had to recreate it from some backups that weren’t quite as up-to-date as I needed them to be.
And I got a lot of duplication in the file. And obviously items that had been done are now back in it. (Remember, I don’t keep “checked off” items)
So how to fix?
Diffing MTC is complicated. Because you do a lot of re-ordering of the queue in MTC, two todo.txt files can be almost identical, but the diff of them is huge. And it’s hard to understand how similar they really are.
Fixing that problem is kind of easy : you sort both copies of the file into alphabetical order, diff them again, and now it’s much easier to see and deal with how much difference there really is.
So I did that. On Linux simply

cat todo.txt | sort  > sorted_todo.txt.

And then

meld sorted_todo.txt sorted_backup.txt

And that was fine.
Deduping a sorted file is easy. I’m sure there’s a command line tool for it already but I wrote a 4 line Python script.
So now I have as up-to-date a version of the todo.txt as it’s possible to have. And it’s deduped.
BUT …
it’s now sorted in alphabetical order. And has 1506 items in it.
This is going to be interesting.
So the philosophy of MTC is precisely that the order / structure of the overall queue isn’t really that important. It obviously IS important because your “Next Item” is what’s at the front of the queue.
But most of the time you deal with a few items in the front. And then searching / pulling / pushing “tags” to estimated distances into the future.
That’s worked fine. But I’ve never done a mass (and arbitrary) reordering of the queue before.
So this is going to be a big test. Are the facilities of MTC, plus my own previous tagging discipline and ability to remember what to search for, sufficient to the job of managing a list of 1500 items now that they are out of order?
In some ways, I’m happy to have to face this challenge. It’s going to be a very interesting test. I’m optimistic that MTC is up to it. But I’m keeping an open mind. I may find some issues. And have to write some extra functionality to resolve it.

Cardigan Bay Screenshots

I’m going to do some video tutorials shortly.

Just need to find a decent mic and a quiet space.

In the meantime, some recent screenshots of Cardigan Bay

How the wiki Looks
Here you can see multiple cards. The second is an automatically generated “Backlinks” card.
Recent Changes
Here I am looking at a page on IndieWeb. Note that my mouse is hovering over the “Bookmark to 4545” bookmarklet in the browser.
Having clicked the bookmarklet, we are taken to the InQueue page of the wiki. And you’ll see that a new card has been created with the link to that page.
Here I have opened up the “card bar” at the bottom of the card with the bookmark. And I can type in the name of another page to send that card to.
Clicking the Send button creates that new “IndieWeb” page, and moves the card with the link to it.
Here’s something different. We’re editing a card with some short Clojure examples. Nothing fancy, just filtering some numbers. Cards are separated with — and types like :evalraw and :evalmd (eval and render as markdown).
Save the edit and here’s the results. The first example rendered as plain text, the second was treated as markdown rendered as an html bullet list.
And here you’ll see that the results of the last Clojure snippet are in a nice table, thanks to our “double comma” table convention.
Here’s an example of “transclusion”. The CardiganBay page we saw above, has been transcluded into a single card within another page.
Finally some embedding … a bit too big for the screenshot, but here you see YouTube, SoundCloud and BandCamp embed within the wiki
And here was the source-code in the page for that.

Fixing the Lower-Case Mistake

After writing The Lower-Case Mistake this morning, that basically convinced me that I had to move back to having CamelCase file-names for Cardigan Bay.
So … a burst of activity this afternoon, and the automatic lower-casing of file-names is gone.
If you create a page called HelloWorld, it will be in a file called HelloWorld.md
The GitHub / GitLab repos are now updated and this is now the official behaviour of Cardigan Bay.
Obviously it’s a breaking change. Because if you were using it yesterday and your HelloWorld page was in file called helloworld.md the new version of the code can’t see it.
The default Bedrock Wiki pages that come with Cardigan Bay have now been fixed. But I’m afraid you’ll have to change any other page names yourself.
I wrote a simple script to help me convert the ThoughtStorms page-base (of over 5000 pages)
It wasn’t perfect, but it did enough of them that I’m happy to commit to making the change. (I’ll clean up the problem cases by hand)
Here’s the code to fix the names. It basically searches all the links in pages to find CamelCase names. Then prints a list of “git mv” commands that would turn the lower-cased version of those named files back into CamelCased ones. You just have to pipe that output to a shell script file and run it and it will do the changes.
Like I say, not perfect, but took a couple of minutes to write, and does the bulk of the job.

from fsquery import FSQuery
import re

fsq = FSQuery("pages/").Ext("md").NoFollow(".work").FileOnly()
d = {}

def grepFiles() :
    for f in fsq :
        for l in f.open_file() :
            l =l.strip()
            sqrwiki = re.compile("(\[\[(\S+?)\]\])")
            allnames = [x[1] for x in sqrwiki.findall(l)]

            for n in allnames :
                if ":" in n : continue
                if ">" in n : continue
                #if not n.islower() : continue
                d[n.lower()] = n
                for k,v in d.items() :
                    print("git mv '%s.md' '%s.md'" % (k,v))

grepFiles()

You’ll have to fiddle with it a bit, and make sure you have https://github.com/interstar/FSQuery installed. But like I say, it does the bulk of the work.
If you have problems (eg. a big investment in Cardigan Bay data but can’t do this migration yourself), leave me a message and I’ll see what I can do to help.

The Lower-Case Mistake

I’m doing a lot of of work on Cardigan Bay at the moment.

And it’s getting pretty good.

I wish I’d done all this sooner. I spent years with wikis, not writing my own wiki-engine because I thought that would be pointless reinventing the wheel. Then when I did, I realized I could make wikis do all the things I wanted from them.

Then I loved Clojure but kept writing my engine in Python because I thought it would be more straight-forward, a known art, and just quicker and easier.

Now I’ve bitten the bullet and moved to writing my wiki in Clojure (something that seemed yet another round of self-indulgence when I started it at the very end of last year) I realize that it’s exactly right, and Clojure is a super-powerful and easy language to do it in.

I’m making great progress and loving Cardigan Bay … (and very soon there’ll be a new release with lots of nice new features … watch this space.)

BUT …

somewhere along the line I have made a huge, huge mistake.

This goes back to my earlier engine, Project ThoughtStorms, I think.

Or maybe I inherited it from somewhere else. I can’t even remember.

The mistake is that wiki page names are obviously CamelCased for readability. A convention I like very much and continue to use.

But I decided to make the file-names all lower-case. In other words, I throw away the capitalization of the name.

The only place that the capitalization exists, is in the link that leads to a page. So links look OK. And from following a link, we can deduce the capitalization we show the next page.

Which works reasonably. But you’ll notice that in RecentChanges and other queries for OrphanPages etc. the pagenames are all lower-case. They have to be, the file system has no knowledge of the capitalization.

That was a pain, but I could live with it.

Now, though, I’m working on the “export this wiki as flat files” functionality. (Which is going to be fundamental to my new thinking … more on which later)

And I have same problem. Running through the directory of pages, I can render them beautifully, but I have no information as to how the pagename should be capitalized. And so can’t give the static pages a reasonable looking title.

That is an absurd restriction which I certainly can’t live with. (And nor will anyone else)

So what is to be done?

I could decide to store the correct page name somewhere else. Either in the page files themselves, or another database. But that breaks the convention that pages are pure markdown files (or slightly extended markdown files). I don’t want to add my own special header convention and force people to deal with that when migrating into or out of Cardigan Bay. It should be possible to work on CB wikis in your own preferred text editor if you choose.

Similarly, I don’t want to depend on another database.

Or I can try to capture the capitalized version of the names when I scan the links into the temporary database at the beginning of each session. And use that when exporting. Which will work for pages which aren’t orphans.

Or I do an export the way I used to back in SdiDesk, not by grabbing everything in the page-store, but by crawling it via the links. Which would give me the capitalized version of the pagenames.

Or maybe I need to change the convention so we don’t lower-case the file-names. But keep the capitalization. I’m tending towards this. But I have 5000 pages of ThoughtStorms in files whose names aren’t capitalized. And another 4000 in my personal wikis. That’s a lot of work to fix.

Anyway, thanks for listening dear reader … just writing this has been useful for me.

Command line HTTP apps.

I just rediscovered a question I wrote on Stack Exchange. I can’t remember much about the context, but the idea is to have an ordinary “web server” application accessed through http, but simply sending plain-text backwards and forwards to a command-line based client.

command line – Is there a “terminal” style program that talks to a server over http? – Unix & Linux Stack Exchange

The question was :

I’m looking for something that acts like a terminal but lets me have a “dialogue” with a server over http. Something like this :


$ connect http://myserver.com
Welcome to myserver.com
Options
A - Fribble the obsticator
B - List Frogits
C - Show the log
Q - Quit
$ A
Obsticator fribbled
blah blah
blah
$ C
Log file
...
$ Q
Bye

Http is a useful protocol and it goes everywhere, it’s easy to write a server that handles it (thanks to libraries). But you aren’t always in a place where you have a browser or could run one. Sure there are browsers like lynx etc. that run in the terminal, but they aren’t particularly easy. Sometimes you just want something that’s more like the old Gopher etc. systems. But with the benefits of modern server technology.

No-one seems to have written one, so I mocked up a client in Python using the Requests (http://requests.readthedocs.io/en/master/) library :

import requests
import sys
url = sys.argv[1]
print "Connecting ", url
r = requests.get(url)
print r.text
flag = True
while flag :
    s = raw_input()
    data = {"opt": s}
    r = requests.get(url, params=data)
    print r.text

A server now just has to respond to messages passed as “opt” on a particular URL, and to respond in plain-text.

Obviously, something like file-downloads etc. would also be useful.