Google are making a Python to Go cross compiler

The idea here is that Google have a lot of Python to run, but Python’s Virtual Machines (especially with the Global Interpreter Lock) aren’t all that good for high-performance parallelism.

OTOH built Go! and the Go target machine for this. So they want to make a drop-in replacement that lets them run existing Python code on this infrastructure.

This is good news for Python. I wonder how long it will take to become the GAE python-hosting infrastructure. Perhaps this is Python’s answer to Elixir.

A burst of development energy in a number of directions recently.

And things are starting to self-organize towards the new ecosystem.

Today’s exciting news : OWL makes a very nice desktop app, thanks to Electron.

Here’s the github repo.

I’ll be doing more testing, compiling, packaging this shortly. So that even non-geeks can play with it. But it seems to work fine.

To be honest, the few times I’ve installed OWL on non-geek friends’ machines, the “run a server and look at OWL in the browser” part has confused / put people off. Now things are VERY much simpler.

This is also going to give me some momentum to add a couple of extra features / ideas I’ve been thinking about over the last couple of years.

In the short term … this desktop repo is likely to be where I develop the next round of functionality. Though I’ll be porting these new features back to the Android and web-based versions.

Main issues in my mind at the moment : I’ve dropped the Python based server for this. We’re now purely Coffee / Javascript. That feels cleaner and more convenient. Will I now try do the same for the web version. It makes sense. But straight node? Express? Meteor?

What’s simplest and leads to least repetition of code?

Long live Mind Traffic Control!

No New Year’s Resolutions or even questions this year. But as an honorary Brazilian, my new year doesn’t really start until after carnival, so now would be the time for it.

However here’s a quick update. The answer to the old MTC question is now resolved for me.

It’s time to put the bullet into the Mind Traffic Control that was. The case against it is overwhelming. MTC was written to learn the exciting new world of Google App Engine and the style of web programming of the mid to late 2000s … Python, something almost Django-like (ie. Rails-like). Backed by a relational database-like thing.

It’s a world I later fell out of love with. Python and Django are … fine. But they aren’t what continues to excite me in 2016.

I toyed with Meteor. But how to host on GAE? And, anyway, in 2016 I have the sickness … I just AM a Lisp programmer. Not necessarily a very good one or particularly experienced one. But I’ve succumbed. This is what I’ve been looking for all my life. It’s what I want to do, going forward.

So a rewrite in Clojure / ClojureScript. That presumably can be hosted on GAE. And I can have a wonderful new reactive UI with Om.

Yes … I’ve been playing around with it … but …

But then again. The old GAE database isn’t a good match for the circular queue of MTC. And worse, I’m not keen on storing my personal data on someone else’s cloud. Yes, it would have to be a single-page app. Yes, it ought to talk to remoteStorage and offer Dropbox etc. too.

These are all wonderful things and yes I am playing with them and want to use them. But … MTC? MTC is a todo-list app. There are a million and one such apps. They’re the “hello world” of browser-based GUI frameworks.

I’d love people to experience what’s good about MTC. But is it likely? How would I cut through the noise of those millions of alternatives? (Some of which are very slick.) Could I really get any kind of audience for a todo-list app in 2016? Does it make sense to put my energy in this direction?

And then again … if I’m going to boot up my browser and run a local server, then I have OWL.

OWL is great for more extensive, smart-disorganized note-taking. It’s just that it doesn’t have some of the charms of MTC. It’s not dynamic … tasks sit around and clog up the pages. You have to navigate around to find them. Sure, it’s pretty easy to navigate around – wikiness makes complexes of documents into small-worlds – but it’s still lacking that immediacy of the river / feed of tasks.

So last year I was fairly convinced that MTC was just going to become OWL hosting. But that isn’t what happened. There is still something to MTC, to the todo-queue concept. And it has resisted being subsumed within the OWL paradigm. My early enthusiasm for OWL was wrong about this. And my initial intuitions vindicated.

So what next? These last months I’ve been drawn back to the command-line. And also to Racket. Which compiles to fast executables. (Clojure is great, but the JVM does take a long time to start.)

And I’ve been admiring (again) todo.txt. Which in many ways is the right approach.

And so … I present : the new Mind Traffic Control. Which is, I admit, nothing but a short Racket program. That reads a file called “todo.txt” from somewhere on your machine. And does MTCish things with it.

Philosophy :

– it’s (right now) a convenient command-line tool.

– it’s compatible with your existing todo.txt file. It doesn’t do everything that todo.sh does. But it has its own tricks.

– in particular, it keeps the queue-ness. You only see the latest item. And most of its commands are about flinging tasks you don’t care about now into the future where you don’t have to think about them (yet).

– obviously it loses a lot of what made the GAE-hosted, web-based MTC interesting. There’s no delegation to other users etc. But I’m not sure many people used that.

– very simple. very minimal.

I have to say … I am EXCITED by this … more excited than I’ve been by any potential refresh I could have made to the old MTC paradigm (even rewriting more or less the same thing in Meteor or ClojureScript / OM). This is fresh and different.

The Future : MTC has a new mission.

Firstly it’s going to be MY todo-queue tool. Previously, the web version was always thought about in terms of “what might people want?”. In practice, almost nobody else wanted it. Now MTC’s mission is “what do I need from a tool?” What maximizes my convenience? I like the command-line. I’m comfortable there. That’s where this is going to be.

Secondly, I’m not that into task-management software. I want software to help me DO stuff. And the focus of MTC going forward is going to be to add features to help me do. For example, I have an item with a link I wanted to read. I now read the link and want to post it to a link-blog. Can that feature be added to MTC? Why not? I come up with an idea for a new project and start putting todo items about it into MTC? Can MTC create the project directories for me? Can todo items be exploded into actual scripts? Within this environment? Once again, a direction worth exploring. (In a sense, MTC with extra tools could be seen as exploring the coming UI paradigm of bots in rivers.)

Syncthing is now my synchronization solution. I don’t think I’m going to worry about clouds and hosting, because I want to use horizontal P2P syncing as the way of making sure the queue is on all my devices.

The original MTC site will be updated shortly. You can, already, export your data from it in todo.txt format. That is now the recommended solution for MTC users. The new version of the site will probably continue to let you do that, without adding anything new to your queue. But it will give guidance on how to install and use the new software.

From hereon-in it’s all Lisp. I’m getting more fluent in Racket. I do have a nagging feeling that maybe I ought to convert to Clojure-like dialect. Which would allow me to use the same code in the browser, on a node server or even compiled using Pixie-Lang. I’m still thinking about this and will make a couple of small experiments in the near future and I’ll pay attention … will other people pick up the Racket code? Will I find myself with a new compelling reason to have MTC back in the browser? Is it worth moving to Rackjure to smooth a potential future port? Right now the code is still trivially small enough that I could port relatively quickly. But I’m watching.

Clojure is a wonderful language. The more I use it, the more I like it.

But the default error reporting is catastrophically bad, forcing you to look through screensful of Java stack-trace for the needle of a bit of code that’s actually yours.

I’m sure there are better solutions, but today I just hacked a very quick and dirty python program to filter this result, highlighting the lines that I care about.


#!/usr/bin/python

import sys
p = sys.argv[1]

for line in sys.stdin :
    if p in line :
        print '33[92m' + line + '33[0m',
    else :
        print line,

I made this code in an executable file called “colourit” in my bin directory. Now I can do this :


lein test |& colourit patterning

to, for example, highlight lines containing the word “patterning”.

A few years ago I looked at some of the programs I was writing, both for public release and libraries / scripts for internal consumption, and realized that even I was confused.

I needed a new map to understand what I was doing. Hence I came up with Project ThoughtStorms. A quick and dirty overview of what I was up to in this space of wikis, personal information and knowledge management.

I think that helps. I know what I’m up to, with the legacy of my ThoughtStorms UseMod wiki and SdiDesk, my experiments with Smallest Federated Wiki, the new OWL stuff, etc. It’s still a mess, but it’s conceptually “encapsulated” in a single place.

Now I find myself with a similar situation with a number of different tries at some tools to make it easier for me to produce static web-sites. Originally there was GeekWeaver, a way for me to generate static sites quickly and easily from an OPML outline. That was a great idea … except … it got mixed up with my n00bie enthusiasm for Lisp and desire to create a powerful Lisp-like programming language.

GeekWeaver also foundered on two other issues : soon after I wrote it, there were no popular OPML outliner tools. Although Dave Winer has resolved that issue and I have, indeed, used Fargo to author GeekWeaver programs.

The other, more subtle and pernicious issue is that few people want to develop a site from scratch these days. Most people use huge frameworks with not just Javascript, but CSS, LESS etc. It really doesn’t make sense to try to use a tool which denies the file-system or the fact that you’ll want to be working with a large number of external files.

So the next time I needed to produce some quick static pages I went back to the drawing-board and came up with BootDown : a very simple Python script that lets you write text in Markdown and wraps it in a BootStrap / BootSwatch template. It’s the opposite of GeekWeaver, it isn’t trying to be a rich and clever language that lets you define your own templates, it’s just a quick way to take advantage of a lot of templates that already exist.

BootDown adds two things to conventional Markdown. A shorthand for declaring div tags. And a page separator. So you can still make a multi-page site inside a single file. And you can explicitly reference the div classes and ids you need.

More recently still, I’ve been getting into a real Lisp : Clojure. Today, that’s the language I really want to be working with. And I have a current project that again involves generating a multi-page, fairly static site. But the inputs are a little bit more complicated : some more structured data, not just text. And it makes sense to store it in an outline. So I’ve started writing Clojure code to read OPML and spit out a BootDown Markdown file, to wrap in HTML.

I can see this project starting to grow too.

And so here I am … three Python code-bases : GeekWeaver, BootDown and the OWL back-end server. Two Clojure projects : the beginnings of a library to work on OPML outlines (OPMLKit) and my unreleased code I’m writing for this current project. And some clunky pipelines to use them together. It’s starting to be another mess. And that’s why I need Project GeekWeaver : an umbrella to pull together these different strands together, at least conceptually.

There are some thoughts … would it make sense to pull at least some of the original GeekWeaver code INTO the OWL back-end server? So that “run transformations on this outline” is a standard OWL feature, and OWL becomes GeekWeaver’s built-in IDE? But what about the OWLdroid version that doesn’t have Python? Should I rewrite that code in CoffeeScript. (Or ClojureScript … a working but still fairly slow option at the moment.) BootDown “just works”. Am I better off scrapping the GeekWeaver idea altogether and just focusing on such practical solutions?

Some of this will become clearer as I work through this current site I’m making. meanwhile Project GeekWeaver is the category and (shortly) the site for the high-level overview.

Note to self :

sudo apt-get remove python2.7

Has a pretty dramatic effect on a Ubuntu box. There are a LOT of packages these days that are dependent on Python somewhere inside them (including a lot of Unity) 🙂

My Quora answer to the question “Is it still reasonable to say mainstream languages are generally trending towards Lisp, or is that no longer true?

Based on my recent experiments with Haskell and Clojure.

Lisp is close to a pure mathematical description of function application and composition. As such, it offers one of the most concise, uncluttered ways to describe graphs of function application and composition; and because it’s uncluttered with other syntactic constraints it offers more opportunities to eliminate redundancy in these graphs.

Pretty much any sub-graph of function combination can be refactored out into another function or macro.

This makes it very powerful concise and expressive. And the more that other programming languages try to streamline their ability to express function combination, the more Lisp-like they will get.

Eliminating syntactic clutter to maximize refactorability will eventually make them approximate Lisp’s "syntaxlessness" and "programmability".

In that sense, Paul Graham is right.

HOWEVER, there’s another dimension of programming languages which is completely orthogonal to this, and which Lisp doesn’t naturally touch on : the declaration of types and describing the graph of type-relations and compositions.

Types are largely used as a kind of security harness so the compiler or editor can check you aren’t making certain kinds of mistakes. And can infer certain information, allowing you to leave some of the work to them. Types can also help the compiler optimise code in many ways : including safer concurrency, allowing the code to be compiled to machine-code with less of the overhead of an expensive virtual machine etc.

Research into advanced type management happens in the ML / Haskell family of languages and perhaps Coq etc..

Ultimately programming is about transforming input data into output data. And function application and composition is sufficient to describe that. So if you think POWER in programming is just about the ability to express data-transformation, then Lisp is probably the most flexibly expressive language to do that, and therefore still is at the top of the hierarchy, the target to which other programming languages continue to aspire.

If you think that the job of a programming language is ALSO to support and protect you when you’re trying to describe that data-transformation, then POWER is also what is being researched in these advanced statically-typed languages. And mainstream languages will also be trying to incorporate those insights and features.

OK. Consider me won over.

Lisp is great to work with. The things that grabbed me about FP in Erlang and Haskell (pattern-matching arguments, partial application, lazy evaluation) are all here.

My code is as concise as Erlang and damned nearly as concise as Haskell (I think the line count is similar though the number of characters per line is about twice as high)

Although I respect Haskell’s type system, Clojure’s dynamic typing holds me up less. I *am*, admittedly, hitting more problems that a type system would have picked up. (Far more than I typically hit in Python) but I think as I get more used to the language this will go down.

I’m not being particularly demanding in terms of libraries. (Quil is almost all the libraries I need at the moment.) But I’m finding that there are all these handy things like spit which make me smile. (Remember what this was like in Java?)

Things that would still be a bit fiddly in Python / CoffeeScript continue to turn out to be easier than I imagined they would be when I implement them in Lisp. That’s partly because I default to approaching them as “how do I write a small lambda that can then map across this list” and mostly, by the time I wrote that function, all I have to do is … er … map it across the list.

Lisp is extra-ordinarily compressible. I keep finding ways to refactor and fold up things I did earlier to make them shorter and shorter. And the more functionality I add to my program the more compressed it seems to get.

I am grinning to myself at the thought that my code might be compilable to both Android and into Javascript. I haven’t tried either yet, but if I manage it then Clojure is now my new favourite language.

CoffeeScript isn’t at all bad, but if ClojureScript works out, then I’ll probably start to move aggressively to use it as my default in-browser language. (And yes, I guess that may mean rewriting OWL in it. OWL is still short enough that I think I could sprint it in a couple of days.)

Similarly, if I can compile Clojure libraries that can be called from Java projects then you’ll start to see me more productive on Android. (Beyond OWLdroid I have a couple of other bits and pieces of apps. written, but not taken them to completion. If Clojure slots painlessly into the workflow, it will be a lot more tempting to dive back in.)

So … yeah … Clojure rocks!