Welcome to Cardigan Bay

Cardigan Bay is my new wiki-engine written in Clojure / ClojureScript.
I’ve been working on it for about three months, and it’s a long way from finished, but now interesting enough for people to start playing with (and hopefully giving me some feedback on).
The goal is for this to be the new engine behind ThoughtStorms, my personal wiki notebooks, and other sites. It will supersede Project ThoughtStorms (the Python codebase), SdiDesk, OWL etc. though hopefully incorporate the best features of those.
It is a major move in my plan to unify much of my software in a single app and code-base. Though it doesn’t, so far, fully merge into Mind Traffic Control.
I’ll be talking more about it over the next few posts. For now, the best overview is the readme page of the project, which you can find on one of these repo-hosting sites :

GraphQL

I’m late to this party.
But I’m starting to look into GraphQL and realize that it’s very much speaking to many of the things I’ve been thinking about recently.
A nice clean schema-first declaration of data that you care about and want to interact with. The schema representation isn’t quite as elegant as I’d ideally hope for, but more or less what you’d expect.
This is a great intro talk from the guy behind the leading Clojure implementation (that looks pretty nice).

Howard Lewis Ship

Meanwhile, this is a good short documentary of where it came from. And, of course, the needs of Facebook explain some of the design decisions (including why it has to be well designed and powerful. Something clunkier wouldn’t get into production at that scale.)

Travel Report

So as noted previously, I went travelling without a notebook, but with a Xaomi tablet, a portable keyboard, and a copy of UserLand Linux.
How did that go?
Surprisingly well. Actually I ended up using another keyboard, a cheap “cover” with keyboard built in, no Bluetooth but micro-USB. And I had to get an adaptor (more expensive than the cover) from micro-USB to USB-C.
But it worked OK.
Except … no question-mark key. Which was a real pain. It had one, theoretically, but no combination of keys seemed to make it come up.
So I would have to unplug the keyboard to get Android’s own virtual keyboard every time I needed a question mark. (Except where Emacs autocomplete of Clojure names provided the name)
Insane!
But apart from that …
lots that was very good.
I ended up loading Ubuntu into UserLand. (This was obviously ARM Ubuntu) … and it all just worked like a Ubuntu Linux should.
I was running Mind Traffic Control, no problem.
I was running Emacs, no problem.
I was coding Clojure, no problem. I coded Python, no problem. I ran and played with SWI-Prolog. No problem.
I didn’t end up working on what I thought I would work on, but I did a remarkable amount of Clojure programming.
Yes it was a tiny screen and keyboard and I’m happy (and more comfortable) to be back home now on my larger laptop. But it didn’t stop me doing real work. Checking it into git, and pushing it to GitHub.
I even did real Python programming, without the external keyboard, in portrait mode, in Emacs.  And, a bit of Prolog.
I was also successfully running a web-server based Clojure application, and interacting with it through the standard Android Chrome browser.
It all basically did work.
This tablet is plenty powerful enough for the kind of things I’m doing.
And Linux is Linux.
And Clojure is awesome. 🙂
So … I’m happy.
What’s still a bit of an outstanding issue is synchronization.
The code I was writing that can be seen by the public, I could just push to GitHub. The data I was working on which wasn’t public, I could only sync when I got home, using rsync.
It’s OK, but it’s a fiddly manual job.
And incomplete rsyncing + git (+ syncthing I wasn’t using on the trip because I had no permanently on machine to sync to) seemed to lead to corruption of git repositories.
But, in general … this is a good (or certainly good enough) tablet. Linux is great. Clojure, Python, Prolog, Emacs and Git are all awesome.
And I’m hoping I can establish a rhythm / modus operandi where I can take the tablet as a laptop substitute (because there IS a significant weight difference)
This is still only the beginning of making my tablet actually useful rather than ThoughtStorms:WhyIsMyPhoneSoUseless
But I’m more confident.

Holiday Reading

I’m about to go on holiday in January. And it’s time for a new adventure in travelling without my laptop. I can’t go anywhere without the safety blanket of a Unix computer. And since my PocketCHIP got stolen, I’ve been without something portable.

Anyway, what you see in the photo is my new Xaomi 8″ Pad 4 tablet. With UserLAnd Debian, running Emacs, with Sam Aaron’s EmacsLive (which is my preferred packet of Emacs modes and extensions for writing Clojure). I’ve opened up the source-code of Liquid, which is a nice project to create a powerful text editor, heavily inspired by Vi and Emacs (though not aiming to be a clone of either) written entirely in pure Clojure.

I’ve got Clojure installed on the machine. Along with git, leiningen, Python3 and SWI-Prolog.

I’m able to compile and run Clojure (including this Liquid project). So that’s one sort of holiday project I might find myself getting up to. Playing with and seeing if I can figure out how to write new “modes” or extend Liquid to my own nefarious purposes.

Or, I can continue my playing with Prolog.

I also have a Bluetooth keyboard, which is nice and small and portable, and seems to be OK. (After a couple of hours playing). Obviously because it’s small so I don’t know if I can adapt to doing a lot with it. But … so … far, the bluetooth connection seems fairly reliable, and not dropping.

So, the whole package is remarkable small and light. I would have liked one of those keyboard covers, but no one seems to be selling them here in Brasilia (the only one I found only has an old microUSB connection, not the USB C that the tablet uses)

In terms of software, including powerful programming languages and a powerful editor (Emacs) … this looks viable. Of course tablets are powerful things these days. The whole UserLAnd Ubuntu with all the above software is taking up only a small chunk of the 64 Gb storage.

The big bottlenecks will be the keyboard and screen and working without a mouse.

CLOjure SHell

I’ve been playing with CloSH, the Clojure Shell.
And I have to say, I’m sold.
If you’d like to have the full power of Clojure. And to write shell scripts that look like this :

(doseq [f (expand "*jpg")
    (println f)
    (sh identify -format "%wx%h" (str f))
    (println))

rather than in hard to remember shell-script language.
Then this sorts you out.
Tab completion is a bit weirdly slow. And there are some glitches. But it basically works as you’d want / expect.
I’m now running it and, as of today, using it to do real things (eg. that batch process of analyzing jpegs, above).

What's up? August 2019 Edition

Gosh … we’re half way through the year already.

So what am I up to in programming?

Clojure

I still think it’s a wonderful language. Not doing as much as I’d like. But it’s still great.

One thing I did a few months ago was port the terminal version of Mind Traffic Control to it. Which you can get here : https://gitlab.com/interstar/MTC-CLJ

And I’m happy with it. Works fine. I’m using it every day.

That means that my experimenting with Racket is unfortunately at an end. I don’t currently see myself going back to using Racket for anything in the near future. Nice as it is, I think I’m committing to Clojure for my Lispish stuff at the moment.

C++

Huh? Yep. I’ve been doing audio programming recently. And, unfortunately, until https://github.com/soul-lang/SOUL goes mainstream, there doesn’t seem to be much alternative.

In particular I’m playing with plugin development for VCVRack, which is surprisingly civilized. And yeah, in that specific context, with their nice API and nanovg it’s all fairly OK.

Which makes me also want to try another go at VST programming with JUCE. (I’m doing a lot of music in FL Studio these days, too). And I’m doing something on Android with Oboe.

Oboe is OK, but Android is hell!

Flutter / Dart

Which brings me to Dart / Flutter which I started playing with today. And already, I’m sold. I never bothered to look at Dart before. It looked like the kind of “bit nicer than Java” language which is probably good to have, but has never got my pulse racing.

But Flutter … literally I’ve wanted to do stuff on Android for ages. And HATED every time I’ve tried to do much with it. The whole experience is ghastly.

But finally … Flutter actually looks like some sanity has been brought to Android. Yey! It’s all in one file. Yey! A language that allows complex data declaration and therefore can use this to define reactive UIs declaratively. Yey! Hot-reload (like Figwheel for ClojureScript).

Finally. Some sanity.

OK, this might be premature. I haven’t actually done anything more than the hello world tutorial. But it does look like it’s an acceptable Android programming language / framework.

Next challenge, can I call native C++ libraries like Oboe from it?

Ferret

I still think Ferret (a compile to C++ Clojure-like) looks promising. And I asked the inventor if I can make libraries for inclusion into C++ code with it.

You can!

Great. So once I’ve got a moment to play, I’ll see if I can write some pure data-model / code stuff in Ferret, compile it to C++ and try to use in a) my music context and b) natively on Android.

Python3

I’m in the process of moving my code-bases to Python3.

I think TSWiki code (and the libraries) are pretty much done. FSQuery theoretically but I found a bug. I think OWL is getting there but not released.

Basically I am on schedule that all my non legacy / abandoned Python is going to be Python3 by the end of this year.

Prolog

I’m still fascinated by the potential of Prolog or Prolog-like declarative / logic / inference-engine enabled languages.

And I’m actually playing a bit with SWI-Prolog at the moment. It is VERY GOOD.

Running a web-server and putting a web front-end to Prolog turns out to be fairly straightforward and very concise. Actually generating HTML from the DCG-based DSL is also surprisingly good. More or less equivalent to my own Python library for this

If I can just figure out how to do some of the things I’m now imagining Prolog is good for (see my last post) in Prolog, then I might well be doing some fun stuff here.

So yeah, that’s me, August 2019 : Clojure, Python3, Dart / Flutter, C++, Prolog

And I’m finally getting my development environment into Docker containers.

How would you design the perfect programming language?

Another Quora answer I should be moving here : Phil Jones’s answer to How would you design the perfect programming language?
 
Well, the obvious way you’d approach it, is start with a reasonable knowledge of using other languages to get a sense for their strengths and weaknesses.
And start with some good role models.
And then see where you can go by taking a good language and incorporating ideas from elsewhere.
It’s no secret that I think that Clojure is the nicest language I’ve ever used.
Clojure’s virtue is that
a) it’s built on very firm, 50 year old foundations, namely Lisp.
AND
b) it was willing to do a clean slate reinvention, throwing out the cruft that had built up in other Lisps and Schemes. And bringing in good ideas from elsewhere.
I think that’s how you make a great language. Start with an already good one. And improve it with housekeeping and tasteful borrowings from elsewhere. (Note the “tasteful” … C++ took a good idea “object orientation” and kludged it into a good language, C. And somehow came up with a worse language. (Phil Jones’s answer to Why is C++ considered a bad language? )
That selectivity. Being willing to throw out bits that either got crufty or just didn’t fit the vision (eg. reader macros in Clojure) is as crucial a part of making a nice language as choosing good features to add.
So how would a great language seem to me today?
As I said here, languages need to express

  • a) computation (ie. functions and compositions of functions)
  • b) constraints
  • c) data-structures
  • d) “architecture”

From a language design point of view, I think “computation” is a solved problem. I think it’s great to do computation with function application and composition. I’m pretty happy with the way Lisp represents that. But I understand people who might prefer a different syntactic sugar for expressing functions like the ML / Haskell families. Or even if you want to start with Smalltalk’s objects passing messages, the “concatenative” approach of Forth, Erlang’s actors, or Prolog’s relations, then you might get somewhere interesting.
I’ll skip constraints (largely types and contracts) for the moment. And focus more on c) and d). A good language must have good ways of expressing complex data structures. This starts with a good way to express literals. Eg. strings, arrays, dictionaries etc. But moves on to expressing more complex data-schema. Perhaps grammatically.
I like Clojure’s EDN, which is like JSON, but having it really built into the language is very powerful. But you can go further with grammatical descriptions of data. Things like Clojure’s Spec. Or Shan’s “sequence calculus”. Or Haskell’s algebraic data-types etc.
Ironically, “object oriented” languages have been rather poor at expressing complex data. The most egregious ugliness of the C++ / Java type OO is due to just how painful it is to construct complexes of objects. You end up having to procedurally call operations on classes and objects saying “create a new instance of an X, now attach a new instance of a Y to it. Now attach to the Y, a reference back the Z from this scope”
It’s that mismatch between a program that thinks of itself as a network of interacting objects, and a language that has no explicit concept of “a network of interacting objects” and no convenient way to express it, that makes Java and friends such hard work. Somewhere deep in the heart of the UML is a good intuition, struggling to find a viable implementation.
That’s what I call “architecture”. The structure of the program itself. But architecture is also the shape of the UI. And the bindings between events and handlers. Or the representation of the overall network of different actors … the client, the server, the database etc.
A language which is good at defining data-structures could be good for defining all these architectural elements if you just added a few more bells and whistles to it. (See Phil Jones’s answer to Which is the best programming language for developing a GUI application? )
I’m fascinated by Jonathan Edwards’ concept of “social data-types”, ie. data-structures with built in declarative rules for how they should be synced and distributed and accessed across a networked application. This seems to me to be exactly the kind of thing we need in new languages : the number of useful algorithms to define how data is moved and synced and controlled around a network is probably small. They are fiddly to implement, but 99.999% of applications just need to state which they use.
For example, it should be possible to say stuff like “this is a thing called a Profile page. It has these fields. Each Profile page belongs to one SystemUser. And each SystemUser will have one ProfilePage. Only the SystemUser who owns the ProfilePage should be able to edit it. But all SystemUsers should be able to see / read it. Changes on the owners client will be automatically synced to the server, and changes on the server will be automatically synced to any non-owner’s client which is looking at the page”.
Don’t get hung up in the English. I’m not asking or interested in this being natural or “English-like” language. I’m saying it ought to be possible to declare these facts simply within the declaration of the data schema, and have the language’s own “synchronization engine” automatically do it.)
Now, of course, languages as diverse as HTML, SQL and Prolog are declarations of stuff with some engines behind the scenes to do the work (inference engines in Prolog, query engines in SQL, layout engines in Prolog)
But each individual engine is limited. Ideally we want to be able to have multiple engines for multiple purposes.
What I’m feeling towards, then, is

  • a language which is great for defining data-structures and architecture (both general schemas and particulars)
  • a language which is good for defining “engines”. I’m not sure I mean “implementing” engines. Perhaps they need to be implemented in some lower level language. But it should be possible in our high level language to say “here are some rules for something called “synchronization” which will be represented in this way within the program, will be handled by that engine, with these constraints or pre/post conditions”

The nice thing about rules is that they are easy to compose and have what Quildreen Motta seems to be calling “extensibility”. You can add or remove rules without worrying about how to slot them into the order of execution. You expect the engine to worry about that, and how to compose these types of rules together meaningfully.
A final point.
Yes, a good interactive mode of development inside a good IDE / REPL is crucial.
But I’ve been thinking for a while if it’s possible to go further.
Programming is still very bottom up. Even if we have a top-down view of our system, when we know it consists of modules X, Y and Z and they have to interact in this way. our development tools don’t really capture this information until we program it at the lowest level.
We have a bunch of tools that help us work with and reason about the higher level. We can write interfaces and type information and protocols etc. But until we instantiate them the language doesn’t really want to know.
We have unit tests. Which we ought to write first in TDD. In something like Clojure Spec, though, we can actually generate unit-test examples from our Spec.
That’s quite good but it’s a side trick.
What I think is one of the big pains of programming is that the computer demands we address problems in an order which suits it.
But imagine if we broke that restriction. Imagine if we could sit down with the IDE and have a “conversation” (remember I’m still not talking about English / natural language, just the ordering) like this :

Me : I’m going to build a system called “MyAPP” with a client and a server.
Computer : I got that, I know roughly what a client and a server are. And I’m storing our work under the name MyApp.
Me : Actually the client needs to run in both browser and on mobiles.
Computer : sure
Me : here’s the data-structure for the User and the UserProfile. And there’s a one-to-one relationship between them.
Computer : OK. Gotcha
Me : and the API basically consists of these commands. Oh, and to get started, here’s a state-machine I’m declaring for one of the interactions.
Computer : Understood. I’ve stubbed out the API and that protocol for you. Navigate here in your browser and try (with the default forms) some interactions based on that state-machine.
Me : thanks. Now I look at them, those forms are good, but I need to use a larger text box for the user description. And the classification_code is one of [:member, :editor, :administrator]
Computer : sure, I’m guessing that’s a drop-down in the UI, but how are you representing it in the UserRecord?
Me : On larger (browser pages) I prefer Radio Buttons, but on smaller pages and the mobile apps, yes a drop-down. In the data model I’ll go with it having to be one of these symbols, you can add that information to the schema of the UserRecord
Computer : already done.
Me : So now, this bit of the API defines the Admin controls. These should be limited to users who have the :admin classification_code.
Computer : Yep.
Me : Now, on that Dashboard page, I want you to add a chart that subscribes to a value called AllUserStats.flurpsPerMonth. It’s going to be on the form of integers against months.
Computer : OK. So I’m showing you some graphs with some random numbers against month data I just made up. Does that look OK? And how do you really really want to calculate flurps?
Me : it looks fine for the moment. So flurps are calculated from woobles. I know you don’t know anything about woobles yet, But woobles are floating point numbers between -2 and +2, and flurps come out of left-folding all the wooble information for each User’s month like this.
Computer : OK. I did that. You now have a flurp_calc transducer that’s hooked on to a stream that subscribes to User.woobleData (I don’t know what that is yet.) And the Flurps Per Month Graph on the admin page is now subscribed to that. I’m still making random shit up in User.woobleData, but at least those numbers are between -2 and +2. If flurp_calc is still wrong, that’s your fault.
Me : No probs. You’re doing good work computer. I gotta rush, but I’ll back tomorrow to pick up where we left off.

Now … just to re-iterate. This is NOT about natural language or Artificial Intelligence that means we can just talk to the computers.
I expect the above dialogue to happen in completely formal unambiguous statements using a sensible and efficient syntax.
What it’s about though, is a development environment that lets me, the programmer, add the decisions / constraints that define the system in the order that’s convenient for me. Behind the scenes there’s a model of the application. And that model is always kept in a state which can be run and tested, as a prototype / wireframe / stubbed or mocked up. It may break because I’ve put a bug into it. But if there is no bug then it’s never not doing something because I haven’t got around to doing something else yet. I’m jumping backwards and forwards between Model, View and Controller. Giving advice on fundamental algorithms, on architecture, on UI tweaks. I don’t need to build my application from the bottom up. I have a working application from the start, albeit one that does nothing. And I just keep adding extra information to refine and incrementally extend that application. In whatever order the ideas come to me.
Finally, I want to emphasize, this is not meant to be a description of a “web-app builder” tool. Knowledge of clients, servers, mobile-devices, web-applications etc. are in the pluggable engines. All the language knows about is :

  • declaring data-structures and architecture
  • declaring rules (how they look, and how they interface with an engine)
  • describing the small amount of extra computation my system needs to do, probably in the form of functions

Beyond that, the development environment needs to know how to add new data-structures / functions and rules in the form of a dialogue, while keeping an up-to-date, coherent model of a system which is automatically generating a working prototype of that system.
Plug the Social Apps Framework (including engine) into the language / IDE at that point, and you now have the ability to grow your system via a “conversation” like the one above.
OK. Final repetition. The “perfect language” (or at least next leap forward) language I’m dreaming of is :

  • a language with great expressivity for defining data-structures and rules
  • a language where we expect to find most of the application (or “business”) logic written in the form of declarations of data-structures and rules which are handled by pluggable engines.
  • a language where it’s easy to add and remove engines. Ideally perhaps the language can help us write the engines too.
  • a language whose development environment affords a “conversational” way of building up a program by adding extra rules / constraints to an always live system
  • I personally would start with Clojure … which already has a good Lispish syntax for functions and macros, and, in EDN, the beginnings of an OK data-structure language. But if you prefer to start from Smalltalk or Forth or Haskell or Prolog and do the Rich Hickey trick of a clean-slate refresh, keeping the good ideas, tastefully borrowing other good ideas, and removing the cruft, then that’s good too.

Related :

Tools for Reverse Engineering Binary File Formats

On Quora I asked a question : https://www.quora.com/What-are-some-good-tools-to-help-reverse-engineer-obscure-binary-file-formats

What tools are there to help reverse engineer obscure binary file formats?

So far the answers aren’t particularly informative. Basically hex editors. But I feel there must be something more. Here’s a comment I replied to someone there :

Well, if I try to imagine “something better”, for example, I can think of a program that I could feed a lot of different example files of the same type to, and have it compare across them to identify common sequences or patterns at particular points. (This kind of software exists for analysing databases of DNA, trying to understand the genes, for example)

I see Professional Text/Hex Editor with Binary Templates has templates for describing file formats. But I’d like to have the equivalent of a simple BNF-style grammar notation to describe formats and be given instant feedback, as I’m experimenting with a grammar, whether it fails in matching any of the files of the type I’m trying to reverse engineer.

If I know some information about the kind of data that is being represented, I’d like to be able to give my tool “hints” about it, which it can use to try to deduce what means what.

For example, I’d like to be able to say “this vector image file contains just a red triangle, that one, just a red square and that other one, just a blue square. Given that, what are the likely candidates to represent red, and what are the likely candidates to represent a square?”

Surely there must be stuff like this out there somewhere?

2019 – Time for Python 3


OK, it’s 2019

Finally time to upgrade to Python3 for all my Python projects.

In particular that means ThoughtStorms wiki and Project ThoughtStorms

If you are using either the project or the thoughtstorms library then be prepared that they will all be moving to Python3 in the near future.

Given that Python2 is going to be deprecated next year, and I don’t really have the time and inclination to support both, I’m moving to Python3 this year. And all my projects and code will be, too.

2to3 makes it all pretty easy.