Backlinks in Cardigan Bay

People seem to be very excited about Roam Research at the moment. I’m sure it has many qualities (including a decent mobile client). But I’m slightly surprised to realize that one thing that fans seem to find very useful (and almost miraculous) is the automatic back-linking. Ie, the ability to see what pages link to the page you are currently on.

Bill Seitz has always been a fan of this feature, ensuring that it’s prominent on his WebSeitz engine. While, I confess, I’ve always thought that links that people bother to make explicitly are more interesting and should be valued more highly than links that are implicit.

So, my current Python-based ThoughtStorms engine, following on from UseMod etc. lets you search for “pages that reference this one” with the usual text search, simply by clicking the page name, but it doesn’t automatically give backlinks to you on each page.

But on ThoughtStorms you’ll see I do often explicitly put something like  “Context : BlahPage, AnotherPage” at the top of pages where I think context is important.

Nevertheless, automatic backlinks are clearly the fashion. And as I’m working on Cardigan-bay: A new wiki engine in Clojure I figured I might as well add this feature.

So, one of the main ideas in Cardigan Bay is that it keeps an internal database of meta-data about the wiki, including which pages link to which, using core.logic, Clojure’s standard miniKanren library.

So adding the backlinks feature is as simple as adding a core.logic query to this database.


(defn links-to [target]
  (pldb/with-db @facts
    (logic/run* [p q]
      (link p q)
      (page p)
      (page q)
      (logic/== target q)
   )))

That’s basically it. The logic query tests that there’s a link relation between p and q, that both p and q are actual  existing pages, and that q is equal to the target argument to the Clojure function.

This is pretty obvious and straightforward. So then I just create a card containing this data and attach it automatically to each page.

Took basically less than an hour to write the whole thing.

And if you get the source from GitHub, it’s now there.

However, right now this is fine for small wikis with a few pages. On the actual ThoughtStorms data (as you see in the screen shot above), which has over 4000 pages and an order of magnitude more links, it’s way too slow. So I’m going to have to figure out how (and more importantly, where) I’m going to cache the results. And when I have to recalculate them, etc.

Nevertheless, automatic backlinks are now added to Cardigan Bay.

Update :

I just did a quick experiment where I took out the (page p) and (page q) clauses from the query. And it’s now much faster.

Testing that the two pages exist doesn’t really matter much. There’s no real way we’re going to get a (link p q) clause if p doesn’t exist. And while there can be broken links ie. a (link p q) where q doesn’t exist, there’s no way we’ll actually be asking the backlink query about a q that doesn’t exist. So no need to join these extra clauses in. That makes this now a single match query on a single relation and much more tractable.

Android : Extended SurfaceView's onDraw() method never called

This has just resolved something I’ve been beating my head against the wall over for the last several days.
Source: android – Extended SurfaceView’s onDraw() method never called – Stack Overflow
WHY? Android?
Why are you so convoluted and perverse?
So … yeah … subclass the SurfaceView class … and you’d expect to be able to custom draw by over-riding its onDraw method, right?
I mean, that’s not an unreasonable assumption to make.
Turns out … no. You have to do some weird black magic. Because by default the onDraw isn’t called for “efficiency” reasons.  The StackOverflow answer sort of explains. But doesn’t really.
Why is Android so convoluted and perverse? Why isn’t there a better API to this?


PS : please take care and stay safe in this time of COVID19. I’m self isolating as far as possible. And I recommend you do too. The more we can slow its spread, the fewer people will die. It’s that simple.

Mind Traffic Control : Site Going Down

I’m going to close down the Mind Traffic Control site on Google App Engine.
The online app has been dead for some time. While I’m very happy with my command-line version which I’m using every day.
The only reason the site is still up on GAE is for any old users to export their data. As that’s been the case for the last couple of years, I’m assuming that if this includes you, you’ve either exported your data or don’t care.
If you haven’t, and still want it, get it soon. (And drop me a comment here) Because you will soon lose the opportunity.
The MTC domain isn’t going anywhere. But will be refocused on my two ongoing projects : the command-line version of MTC and Cardigan Bay. And whatever integration I finally figure out between them.
Both of these projects are written in Clojure. And I’m excited about them.


PS : please take care and stay safe in this time of COVID19. I’m self isolating as far as possible. And I recommend you do too. The more we can slow its spread, the fewer people will die. It’s that simple.

Welcome to Cardigan Bay

Cardigan Bay is my new wiki-engine written in Clojure / ClojureScript.
I’ve been working on it for about three months, and it’s a long way from finished, but now interesting enough for people to start playing with (and hopefully giving me some feedback on).
The goal is for this to be the new engine behind ThoughtStorms, my personal wiki notebooks, and other sites. It will supersede Project ThoughtStorms (the Python codebase), SdiDesk, OWL etc. though hopefully incorporate the best features of those.
It is a major move in my plan to unify much of my software in a single app and code-base. Though it doesn’t, so far, fully merge into Mind Traffic Control.
I’ll be talking more about it over the next few posts. For now, the best overview is the readme page of the project, which you can find on one of these repo-hosting sites :

GraphQL

I’m late to this party.
But I’m starting to look into GraphQL and realize that it’s very much speaking to many of the things I’ve been thinking about recently.
A nice clean schema-first declaration of data that you care about and want to interact with. The schema representation isn’t quite as elegant as I’d ideally hope for, but more or less what you’d expect.
This is a great intro talk from the guy behind the leading Clojure implementation (that looks pretty nice).

Howard Lewis Ship

Meanwhile, this is a good short documentary of where it came from. And, of course, the needs of Facebook explain some of the design decisions (including why it has to be well designed and powerful. Something clunkier wouldn’t get into production at that scale.)

Travel Report

So as noted previously, I went travelling without a notebook, but with a Xaomi tablet, a portable keyboard, and a copy of UserLand Linux.
How did that go?
Surprisingly well. Actually I ended up using another keyboard, a cheap “cover” with keyboard built in, no Bluetooth but micro-USB. And I had to get an adaptor (more expensive than the cover) from micro-USB to USB-C.
But it worked OK.
Except … no question-mark key. Which was a real pain. It had one, theoretically, but no combination of keys seemed to make it come up.
So I would have to unplug the keyboard to get Android’s own virtual keyboard every time I needed a question mark. (Except where Emacs autocomplete of Clojure names provided the name)
Insane!
But apart from that …
lots that was very good.
I ended up loading Ubuntu into UserLand. (This was obviously ARM Ubuntu) … and it all just worked like a Ubuntu Linux should.
I was running Mind Traffic Control, no problem.
I was running Emacs, no problem.
I was coding Clojure, no problem. I coded Python, no problem. I ran and played with SWI-Prolog. No problem.
I didn’t end up working on what I thought I would work on, but I did a remarkable amount of Clojure programming.
Yes it was a tiny screen and keyboard and I’m happy (and more comfortable) to be back home now on my larger laptop. But it didn’t stop me doing real work. Checking it into git, and pushing it to GitHub.
I even did real Python programming, without the external keyboard, in portrait mode, in Emacs.  And, a bit of Prolog.
I was also successfully running a web-server based Clojure application, and interacting with it through the standard Android Chrome browser.
It all basically did work.
This tablet is plenty powerful enough for the kind of things I’m doing.
And Linux is Linux.
And Clojure is awesome. 🙂
So … I’m happy.
What’s still a bit of an outstanding issue is synchronization.
The code I was writing that can be seen by the public, I could just push to GitHub. The data I was working on which wasn’t public, I could only sync when I got home, using rsync.
It’s OK, but it’s a fiddly manual job.
And incomplete rsyncing + git (+ syncthing I wasn’t using on the trip because I had no permanently on machine to sync to) seemed to lead to corruption of git repositories.
But, in general … this is a good (or certainly good enough) tablet. Linux is great. Clojure, Python, Prolog, Emacs and Git are all awesome.
And I’m hoping I can establish a rhythm / modus operandi where I can take the tablet as a laptop substitute (because there IS a significant weight difference)
This is still only the beginning of making my tablet actually useful rather than ThoughtStorms:WhyIsMyPhoneSoUseless
But I’m more confident.

Holiday Reading

I’m about to go on holiday in January. And it’s time for a new adventure in travelling without my laptop. I can’t go anywhere without the safety blanket of a Unix computer. And since my PocketCHIP got stolen, I’ve been without something portable.

Anyway, what you see in the photo is my new Xaomi 8″ Pad 4 tablet. With UserLAnd Debian, running Emacs, with Sam Aaron’s EmacsLive (which is my preferred packet of Emacs modes and extensions for writing Clojure). I’ve opened up the source-code of Liquid, which is a nice project to create a powerful text editor, heavily inspired by Vi and Emacs (though not aiming to be a clone of either) written entirely in pure Clojure.

I’ve got Clojure installed on the machine. Along with git, leiningen, Python3 and SWI-Prolog.

I’m able to compile and run Clojure (including this Liquid project). So that’s one sort of holiday project I might find myself getting up to. Playing with and seeing if I can figure out how to write new “modes” or extend Liquid to my own nefarious purposes.

Or, I can continue my playing with Prolog.

I also have a Bluetooth keyboard, which is nice and small and portable, and seems to be OK. (After a couple of hours playing). Obviously because it’s small so I don’t know if I can adapt to doing a lot with it. But … so … far, the bluetooth connection seems fairly reliable, and not dropping.

So, the whole package is remarkable small and light. I would have liked one of those keyboard covers, but no one seems to be selling them here in Brasilia (the only one I found only has an old microUSB connection, not the USB C that the tablet uses)

In terms of software, including powerful programming languages and a powerful editor (Emacs) … this looks viable. Of course tablets are powerful things these days. The whole UserLAnd Ubuntu with all the above software is taking up only a small chunk of the 64 Gb storage.

The big bottlenecks will be the keyboard and screen and working without a mouse.

CLOjure SHell

I’ve been playing with CloSH, the Clojure Shell.
And I have to say, I’m sold.
If you’d like to have the full power of Clojure. And to write shell scripts that look like this :

(doseq [f (expand "*jpg")
    (println f)
    (sh identify -format "%wx%h" (str f))
    (println))

rather than in hard to remember shell-script language.
Then this sorts you out.
Tab completion is a bit weirdly slow. And there are some glitches. But it basically works as you’d want / expect.
I’m now running it and, as of today, using it to do real things (eg. that batch process of analyzing jpegs, above).

What's up? August 2019 Edition

Gosh … we’re half way through the year already.

So what am I up to in programming?

Clojure

I still think it’s a wonderful language. Not doing as much as I’d like. But it’s still great.

One thing I did a few months ago was port the terminal version of Mind Traffic Control to it. Which you can get here : https://gitlab.com/interstar/MTC-CLJ

And I’m happy with it. Works fine. I’m using it every day.

That means that my experimenting with Racket is unfortunately at an end. I don’t currently see myself going back to using Racket for anything in the near future. Nice as it is, I think I’m committing to Clojure for my Lispish stuff at the moment.

C++

Huh? Yep. I’ve been doing audio programming recently. And, unfortunately, until https://github.com/soul-lang/SOUL goes mainstream, there doesn’t seem to be much alternative.

In particular I’m playing with plugin development for VCVRack, which is surprisingly civilized. And yeah, in that specific context, with their nice API and nanovg it’s all fairly OK.

Which makes me also want to try another go at VST programming with JUCE. (I’m doing a lot of music in FL Studio these days, too). And I’m doing something on Android with Oboe.

Oboe is OK, but Android is hell!

Flutter / Dart

Which brings me to Dart / Flutter which I started playing with today. And already, I’m sold. I never bothered to look at Dart before. It looked like the kind of “bit nicer than Java” language which is probably good to have, but has never got my pulse racing.

But Flutter … literally I’ve wanted to do stuff on Android for ages. And HATED every time I’ve tried to do much with it. The whole experience is ghastly.

But finally … Flutter actually looks like some sanity has been brought to Android. Yey! It’s all in one file. Yey! A language that allows complex data declaration and therefore can use this to define reactive UIs declaratively. Yey! Hot-reload (like Figwheel for ClojureScript).

Finally. Some sanity.

OK, this might be premature. I haven’t actually done anything more than the hello world tutorial. But it does look like it’s an acceptable Android programming language / framework.

Next challenge, can I call native C++ libraries like Oboe from it?

Ferret

I still think Ferret (a compile to C++ Clojure-like) looks promising. And I asked the inventor if I can make libraries for inclusion into C++ code with it.

You can!

Great. So once I’ve got a moment to play, I’ll see if I can write some pure data-model / code stuff in Ferret, compile it to C++ and try to use in a) my music context and b) natively on Android.

Python3

I’m in the process of moving my code-bases to Python3.

I think TSWiki code (and the libraries) are pretty much done. FSQuery theoretically but I found a bug. I think OWL is getting there but not released.

Basically I am on schedule that all my non legacy / abandoned Python is going to be Python3 by the end of this year.

Prolog

I’m still fascinated by the potential of Prolog or Prolog-like declarative / logic / inference-engine enabled languages.

And I’m actually playing a bit with SWI-Prolog at the moment. It is VERY GOOD.

Running a web-server and putting a web front-end to Prolog turns out to be fairly straightforward and very concise. Actually generating HTML from the DCG-based DSL is also surprisingly good. More or less equivalent to my own Python library for this

If I can just figure out how to do some of the things I’m now imagining Prolog is good for (see my last post) in Prolog, then I might well be doing some fun stuff here.

So yeah, that’s me, August 2019 : Clojure, Python3, Dart / Flutter, C++, Prolog

And I’m finally getting my development environment into Docker containers.