Been working quite a lot on Project ThoughtStorms, the new Python / Bottle based wiki engine behind ThoughtStorms in the last few days.

I can’t believe that I’ve spent 15 years building a wiki without creating my own software.

That error is going to be rectified from now on.

Sure, most of what I’m doing is adding the kind of functionality that most wikis already have : RecentChanges, Search, Sister Sites etc. But it’s pretty quick to write. And as I bring these functions back, ThoughtStorms becomes more subtle and alive.

The truth is that ThoughtStorms’ content is woefully outdated. There are a tonne of broken links, and news stories that were current in 2005, but equivalent stories between about 2008 and 2016 are no-where.

It needs a jolly good clean up / clean out / refactoring. But I’m starting to have some ideas about how to write some more innovative custom code to help with that.

It’s very much work in progress. And there’s still plenty that’s half-baked and unfinished.

But I’m starting to feel that ThoughtStorms is coming back to life.

Posted in Me.

A quick update (for me, as much as anyone else) of where I’m at and what I’m currently thinking about … March 2017:

ClojureScript and Reagent

I want to get up to speed on this as my standard way of writing browser based software. React seems to have conquered the world as browser based, reactive framework. And with React Native it’s also come to mobiles. It seems that the ClojureScript / React Native story is becoming compelling too.

So, yes, this is mainly how I want to do GUIs moving forward. I’m doing some experiments. Expect, eventually, Patterning and some of my other projects to move to this.

WebAudio

A friend of mine has sold me on the idea of going back to audio in the web-browser. Which is obviously a great way to make little musical apps that are easy to share with people. He’s not so keen on ClojureScript but does like Javascript in a functional style with Redux and Immutable.js so we’ll probably be using them on a collaboration. I’m considering porting versions of Gbloink! and Zewp! to it. Along with more of my current Processing based “ArtToys” Library.

Lisps on Android

It may be that ClojureScript + ReactNative is the right way to go. But I’m also looking at Clojure on Android. And Racket on Android.

Reading online it seems that Clojure on Android has been improving, but is maybe not as efficient as ClojureScript with ReactNative. And, also obliges you to deal more directly with the Android Java frameworks. Maybe something that’s useful for larger, more framework intense, applications.

I was originally tempted by LambdaNative, a Scheme for Android app. development. But it now looks like Racket’s Android story has improved considerably and is quite similar. Especially sharing the focus on using OpenGL for visuals. Racket on Android basically compiles to C and uses the NDK, which might be a faff to get working, but might also be more efficient. One to have in the arsenal.

Also, of course, Racket on Android would let me keep a common Racket code-base for both the command-line and an Android version of Mind Traffic Control.

Rosette

I’m kind of intrigued by Rosette and the idea of “solver aided” programming. And being a Racket DSL is a bonus. I want to get my head around this, this year.

Music Languages

Last year. I did a lot on Sonic-Pi. This year I thought I’d go back and take another look at Overtone. Which is, in many ways, the predecessor of Sonic-Pi done as a Clojure library.

It’s not quite as simple as Sonic Pi with its integrated development environment but Clojure is
more powerful than Ruby.

While I still think that Faust looks cool. I’ve also been tempted by the lower-level simplicity of CSOUND … wondering if it might make sense to write a CSOUND generator in Clojure. I’m slightly pulled backwards and forwards on this question. Plus, Overtone provides its own synth-design functions.

Project ThoughtStorms

So, I haven’t trumpeted it here. But last year I moved away from the Smallest Federated Wiki for ThoughtStorms. See ThoughtStorms:LeavingTheSFW for the full story.

Here I’ll just note … I’m very happy with the decision, and I have lots of ideas for how to develop my currently very minimal wiki engine to do more of the things I want to do. In many ways, this is like the MTC reboot of last year.

Small Boards

I’ve become a bit of a collector of small hobbyist / maker boards, from Arduino to last year’s CHIP. I’ve just got a couple of BBC Micro:Bits which are great fun. These are a really smart and original idea in what’s becoming a crowded and competitive market. The computing power is pretty minimal. But what gives them an edge is that they’re packed with sensors. And come with their own primitive display.

I’m looking forward to playing with and giving these a good run for their money.

Interestingly they run MicroPython. Something I hadn’t really looked at before.

Python

All of which means, despite last year’s thinking, that I’m actually renewing my relationship with Python. I really thought I was deprecating it in favour of Javascript. This year, that seems fairly unlikely. Python just is nicer than Javascript in so many ways. It’s the back-end of my new wiki, which I’m planning to extend. It’s the language of the Microbit and potentially useful on other boards. It’s getting ported to the GoLang infrastructure and perhaps has a great future in front of it there. Meanwhile, even a tiny serverside JS program with node.js seems to install a million dependencies.

So Python is definitely here to stay in my near future.

QaSaC

A rather crazy art-project I dabbled with last year spawned an interesting toy / esolang : QaSaC. This is little experimental “concatenative” language like Forth and Joy cross-pollinated with a visual data-flow language like Pure Data.

I think there’s some potential here. And I’ve really enjoyed the concatenative / stack-based thinking and elegance of this kind of programming. So I am working (albeit slowly and interspersed with everything else) on getting an actual viable version of this out soon.

OK. I think that’s a pretty comprehensive list of my obsessions for 2017. Let’s see how much progress I make with them all. This year, I promise, I’ll be trying to release, and blog, more often about them.

Posted in Me.

Clojure is a wonderful language. The more I use it, the more I like it.

But the default error reporting is catastrophically bad, forcing you to look through screensful of Java stack-trace for the needle of a bit of code that’s actually yours.

I’m sure there are better solutions, but today I just hacked a very quick and dirty python program to filter this result, highlighting the lines that I care about.


#!/usr/bin/python

import sys
p = sys.argv[1]

for line in sys.stdin :
    if p in line :
        print '33[92m' + line + '33[0m',
    else :
        print line,

I made this code in an executable file called “colourit” in my bin directory. Now I can do this :


lein test |& colourit patterning

to, for example, highlight lines containing the word “patterning”.

Posted in Uncategorised.

A few years ago I looked at some of the programs I was writing, both for public release and libraries / scripts for internal consumption, and realized that even I was confused.

I needed a new map to understand what I was doing. Hence I came up with Project ThoughtStorms. A quick and dirty overview of what I was up to in this space of wikis, personal information and knowledge management.

I think that helps. I know what I’m up to, with the legacy of my ThoughtStorms UseMod wiki and SdiDesk, my experiments with Smallest Federated Wiki, the new OWL stuff, etc. It’s still a mess, but it’s conceptually “encapsulated” in a single place.

Now I find myself with a similar situation with a number of different tries at some tools to make it easier for me to produce static web-sites. Originally there was GeekWeaver, a way for me to generate static sites quickly and easily from an OPML outline. That was a great idea … except … it got mixed up with my n00bie enthusiasm for Lisp and desire to create a powerful Lisp-like programming language.

GeekWeaver also foundered on two other issues : soon after I wrote it, there were no popular OPML outliner tools. Although Dave Winer has resolved that issue and I have, indeed, used Fargo to author GeekWeaver programs.

The other, more subtle and pernicious issue is that few people want to develop a site from scratch these days. Most people use huge frameworks with not just Javascript, but CSS, LESS etc. It really doesn’t make sense to try to use a tool which denies the file-system or the fact that you’ll want to be working with a large number of external files.

So the next time I needed to produce some quick static pages I went back to the drawing-board and came up with BootDown : a very simple Python script that lets you write text in Markdown and wraps it in a BootStrap / BootSwatch template. It’s the opposite of GeekWeaver, it isn’t trying to be a rich and clever language that lets you define your own templates, it’s just a quick way to take advantage of a lot of templates that already exist.

BootDown adds two things to conventional Markdown. A shorthand for declaring div tags. And a page separator. So you can still make a multi-page site inside a single file. And you can explicitly reference the div classes and ids you need.

More recently still, I’ve been getting into a real Lisp : Clojure. Today, that’s the language I really want to be working with. And I have a current project that again involves generating a multi-page, fairly static site. But the inputs are a little bit more complicated : some more structured data, not just text. And it makes sense to store it in an outline. So I’ve started writing Clojure code to read OPML and spit out a BootDown Markdown file, to wrap in HTML.

I can see this project starting to grow too.

And so here I am … three Python code-bases : GeekWeaver, BootDown and the OWL back-end server. Two Clojure projects : the beginnings of a library to work on OPML outlines (OPMLKit) and my unreleased code I’m writing for this current project. And some clunky pipelines to use them together. It’s starting to be another mess. And that’s why I need Project GeekWeaver : an umbrella to pull together these different strands together, at least conceptually.

There are some thoughts … would it make sense to pull at least some of the original GeekWeaver code INTO the OWL back-end server? So that “run transformations on this outline” is a standard OWL feature, and OWL becomes GeekWeaver’s built-in IDE? But what about the OWLdroid version that doesn’t have Python? Should I rewrite that code in CoffeeScript. (Or ClojureScript … a working but still fairly slow option at the moment.) BootDown “just works”. Am I better off scrapping the GeekWeaver idea altogether and just focusing on such practical solutions?

Some of this will become clearer as I work through this current site I’m making. meanwhile Project GeekWeaver is the category and (shortly) the site for the high-level overview.

Posted in Uncategorised.

Note to self :

sudo apt-get remove python2.7

Has a pretty dramatic effect on a Ubuntu box. There are a LOT of packages these days that are dependent on Python somewhere inside them (including a lot of Unity) 🙂

Posted in Uncategorised.

OK. Consider me won over.

Lisp is great to work with. The things that grabbed me about FP in Erlang and Haskell (pattern-matching arguments, partial application, lazy evaluation) are all here.

My code is as concise as Erlang and damned nearly as concise as Haskell (I think the line count is similar though the number of characters per line is about twice as high)

Although I respect Haskell’s type system, Clojure’s dynamic typing holds me up less. I *am*, admittedly, hitting more problems that a type system would have picked up. (Far more than I typically hit in Python) but I think as I get more used to the language this will go down.

I’m not being particularly demanding in terms of libraries. (Quil is almost all the libraries I need at the moment.) But I’m finding that there are all these handy things like spit which make me smile. (Remember what this was like in Java?)

Things that would still be a bit fiddly in Python / CoffeeScript continue to turn out to be easier than I imagined they would be when I implement them in Lisp. That’s partly because I default to approaching them as “how do I write a small lambda that can then map across this list” and mostly, by the time I wrote that function, all I have to do is … er … map it across the list.

Lisp is extra-ordinarily compressible. I keep finding ways to refactor and fold up things I did earlier to make them shorter and shorter. And the more functionality I add to my program the more compressed it seems to get.

I am grinning to myself at the thought that my code might be compilable to both Android and into Javascript. I haven’t tried either yet, but if I manage it then Clojure is now my new favourite language.

CoffeeScript isn’t at all bad, but if ClojureScript works out, then I’ll probably start to move aggressively to use it as my default in-browser language. (And yes, I guess that may mean rewriting OWL in it. OWL is still short enough that I think I could sprint it in a couple of days.)

Similarly, if I can compile Clojure libraries that can be called from Java projects then you’ll start to see me more productive on Android. (Beyond OWLdroid I have a couple of other bits and pieces of apps. written, but not taken them to completion. If Clojure slots painlessly into the workflow, it will be a lot more tempting to dive back in.)

So … yeah … Clojure rocks!

Posted in Uncategorised.

I’ve gone back to doing some work on Gates of Dawn, my terse Python library for creating PureData patches.

The idea here is I want to make synthesizers in PD but I don’t particularly like drawing boxes and connectors using the clunky UI.

Here’s a brief overview from the Readme file. Showing some simple code examples and introducing the key ideas.

Idea #1 : Gates of Dawn uses function composition as the way to wire together the units in Pure Data.

In PureData traditionally, you’d do something like wire the number coming out of a slider control to the input of an oscillator and then take that to the dac audio output. Here’s how to express that using Gates of Dawn.

    dac_ ( sin_ ( slider("pitch",0,1000) ) )

To create the slider you call the slider function (giving a label and a range). Then you pass the result of calling that as an argument to the sin_ function (which creates an osc~ object in PD). The output of that function is passed as an argument to the dac~. (Note we are now trying to use the convention that functions that represent signal objects (ie. those that have a ~ in their name in PD) will have a _ suffix. This is not ideal but it’s the best we can do in Python.)

In Gates of Dawn programs are structured in terms of a bunch of functions which represent either individual PD objects or sub-assemblies of PD objects. You pass as arguments to the functions those things that are upstream and coming in the inlet, and the return value of the function is suitable to be passed to downstream objects.

Things obviously get a bit more complicated than that, but before we go there, let’s just grasp the basic outline of a Gates of Dawn program.

Idea #2 : Files are managed inside with patch() context blocks.

Here’s the complete program.

from god import *

with patch("hello.pd") as f :
    dac_ ( sin_ ( slider("pitch",0,1000) ) )

We use Python’s "with" construction in conjunction with our patch() function to wrap the objects that are going into a file. This is a complete program which defines the simple slider -> oscillator -> dac patch and saves it into a file called "hello.pd"

Try the example above by putting it into a file called hello.py in the examples directory and running it with Python. Then try opening the hello.pd file in PureData.

If you want to create multiple pd files from the same Python program you just have to wrap each in a separate with patch() block.

Idea #3 : Variables are reusable points in the object graph.

This should be no surprise, given the way function composition works. But we can rewrite that simple patch definition like this :

s = sin_ ( slider("pitch",0,1000) )
dac_(s)

The variable s here is storing the output of the sin_ function (ie. the outlet from the oscillator). The next line injects it into the dac.

This isn’t useful here, but we’ll need it when we want to send the same signal into two different objects later on.

NB: Don’t try to reuse variables between one patch block and another. There’s a weird implementation behind the scenes and it won’t preserve connections between files. (Not that you’d expect it to.)

Idea #4 : Each call of a Gates of Dawn function makes a new object.

with patch("hello.pd") as f :
    s = sin_ ( slider("pitch",0,1000) )
    s2 = sin_ ( slider("pitch2",0,1000) )
    dac_(s,s2)

Here, because we call the functions twice, we create two different oscillator objects, each with its own slider control object.

Note that dac_ is an unusual function for Gates of Dawn, in that it takes any number of signal arguments and combines them.

Idea #5 : You can use Python functions to make cheap reusable sub-assemblies.

Here’s where Gates of Dawn gets interesting. We can use Python functions for small bits of re-usable code.

For example here is simple function that takes a signal input and puts it through a volume control with its own slider.

def vol(sig,id=1) :
    return mult_(sig,num(slider("vol_%s" % id,0,1)))

Its first argument is any signal. The second is optional and used to make a unique label for the control.

We can combine it with our current two oscillator example like this :

def vol(sig,id=1) :
    return mult_(sig,num(slider("vol_%s" % id,0,1)))

with patch("hello.pd") as f :
    s = vol (sin_ ( slider("pitch",0,1000) ), "1")
    s2 = vol (sin_ ( slider("pitch2",0,1000) ), "2")
    dac_(s,s2)

Notice that we’ve defined the vol function once, but we’ve called it twice, once for each of our oscillators. So we get two copies of this equipment in our patch.

Of course, we can use Python to clean up and eliminate the redundancy here.

def vol(sig,id=1) :
    return mult_(sig,num(slider("vol_%s" % id,0,1)))

def vol_osc(id) :
    return vol( sin_( slider("pitch_%s"%id,0,1000) ), id)

with patch("hello.pd") as f :
   dac_(vol_osc("1"),vol_osc("2"))

Idea #6 : UI is automatically layed-out (but it’s work in progress).

You’ll notice, when looking at the resulting pd files, that they’re ugly but usable. Gates of Dawn basically thinks that there are two kinds of PD objects. Those you want to interact with and those you don’t. All the objects you don’t want to see are piled up in a column on the left. All the controls you need to interact with are layed-out automatically in the rest of the screen so you can use them.

This is still very much work in progress. The ideal for Gates of Dawn is that you should be able to generate everything you want, just from the Python script, without having to tweak the PD files by hand later. But we’re some way from that at this point. At the moment, if you need to make a simple and usable PD patch rapidly, Gates of Dawn will do it. But it’s not going to give you a UI you’ll want to use long-term.

Idea #7 : You still want to use PD’s Abstractions

Although Python provides convenient local reuse, you’ll still want to use PD’s own Abstraction mechanism in the large. Here’s an example of using it to make four of our simple oscillators defined previously :

with patch("hello.pd") as f :
    outlet_ ( vol_osc("$0") )
    guiCanvas()

with patch("hello_main.pd") as f :
    dac_(
        abstraction("hello",800,50),
        abstraction("hello",800,50),
        abstraction("hello",800,50),
        abstraction("hello",800,50)    
    )

In this example we have two "with patch()" blocks. The first defines a "hello.pd" file containing a single vol_osc() call. (This is the same vol_osc function we defined earlier.)

Note some of the important extras here : * outlet_() is the function to create a PD outlet object. When the hello.pd patch is imported into our main patch, this is how it will connect to everything else. * You must add a call to guiCanvas() inside any file you intend to use as a PD Abstraction. It sets up the graph-on-parent property in the patch so that the UI controls will appear in any container that uses it. * Note that we pass $0 to the vol_osc function. $0 is a special variable that is expanded by PD into a different random number for every instance of a patch that’s being included inside a container. PD doesn’t have namespaces so any name you use in an Abstraction is repeated in every copy. This can be problematic. For example a delay may use a named buffer as storage. If you import the same delay Abstraction twice, both instances of the delay will end up trying to use the same buffer, effectively merging the two delayed streams into one. Adding the $0 to the beginning of the name of the buffer will get around this problem as each instance of the delay will get a unique name. In our simple example we don’t need to use $0. But I’ve added it as the label for our vol_osc to make the principle clear.

The second "with patch()" block defines a containing patch called hello_main.pd. It simply imports the hello.pd Abstraction 4 times and passes the four outputs into the dac_.

Note that right now, layout for abstractions is still flaky. So you’ll see that the four Abstractions are overlapping. You’ll want to go into edit mode and separate them before you try running this example. Once you do that, though, things should work as expected.

Posted in Uncategorised.