Gates of Dawn Update

I’ve gone back to doing some work on Gates of Dawn, my terse Python library for creating PureData patches.

The idea here is I want to make synthesizers in PD but I don’t particularly like drawing boxes and connectors using the clunky UI.

Here’s a brief overview from the Readme file. Showing some simple code examples and introducing the key ideas.

Idea #1 : Gates of Dawn uses function composition as the way to wire together the units in Pure Data.

In PureData traditionally, you’d do something like wire the number coming out of a slider control to the input of an oscillator and then take that to the dac audio output. Here’s how to express that using Gates of Dawn.

    dac_ ( sin_ ( slider("pitch",0,1000) ) )

To create the slider you call the slider function (giving a label and a range). Then you pass the result of calling that as an argument to the sin_ function (which creates an osc~ object in PD). The output of that function is passed as an argument to the dac~. (Note we are now trying to use the convention that functions that represent signal objects (ie. those that have a ~ in their name in PD) will have a _ suffix. This is not ideal but it’s the best we can do in Python.)

In Gates of Dawn programs are structured in terms of a bunch of functions which represent either individual PD objects or sub-assemblies of PD objects. You pass as arguments to the functions those things that are upstream and coming in the inlet, and the return value of the function is suitable to be passed to downstream objects.

Things obviously get a bit more complicated than that, but before we go there, let’s just grasp the basic outline of a Gates of Dawn program.

Idea #2 : Files are managed inside with patch() context blocks.

Here’s the complete program.

from god import *
with patch("hello.pd") as f :
    dac_ ( sin_ ( slider("pitch",0,1000) ) )

We use Python’s "with" construction in conjunction with our patch() function to wrap the objects that are going into a file. This is a complete program which defines the simple slider -> oscillator -> dac patch and saves it into a file called "hello.pd"

Try the example above by putting it into a file called in the examples directory and running it with Python. Then try opening the hello.pd file in PureData.

If you want to create multiple pd files from the same Python program you just have to wrap each in a separate with patch() block.

Idea #3 : Variables are reusable points in the object graph.

This should be no surprise, given the way function composition works. But we can rewrite that simple patch definition like this :

s = sin_ ( slider("pitch",0,1000) )

The variable s here is storing the output of the sin_ function (ie. the outlet from the oscillator). The next line injects it into the dac.

This isn’t useful here, but we’ll need it when we want to send the same signal into two different objects later on.

NB: Don’t try to reuse variables between one patch block and another. There’s a weird implementation behind the scenes and it won’t preserve connections between files. (Not that you’d expect it to.)

Idea #4 : Each call of a Gates of Dawn function makes a new object.

with patch("hello.pd") as f :
    s = sin_ ( slider("pitch",0,1000) )
    s2 = sin_ ( slider("pitch2",0,1000) )

Here, because we call the functions twice, we create two different oscillator objects, each with its own slider control object.

Note that dac_ is an unusual function for Gates of Dawn, in that it takes any number of signal arguments and combines them.

Idea #5 : You can use Python functions to make cheap reusable sub-assemblies.

Here’s where Gates of Dawn gets interesting. We can use Python functions for small bits of re-usable code.

For example here is simple function that takes a signal input and puts it through a volume control with its own slider.

def vol(sig,id=1) :
    return mult_(sig,num(slider("vol_%s" % id,0,1)))

Its first argument is any signal. The second is optional and used to make a unique label for the control.

We can combine it with our current two oscillator example like this :

def vol(sig,id=1) :
    return mult_(sig,num(slider("vol_%s" % id,0,1)))
with patch("hello.pd") as f :
    s = vol (sin_ ( slider("pitch",0,1000) ), "1")
    s2 = vol (sin_ ( slider("pitch2",0,1000) ), "2")

Notice that we’ve defined the vol function once, but we’ve called it twice, once for each of our oscillators. So we get two copies of this equipment in our patch.

Of course, we can use Python to clean up and eliminate the redundancy here.

def vol(sig,id=1) :
    return mult_(sig,num(slider("vol_%s" % id,0,1)))
def vol_osc(id) :
    return vol( sin_( slider("pitch_%s"%id,0,1000) ), id)
with patch("hello.pd") as f :

Idea #6 : UI is automatically layed-out (but it’s work in progress).

You’ll notice, when looking at the resulting pd files, that they’re ugly but usable. Gates of Dawn basically thinks that there are two kinds of PD objects. Those you want to interact with and those you don’t. All the objects you don’t want to see are piled up in a column on the left. All the controls you need to interact with are layed-out automatically in the rest of the screen so you can use them.

This is still very much work in progress. The ideal for Gates of Dawn is that you should be able to generate everything you want, just from the Python script, without having to tweak the PD files by hand later. But we’re some way from that at this point. At the moment, if you need to make a simple and usable PD patch rapidly, Gates of Dawn will do it. But it’s not going to give you a UI you’ll want to use long-term.

Idea #7 : You still want to use PD’s Abstractions

Although Python provides convenient local reuse, you’ll still want to use PD’s own Abstraction mechanism in the large. Here’s an example of using it to make four of our simple oscillators defined previously :

with patch("hello.pd") as f :
    outlet_ ( vol_osc("$0") )
with patch("hello_main.pd") as f :

In this example we have two "with patch()" blocks. The first defines a "hello.pd" file containing a single vol_osc() call. (This is the same vol_osc function we defined earlier.)

Note some of the important extras here : * outlet_() is the function to create a PD outlet object. When the hello.pd patch is imported into our main patch, this is how it will connect to everything else. * You must add a call to guiCanvas() inside any file you intend to use as a PD Abstraction. It sets up the graph-on-parent property in the patch so that the UI controls will appear in any container that uses it. * Note that we pass $0 to the vol_osc function. $0 is a special variable that is expanded by PD into a different random number for every instance of a patch that’s being included inside a container. PD doesn’t have namespaces so any name you use in an Abstraction is repeated in every copy. This can be problematic. For example a delay may use a named buffer as storage. If you import the same delay Abstraction twice, both instances of the delay will end up trying to use the same buffer, effectively merging the two delayed streams into one. Adding the $0 to the beginning of the name of the buffer will get around this problem as each instance of the delay will get a unique name. In our simple example we don’t need to use $0. But I’ve added it as the label for our vol_osc to make the principle clear.

The second "with patch()" block defines a containing patch called hello_main.pd. It simply imports the hello.pd Abstraction 4 times and passes the four outputs into the dac_.

Note that right now, layout for abstractions is still flaky. So you’ll see that the four Abstractions are overlapping. You’ll want to go into edit mode and separate them before you try running this example. Once you do that, though, things should work as expected.

Quora Scraper

I love Quora. It’s a great site and community. But I started getting a bit concerned how much writing I was doing there which was (potentially) disappearing inside their garden and not part of the body of thinking I’m building up on ThoughtStorms (or even my blogs).

Fortunately, I discovered Quora has an RSS feed of my answers, so I can save them to my local machine. (At some point I’ll think about how to integrate them into ThoughtStorms; should I just make a page for each one?)

Anyway here’s the script (powered by a lot of Python batteries.)

And this turns the files back into a handy HTML page.

Java Hater

Someone on Quora asked me to answer a question on my personal history of using Java. It became a kind of autobiographical confession. 

I’ve never had a good relationship with Java.

My first OO experience was with Smalltalk. And that spoiled me for the whole C++ / Java family of strongly typed, compiled OO languages. 

Because I’d learned Smalltalk and this new fangled OO thing when it was still relatively new (in the sense of the late 80s!) I thought I had it sussed. But actually I had very little clue. I enthusiastically grabbed the first C++ compiler I could get my hands on and proceeded to spend 10 years writing dreadful programs in C++ and then Java. I had assumed that the OOness of both these languages made them as flexible as I remembered Smalltalk to be. I thought that OO was the reason for Smalltalk’s elegance and that C++ and Java automatically had the same magic. 

Instead I created bloated frameworks of dozens of classes (down to ones handling tiny data fragments that would have been much better as structs or arrays). I wrote hugely brittle inheritance hierarchies. And then would spend 3 months having to rewrite half my classes, just to be able to pass another argument through a chain of proxies, or because somewhere in the depths of objects nested inside objects inside objects I found I needed a new argument to a constructor. The problem was, I was programming for scientific research and in industry but I hadn’t really been taught how to do this stuff in C++ or Java. I had no knowledge of the emerging Pattern movement. Terms like “dependency injection” probably hadn’t even been invented. 

I was very frustrated. And the funny thing I started to notice was that when I had to write in other languages : Perl, Javascript, Visual Basic (Classic), even C, I made progress much faster. Without trying to model everything in class hierarchies I found I just got on and got the job done. Everything flowed much faster and more smoothly.

Perl’s objects looked like the ugliest kludge, and yet I used them happily on occasion. In small simulations C structs did most of what I wanted objects to do for me (and I did finally get my head around malloc, though I never really wrote big C programs). And I had no idea what the hell was going on with Javascript arrays, but I wrote some interesting, very dynamic, cross browser games in js (this is 1999) using a bunch of ideas I’d seen in Smalltalk years before (MVC, a scheduler, observer patterns etc.) and it just came out beautifully. 

It wasn’t until the 2000s that I started to find and read a lot of discussions online about programming languages, their features, strength and weaknesses. And so I began my real education as a programmer. Before this, a lot of the concepts like static and dynamic typing were vague to me. I mean, I knew that some languages you had to declare variables with a type and in some you didn’t. But it never really occurred to me that this actually made a big difference to what it was like to USE a language. I just thought that it was a quirk of dialect and that good programmers took these things in their stride. I assumed that OO was a kind of step-change up from mere procedural languages, but the important point was the ability to define classes and make multiple instances of them. Polymorphism was a very hazy term. I had no real intuitions about how it related to types or how to use it to keep a design flexible.

Then, in 2002 I had a play with Python. And that turned my world upside-down.
For the first time, I fell in love with a programming language. (Or maybe the first time since Smalltalk, which was more of a crush).

Python made everything explicit. Suddenly it was clear what things like static vs. dynamic typing meant. That they were deep, crucial differences. With consequences. That the paraphernalia of OO were less important than all the other stuff. That the fussy bureaucracy of Java, the one class per file, the qualified names, the boilerplate, was not an inevitable price you had to pay to write serious code, but a horribly unnecessary burden.
Most of all, Python revealed to me the contingency of Java. In the small startup where I’d been working, I had argued vehemently against rewriting our working TCL code-base in Java just because Java was OO and TCL wasn’t. I thought this was a waste of our time and unnecessary extra work. I’d lost the argument, the rewrite had taken place, and I hated now having to do web-stuff with Java. Nevertheless, I still accepted the principle that Java was the official, “grown up” way to do this stuff. Of course you needed proper OO architecture to scale to larger services, to “the enterprise”. Ultimately the flexibility and convenience of mere “scripting” languages would have to be sacrificed in favour of discipline. (I just didn’t think we or our clients needed that kind of scaling yet.) 

What Python showed me was we weren’t obliged to choose. That you could have “proper” OO, elegant, easy to read code, classes, namespaces, etc. which let you manage larger frameworks in a disciplined manner and yet have it in a language that was light-weight enough that you could write a three line program if that’s what you needed. Where you didn’t need an explicit compile phase. Or static typing. Or verbosity. Or qualified names. Or checked exceptions. What I realised was that Java was not the inevitable way to do things, but full of design decisions that were about disciplining rather than empowering the programmer. 

And I couldn’t stomach it further. Within a few months of discovering Python I had quit my job. Every time I opened my machine and tried to look at a page of Java I felt literally nauseous. I couldn’t stand the difference between the power and excitement I felt writing my personal Python projects, and the frustration and stupidity I felt trying to make progress in Java. My tolerance for all Java’s irritations fell to zero. Failing to concentrate I would make hundreds of stupid errors : incompatible types, missing declarations or imports, forgetting the right arguments to send to library methods. Every time I had to recompile I would get bored and start surfing the web. My ability to move forward ground to a halt.
I was so fucking happy the day I finally stopped being a Java programmer.

Postscript : 

1) Something I realized a while after my bad experience was how important the tools are. My period in Java hell was trying to write with Emacs on a small-screen laptop without any special Java tools (except basic Java syntax colouring). I realize this is far from the ideal condition to write Java and that those who are used to Eclipse or IntelliJ have a totally different experience and understanding of the language. 

2) A few years later, I taught the OO course in the local university computer science department. All in Java. By that time, I’d read a couple of Pattern books. Read Kent Beck’s eXtreme Programming. Picked up some UML. And I had a much better idea what Polymorphism really means, how to use Interfaces to keep designs flexible, and why composition is better than inheritance. I tried to get the students to do a fair amount of thinking about and practising refactoring code, doing test driven development etc. It all seemed quite civilized, but I’m still happy I’m not writing Java every day. 

3) A couple of years ago I did do quite a lot of Processing. I was very impressed how the people behind it managed to take a lot of the pain of Java away from novice programmers. I wonder how far their approach could be taken for other domains.

Personal Question

Question : Hey Phil, do you actually do any programming these days?

Answer : Yes. Quite a lot at the moment. Though it’s a bit all over the shop.

I’m dipping a toe into Android programming. (And, hmmm … Java …. I thought I’d got over my Java hangups by doing a lot of Processing, but it turns out that Processing just hides the crap and Android doesn’t. Why hasn’t Google picked up on Processing to turn it into a first-class Android art / game app. development environment?)

I’m mainly writing CoffeeScript. Some stuff related to my ongoing 3D modelling / desktop manufacturing projects. (Did I forget to mention those? I’m sure there’s a half-written blogpost somewhere.) Some work towards an SdiDesk-derived network diagramming plugin for Smallest Federated Wiki (held up by silly problems). Some other bits and pieces. I’ve recently been playing with Jison, which rocks. And I’m about to investigate angular.js which looks pretty good.

There’s a project for small stand-alone web-servers that I’ll talk about more if / when it takes off.

I’ve been trying to compile example VST instruments  (C++) for some of my work with the Brasilia Laptop Orchestra, but it’s driving me crazy. (I may go back to Pure Data which can be embedded in a VST.)

A bit of PHP, just simple small web-services.

I’m going to be teaching an Arduino course soon. So I’ll be writing a bit of C and I want to try Occam-.

I’m still writing Python too. Mainly for short file transformation scripts or to prototype algorithms that later get translated into CoffeeScript.

Some of this stuff is headed for GitHub soon.

Command 'n' Cursor

I’ve been travelling with my trusty (but ageing) eeepc netbook this last week. There’s much to love about it but it’s starting to feel slow in comparison with my other machine.

Increasingly when I use the netbook I try to get away with doing things in a ctrl-alt-f1 shell without logging in to the GUI at all. I’m starting to wish more software could be used in this environment so I began to look at Curses, the standard library for text-window UIs. There’s a convenient Python wrapper of course. And there’s another nice library in Python : Cmd, for creating a command-line driven apps. That is, not programs that literally run as small tools on the shell with command-line arguments, but programs which have their own internal “repl” style loop which you drive by typing in commands. Cmd handily hides the details from you, letting you declare a subclass of the Cmd class which simply defines handlers for specific commands. It’s not a million miles away from something I ended up writing to handle the commands in SdiDesk.

For some of my projects it would be useful to combine the two modes : to have Cmd style input driving a 2D textual output using Curses. Unfortunately Cmd and Curses don’t obviously play well together.  Both of them want to take over the input, with Curses thinking in terms of keystrokes while Cmd still expects full lines.

Nevertheless, after a bit of exploration, and learning about Curses’s textpads and Cmd’s supplementary methods, I’m starting to get the two to co-operate. As this gist shows :

It doesn’t do anything yet. Just handles a “greet NAME” command that prints “hello NAME”. And a “quit” command that exits the program. But it has combined Cmd inputs with Curses output.