qzhuyan responds to me tweeting Emacs on PocketCHIP.

Which is, er, very true. And wonderful.

But this haunts me continuously, as I explore the Mind Traffic Geometry of tools that support me tracking tasks and outlining ideas.

Will I one day end up simply falling into Emacs Org Mode? Isn’t that basically everything I really want?

Am I wasting my time with quixotic effort of writing my own software for this stuff when I could be writing something newer and more important?

Another thing that’s pushing me to think about this : this week I’ve been playing with Faust. A wonderful language for writing signal processing networks (ie. synthesizers, audio FX etc.) that compiles to multiple back-ends … including PureData, Supercollider, VST plugins and stand-alone programs.

It’s basically where I imagined Gates of Dawn eventually going.

But rather than a Python library, its a very nice “little-language”, with great operators for describing composition of data-flow blocks. It’s well developed and supported. I’m trying it out for writing small synths / FX units I can run on small boards like CHIP and Raspberry Pi.

I can see myself doing a lot with this. But it’s basically going to kill Gates of Dawn. Maybe there’s room for a Python library for those who don’t want to learn Faust. But for me, Faust is looking extremely viable.

So … another wasted project?

Perhaps I need to look at this positively. I’m not old. But I’m not as young as I used to be. I don’t have so many projects left ahead that I can afford to squander them. Perhaps its time to pivot. Time for a cull. A “spring-clean”. To remove some more cruft projects that occupy too much of my mind, but are actually just weak “me-too” versions of existing things that I would use perfectly happily if I made the effort to learn them. Enough with the Not Invented Here syndrome.

I’m not saying that OWL or MTC are going anywhere yet. I use them, and they work for me. And they are DIFFERENT from OrgMode, or todo.txt or any similar thing out there. They are what I want.

But I need to embrace this change. There are so many exciting NEW opportunities, there’s no point getting hung up on the old stuff.

Dawn is over. For me it’s 2PM. And there’s plenty of work to be done.

Today I killed the old Mind Traffic Control on Google App Engine. And replaced it with a new, fairly basic, site. (Though one that’s quite pretty, in a Packt Publishing kind of way.)

This is the beginning of a whole new MTC ecosystem.

1) I’m no longer interested in hosting your todos on Google App Engine. The only functionality that’s left on the site is the “export”. You can get the tasks you put into the old MTC in one of .txt, .csv or .opml formats.

.txt means you can use either an ordinary text editor, todo.txt or the new command-line based mtc program on your own machine.

.csv means you can use a spreadsheet if you prefer

.opml can be read into an outliner such as OWL.

2) The site has a new mission. Right now, it’s simply pointing visitors to the two pieces of software that I use to manage my todos and information : MTC command-line (in Racket) and OWL (in its three different versions : desktop, Android and web-served).

These are currently all simply source-code, hosted on GitHub. (Both MTC-racket and OWL are free-software, under the GPL).

It’s a geek view. But going forward, I’m going to be preparing, packaging and documenting these for actual users.

3) Right now, I still don’t know the relationship between MTC and OWL. They’re different programs, with different “mind traffic geometry”, in different languages. But I use both and I’m constantly musing about how they can and should interact.

The only thing which is clear to me at the moment, is that they should be brought together under the common Mind Traffic Control “brand”. As a statement of intent.

The question of 2014 is getting resolved.

4) “What about Project ThoughtStorms?”, you ask.

Well, I tend to think of “Project ThoughtStorms” as the developers’ view on my knowledge management / personal productivity software. While “Mind Traffic Control” is the user perspective.

There’s more to it than that. Project ThoughtStorms covers my experiments and add-ons to the Smallest Federated Wiki and thinking about wiki in general. MTC heavily emphasizes the dynamic flow of tasks. But that’s the broadest overview.

In practice I’ll continue to refer to MTC-racket and OWL within the contexts of both Mind Traffic Control and Project ThoughtStorms.

5) I am VERY happy to be deleting code. And collapsing several different overlapping ideas and codebases into … er … fewer. It’s a therapeutic decluttering that’s clearing space in my mind, and helping me focus and drive the surviving projects forward faster. It’s good.

But there is just a slight twinge of sadness. As I realize that this is a mile-stone in letting go of Python, a language I had a long and passionate engagement with. Both the old MTC code-base AND the server-side PageStore of OWL are Python.

But since porting OWL to Electron I can see that its future is very much in the Javascript (node / CoffeeScript) camp. Meanwhile the new MTC codebase is in Racket. And my latest quick conversion script persuades me that Racket is a good language for other small-scale tools. I have about a dozen of them in Python. But I’m pretty sure that new development in these areas will almost certainly be Lisp. (Racket or Clojure).

(Actually there is a project that’s still in Python where you’ll see some further development soon … but I’ll leave that to another post.)

A burst of development energy in a number of directions recently.

And things are starting to self-organize towards the new ecosystem.

Today’s exciting news : OWL makes a very nice desktop app, thanks to Electron.

Here’s the github repo.

I’ll be doing more testing, compiling, packaging this shortly. So that even non-geeks can play with it. But it seems to work fine.

To be honest, the few times I’ve installed OWL on non-geek friends’ machines, the “run a server and look at OWL in the browser” part has confused / put people off. Now things are VERY much simpler.

This is also going to give me some momentum to add a couple of extra features / ideas I’ve been thinking about over the last couple of years.

In the short term … this desktop repo is likely to be where I develop the next round of functionality. Though I’ll be porting these new features back to the Android and web-based versions.

Main issues in my mind at the moment : I’ve dropped the Python based server for this. We’re now purely Coffee / Javascript. That feels cleaner and more convenient. Will I now try do the same for the web version. It makes sense. But straight node? Express? Meteor?

What’s simplest and leads to least repetition of code?

A few years ago I looked at some of the programs I was writing, both for public release and libraries / scripts for internal consumption, and realized that even I was confused.

I needed a new map to understand what I was doing. Hence I came up with Project ThoughtStorms. A quick and dirty overview of what I was up to in this space of wikis, personal information and knowledge management.

I think that helps. I know what I’m up to, with the legacy of my ThoughtStorms UseMod wiki and SdiDesk, my experiments with Smallest Federated Wiki, the new OWL stuff, etc. It’s still a mess, but it’s conceptually “encapsulated” in a single place.

Now I find myself with a similar situation with a number of different tries at some tools to make it easier for me to produce static web-sites. Originally there was GeekWeaver, a way for me to generate static sites quickly and easily from an OPML outline. That was a great idea … except … it got mixed up with my n00bie enthusiasm for Lisp and desire to create a powerful Lisp-like programming language.

GeekWeaver also foundered on two other issues : soon after I wrote it, there were no popular OPML outliner tools. Although Dave Winer has resolved that issue and I have, indeed, used Fargo to author GeekWeaver programs.

The other, more subtle and pernicious issue is that few people want to develop a site from scratch these days. Most people use huge frameworks with not just Javascript, but CSS, LESS etc. It really doesn’t make sense to try to use a tool which denies the file-system or the fact that you’ll want to be working with a large number of external files.

So the next time I needed to produce some quick static pages I went back to the drawing-board and came up with BootDown : a very simple Python script that lets you write text in Markdown and wraps it in a BootStrap / BootSwatch template. It’s the opposite of GeekWeaver, it isn’t trying to be a rich and clever language that lets you define your own templates, it’s just a quick way to take advantage of a lot of templates that already exist.

BootDown adds two things to conventional Markdown. A shorthand for declaring div tags. And a page separator. So you can still make a multi-page site inside a single file. And you can explicitly reference the div classes and ids you need.

More recently still, I’ve been getting into a real Lisp : Clojure. Today, that’s the language I really want to be working with. And I have a current project that again involves generating a multi-page, fairly static site. But the inputs are a little bit more complicated : some more structured data, not just text. And it makes sense to store it in an outline. So I’ve started writing Clojure code to read OPML and spit out a BootDown Markdown file, to wrap in HTML.

I can see this project starting to grow too.

And so here I am … three Python code-bases : GeekWeaver, BootDown and the OWL back-end server. Two Clojure projects : the beginnings of a library to work on OPML outlines (OPMLKit) and my unreleased code I’m writing for this current project. And some clunky pipelines to use them together. It’s starting to be another mess. And that’s why I need Project GeekWeaver : an umbrella to pull together these different strands together, at least conceptually.

There are some thoughts … would it make sense to pull at least some of the original GeekWeaver code INTO the OWL back-end server? So that “run transformations on this outline” is a standard OWL feature, and OWL becomes GeekWeaver’s built-in IDE? But what about the OWLdroid version that doesn’t have Python? Should I rewrite that code in CoffeeScript. (Or ClojureScript … a working but still fairly slow option at the moment.) BootDown “just works”. Am I better off scrapping the GeekWeaver idea altogether and just focusing on such practical solutions?

Some of this will become clearer as I work through this current site I’m making. meanwhile Project GeekWeaver is the category and (shortly) the site for the high-level overview.

Good news. The latest version of BTSync actually lets me specify which folder on my Android tablet I want to sync.

That means I can explicitly tell it to sync. the OWL directory that OWLdroid uses by default. (Somewhere between updates of Android and BTSync last year, this facility got lost.)

So it’s looking like BTSyncing between my tablet and laptop are working again. So OWLdroid is a viable piece of software again. (The syncing with the laptop makes a huge difference in day-to-day usefulness.)

Yes, I know this isn’t great. I need to figure out how to ensure that OWLdroid works. The longer term plan is to move to having remoteStorage.js as an option. But that’s a little way down my todo-queue.

So, Dave has a nice template for showing essay-length writing.

And naturally, the way things are evolving in 2015, it’s a “single-page web-app” which can pull in the actual content from another URL.

Of course, I’d like to write essays using an outliner. And I’m using OWL which is really just the Concord editor that saves files in OPML. But myword.io has a new custom JSON format for its input (that also seems to allow Markdown).

So here’s the question : what’s the story for converting an outline to an essay? Is myword going to evolve to import OPML directly? Is the myword JSON format going to become a standard? How should OPML turn into myword documents?

In one sense it’s easy. You can always just flatten OPML and dump it to a raw text file. Perhaps with some indentation based on outline depth. (Although that might fight the Markdown.)

Or are there plans to represent any structure in the essay (sections, section headings etc.) with structure in the OPML? Is there a project to work on this? Or thoughts on a standard that I could start to support?

Obviously I have a couple of my own solutions in this area : GeekWeaver and bootdown. But if there are any emerging standards / conventions planned, then I’m up for supporting them. Both GeekWeaver and BootDown are useful, but they’re both rather old-skool command-line Python scripts which don’t necessarily fit into the workflow of many people in 2015.

Update : Dave replied :

It is what it is.

I have no plans at this time to do anything with it.

I might swing back around to this, or not.

MyWord is MIT Licensed, so if you really want it, you can do it.

OK. So right now, I’ll probably just do a flat export of text written in OPML and a quick JSON wrapper. And then think further. As this ties into questions about GeekWeaver, the future of some of my Python code base (as opposed to moving more into the browser)

Dave also points to his code :

Here’s the code I added to Fargo to generate the myword.io JSON.

https://gist.github.com/scripting/2a612b0e2cbb7077482a

As you can see, there ain’t much there! 🙂

Still. Very useful.

Bloody hell! I just wrote a blog post and WordPress lost it.

Grrrr …

The short resume was :

– yes, I’m late (Feb) continuing my “Welcome to 2015” blog posts.

– last year I got the functional programming bug. Haskell is pretty damned good, but I’ve fallen for Clojure because a) dynamic types, b) Lisp syntax(lessness), c) Paredit.

– last year I did more Python, but not much OWL. This year, OWL is calling again. And as well as making me want to rethink Mind Traffic Control in light of it, is also seeming to demand some kind of pivot on GeekWeaver.

I’ll be coding and blogging more on this ongoing development in coming weeks / months.

I’ve also been playing a bit with remoteStorage, but nothing to show yet.

So, 2015, here already?

Time to take stock. I haven’t been blogging as much as I should, but I’ve been pretty busy, so here’s the first of a couple of posts that give a quick run-down of the last year and thoughts on the next.

I started 2014 with a number of questions that I thought would guide my development work / play over the year. And then promptly went off and did something completely different. But it wasn’t an entirely wasted exercise. So let’s review those questions :

1) Given how satisfied I was with OWL, what was the future of Mind Traffic Control?
2) Given how satisfied I was with OWL, what was the future of Project ThoughtStorms?
3) How do I program on my tablet?
4) Can I learn visual design?
5) How do I support Mozilla and the new (open) web-stack?

So what happened?

1) Due to my other interests (see the next post) Mind Traffic Control got a stay of execution in 2014. I didn’t kill it and didn’t try to update it. What I did find myself doing (fairly recently) was throwing some new tasks into it. One reason MTC was a bit stale for me was that I wasn’t actually using it. But half-way through the year, my tablet broke, and when I got a new one, the rather convenient btsync connection between it and my laptop was impossible to get running again. This seriously degraded the value of OWL / OWLdroid for me, as btsyncing was my only way of keeping them automatically synced. (We’ll return to this theme in a minute, but just note here that MTC picked up some of the slack.)

2) I barely glanced at this question again after blogging it. I’ve been updating Thoughtstorms wiki sporadically this year. And it’s been perfectly viable without an OWL connection. Consider this a fairly non-urgent question as OWL and SFW seem pretty orthogonal in practice.

3) Right now, the best programming experiences I’ve had on the tablet are with Quoda and Clojure REPL. Both are pretty good as far as they go … which isn’t really far enough. I’ll come back to Clojure in a minute. But I’ll note that Quoda is the best code editor I’ve found so far (mainly for the very nice extra keys it provides). And Clojure REPL is, indeed, a REPL for Clojure. Which is good for trying out ideas. But far from a full blown Clojure dev. environment.

4) I’ve certainly been playing with visual designs this year (in Patterning), but I haven’t really touched on what I was getting at in this question at all. Maybe 2015?

5) I didn’t do much about this question in 2014, either. But that’s about to change.

I’ve just discovered the Unhosted movement and remoteStorage. And I believe this is going to very fundamentally shape what I’m thinking about and doing in 2015. I’m increasingly concerned with the risks to privacy and the concentration of power that comes from storing all our data in the clouds of a few large corporations. And remoteStorage seems a good solution to decouple the storage provision from app. provision and put control back in the hands of the user. More importantly, it seems to have both the right attitude and some momentum behind it.

So, in 2015, unlike 2014, I am going to make new year resolutions / commitments. In particular, I’m publicly committing, right now, that I’ll be adding remoteStorage to OWL and OWLdroid. This is not only good for you in the sense that it respects your right to privacy and control over your data, but it’s good for me, in that it should be a workaround for my Android file-system / btsync problems.

In fact, this year I’ve been playing with the javascript Dropbox API. And if I hadn’t discovered remoteStorage I’d have added Dropbox as an alternative file-system option, but it seems remoteStorage should nicely abstract away from that, allowing users to choose between Dropbox, Google Docs and any other storage provider.

OK. That’s the first big announcement for the year. In the next post I’ll come back to what I actually DID get obsessed with this year, and what I think that will mean for 2015.

Just found that this bug in Android is causing problems for my OWL / OWLdroid syncing.

Issue 38282 – Android MTP support does not show recent files until the device is
rebooted
.

Basically, when OWLdroid creates a new page, it writes a new OPML file to the Android’s local storage. But because Android now has some fairly complicated abstraction layers between your program and storage, including caching of directory listings, it means that although the file is there, not everything can see it.

I’ve been noticing this problem, possibly for a while.
Initially, I thought there were still a couple of (fairly infrequent and hard to reproduce) issues with the btsyncing between OWLdroid and my machine.

Then, the btsync seemed to break altogether as it seems that it can no longer work on the same directory that OWL does. (It’s not Bittorrent Labs’ fault, it semes there’s been a change to Android that restricts where apps. can write)

So I planned to update OWLdroid to be more compatible.

In the meantime, as a work-around I was syncing the directories via USB cable and meld. I used meld rather than rsync because it sees the MTP protocol which Android uses to connect to a PC.

However, today I was confronted with a mystery that brought me up short : meld could see and work on all pages except a new page I’d created two days ago on the Android device, which was invisible to it.

Huh?

I could see the same file in Ghost Commander (my Android file-system explorer). But meld couldn’t see it. Further investigation showed that not even Nautilus (via. MTP) could see it. Or any subsequent pages I created.

After a bit of Googling, I discovered the bug linked above. Something that was been reported to Google two years ago and still doesn’t seem important enough to fix.

As far as I can tell, it’s a cache refresh when a file is created or rewritten. Obviously it OUGHT to be fairly simple to fix. Or if automatic refreshing is not the desired behaviour, at least to have an option in the Android file-system API that can force it.

Otherwise, Android become useless for any app. which is working on files. And while Google’s priorities may be elsewhere (the conspiracy theorist in me wonders if they’re trying to drive everyone towards their cloud storage), surely they can’t hope for Android to take off as an ecosystem which people can use to do real work (say, in the enterprise), if you can’t write apps. that reliably work on files (and have your changes seen by connected machines).

Anyway, it’s frustrating. And until I see some way to reliably solve this problem, it certainly knocks back the (already fairly leisurely, “when it’s ready”) release schedule for OWLdroid to public via the Play store. 🙁