Working with the file system is too verbose. Let’s make it more like JQuery!

Working with the file system in Python is too verbose.


Every time I want to do something with files I need to remember whether it’s the `os` or `sys` library that I need to import. I need to remember or look up half a dozen other functions to grab particular bits of metadata from those files.

I’ve written countless nested loops or recursive functions to walk the tree of files over the years.

And often I want to do something quick to a bunch of files, start writing shell-script. Then realize I hate and can’t remember shell-script and think it would be so much easier in Python. Then I open the editor to try to write my code in python, and I realize it’s too much trouble and go back to mashing around in bash again.

And then I remembered JQuery with its refreshingly easy abstractions for maneuvering in, and manipulating a tagged tree-shaped data-structure. And the file-system is just a tree, right? So why does it have to be hard? Why shouldn’t working with files just be like working with JQuery?

This is such an obvious idea, I’m sure someone must have done it.

But I couldn’t find it. So I had to write it myself a couple of months ago.

So, let me present … FSQuery . Now available on PyPI and GitHub.

Here’s what it looks like :

from fsquery import FSQuery

fsq = FSQuery(path).Match(".js$").NoFollow("vendor").FileOnly()  
for n in fsq :
    print n.abs

You create an FSQuery object with a path to the root of the files you’re interested in. You then load up extra filters / queries on the query by chaining them together.

Finally you can treat the whole thing as an iterable collection and loop through its results.

It returns object of class FSNode, which represent nodes in the file-system, either files or directories. The modifier FileOnly(), restricts the query to only return files. You only need to add this once to the query.

The NoFollow() method tells the query to avoid directories that match the name. But has no effect on file names. In the above example, “vendor.txt” would still be included in the results if that file is anywhere other than under the vendor directory. (This beats just piping find through grep in the terminal.) You can add as many NoFollow filters as you like to a query.

On the other hand, Match() is an inclusive filter. Only files whose names are explicitly matched end up in the results. However, this filter isn’t applied to directories. FQuery will still explore and return directories whether they match this or not. You will usually want to combine a Match with a FilesOnly to get the effect you want (eg. in this case, all the .js files anywhere except under the vendor directory)

We can even look inside files with the Contains filter, eg.

fsq = FSQuery(path).NoFollow("vendor").Ext("py").Contains("GNU Lesser General Public License").FileOnly()

Note that this is implemented purely in Python in a very non efficient way. (Ie. I just open up each file and run through it looking for the string.) It can be slow with large chunks of the file system.

Note also the Ext() filter for file-name extensions. This is easier than regexing the whole file-name if you’re just looking for files of type “py”. Be aware that if you try to have two Ext() filters on the same query, you will get no files returned. No file can have two different extensions at the same time.

More documentation, and more advanced tricks can be seen on the GitHub site. This is also the first library I’ve put on PyPI. So installing in your own project is as simple as

pip install fsquery

August is Patterning Month

August is Patterning month again.

I’m back to work on the Patterning library. And, in particular, getting it working properly in the ClojureScript, in-browser version. I’m going to be using devcards, figwheel, spec and other good tools in the Clojure community.

I’ll be revamping the site and new versions of the code.

Watch this space …

Project ThoughtStorms Update

Been working quite a lot on Project ThoughtStorms, the new Python / Bottle based wiki engine behind ThoughtStorms in the last few days.
I can’t believe that I’ve spent 15 years building a wiki without creating my own software.
That error is going to be rectified from now on.
Sure, most of what I’m doing is adding the kind of functionality that most wikis already have : RecentChanges, Search, Sister Sites etc. But it’s pretty quick to write. And as I bring these functions back, ThoughtStorms becomes more subtle and alive.
The truth is that ThoughtStorms’ content is woefully outdated. There are a tonne of broken links, and news stories that were current in 2005, but equivalent stories between about 2008 and 2016 are no-where.
It needs a jolly good clean up / clean out / refactoring. But I’m starting to have some ideas about how to write some more innovative custom code to help with that.
It’s very much work in progress. And there’s still plenty that’s half-baked and unfinished.
But I’m starting to feel that ThoughtStorms is coming back to life.

MTC on PocketCHIP

You all probably knew where this was going, right?

Mind Traffic Control (Racket version) running on PocketCHIP
Mind Traffic Control (Racket version) running on PocketCHIP.

Of course, it’s been my priority to run the new MTC on the PocketCHIP. And it runs fine, without any special conversion; just needed to figure out how to install a library it depended on without going through drracket.
Now I’m off for my celebratory bike ride. 🙂

Tweaking / Refactoring OWL / OWLdroid

A bit of tweaking and refactoring OWL and OWLdroid over the last couple of days. If you’re trying them out, thinking of doing any development on OWL, it’s worth updating.
What’s up :
– the main purpose of this work was to bring the OWL and OWLdroid versions of the code-base back together. They’d drifted apart in my first attempt to figure out how to get this stuff working as an Android app. Now all the directory structures are back in the same place. The variation between versions is largely pulled out into specific local_setup.js files and a certain amount of cruft, debugging messages etc. have been removed.
This is obviously going to make it easier to do common development moving forward.
– there’s a new “Quick” menu which has buttons for moving the node up, down, left, right, and deleting it. This is mainly useful in OWLdroid where the menu was the only way to move nodes and there was no way to remove them. Nodes still have to be selected into the white-on-black non edit mode before they can be deleted (a useful precaution). This is awkward on OWLdroid, because there’s still the problem that the font-awesome arrows that show open / closed state of the items aren’t appearing. But it is possible to select this mode and, therefore, now possible to delete nodes on Android. (which previously wasn’t the case.)
– In OWLdroid I’ve experimentally moved the nav-bar up to the main menu bar. I’m concerned that this will screw up the layout on small screens. (Please tell me if it does.) Though it’s more or less OK on my Nexus 7 and I’d assume any tablet or in landscape mode.
There is a BIG advantage of doing this. The nav-bar controls no longer scroll off the top of long pages. So you can easily go back / forward or create links however far down the page you are. It’s definitely more usable for me.
I haven’t made the change in normal OWL because I think the cost / benefit calculation re: scrolling / layout problems probably works out a little differently. But if I get requests, then I’ll do it.
As a side-effect, it’s also removed something that was a catastrophic trap for the unwary in OWLdroid. The default bootstrap layout has a “brand” at the top left, which is a link back to the main page. When this was left in the Android version, if you clicked on it, it would cause weird behaviour. (On my tablet it would try to open the file system) That was confusing and hard to get back from. The branding link is now gone so this is no longer a danger.
– I’m still chasing down the Safari bug. It’s basically something to do with the way I handle requests for non-existent pages. The code which picks that up and deals with it on Firefox throws an error in Safari. The bug is appearing in jquery code, but it’s obviously because a null value is getting in somewhere. Just have to figure out what I’m relying on and make it an explicit test.

Questions for 2014 : #2 Project ThoughtStorms

Second in a series of questions occupying my mind at the beginning of 2014. Which may (or may not) inform what I’ll be working on.
2) What About Project ThoughtStorms?
Unlike the first question in the series, this one is relatively light.
It’s obvious that OWL is the new SdiDesk. And where my private personal organizer / wiki-notebook etc. work should now be focussed. It’s equally obvious that it’s not, in itself, a publication medium. Smallest Federated Wiki is still looking good for that. SFW also has some great features : federation being one. The embedded media types being another. It’s actively improving. And looks like it’s become the “official” wiki in the node.js library. ThoughtStorms isn’t moving off SFW in the foreseeable future.
So Project ThoughtStorms is still, largely, about tools that support SFW. At the same time, this dichotomy between one tool for writing and one tool for presenting is a little bit … disconcerting. It’s natural in the outlining world, of course, a separation between outliner as authoring tool and blog / slide-show / book as rendering. But in the wiki world, where writing / editing / reading are all blurred together in a kind of closed loop, it feels wrong.
Nevertheless, there’s little real question there. In 2014 OWL is my writing tool. SFW is my public thinking space. The open question is about what bridges to build between the worlds.
Should OWL export complete SFW pages? (Easyish, I think). Should it import them? (To be consistent with my promise not to abandon SdiDesk users?) Should I try to get it to speak the federation protocol of SFW? Where should I be capturing quick notes and draft thoughts that aren’t (yet) ready for publication?
No great soul-searching here. But a bit of quizzical head-scratching required.
(Next question coming soon.)

Questions for 2014 : #1 Mind Traffic Control

I’m not going to be making predictions for 2014. Or promises that I may or may not keep. Or even resolutions. At least, not public ones : who knows what crazy trajectories my life will take this year?
But I do have a couple of questions. These are the things occupying my mind at the beginning of 2014. And which may plausibly inform what I do.
1) What’s the relation between OWL and Mind Traffic Control?
There’s little doubt in my mind now. OWL is great. It’s the future. At least, as far as my private notebook is concerned. And that means it’s now my de facto to-do list manager as well.
Which is kind of embarrassing for Mind Traffic Control.
MTC does have some virtues :

  • firstly, a kick-ass name that I love. “Mind Traffic Control” is the best name ever! For personal organizer stuff.
  • it’s solid. It has remembered everything I put into it from the moment it was written. It’s all still hanging around to haunt me. (Even if I kind of wish some of it wouldn’t)
  • the task delegation is pretty sweet. I don’t know anything else as simple as MTC for delegating tasks to other people and tracking what you delegated. To be honest, I don’t do a lot of delegating tasks to people. But if I did, I’d want to use MTC.
  • the deferral part works as advertised and it’s a nice feature for a task-tracking app.
  • MTC is slicker than you remember, if you haven’t looked at it for a while. In 2013 I did a bunch of Ajaxing it up so it’s now sending more little json fragments around and doing a lot less recreating the page.

But it also has big problems.

  • It’s not OWL. (See above. Doh!)
  • There is a major impedance mismatch between queues and the ORM-style storage system that Google App Engine uses. Moving items around on the queues seems to be really inefficient. To such an extent that whenever I do a burst of development / testing on the live server, Google kick me off for going over my quota of data queries. Querying also seems to take longer than it needs.
  • It’s on the GAE cloud, at a time when I and, I think, many others are becoming wary of storing their personal / private data on servers belonging to large corporations under the jurisdiction of the NSA. And at a time when I, myself, am seeking to become less entwined with Google and its technologies.
  • While the UI has improved, it’s still not as slick as it ought to be. It still needs a lot of work to look as good or work as well as modern web-apps.
  • It never really caught on. People try it, some have used it off and on over a couple of years, but I don’t think I have any long-term users except me.

So if I look deep into the abyss and am honest, MTC is a failed experiment. Whatever its theoretical virtues.
And in 2014 I think there’s a straight choice :
Either admit to myself that MTC is no longer a going concern. Perhaps kill it entirely. (Like I did with Platform Wars last year.)
Or, to use a fashionable term, pivot the site to become something else. Probably something with OWL at its core. I don’t quite know what that would mean. Maybe doing the hard work of remodelling the queue data-model (use localStorage? synchronized to large BLOBs on the server?) and then trying to put OWL / Concord on the front-end of that? Maybe just a one-time export of MTC data? (Remember, MTC has exported to OPML since its inception.) Maybe becoming a GAE hosted farm of OWL wikis? If so, what, if anything, of the original queue and delegation model could / should be preserved / recreated?
Today, I only have these questions rather than answers. But this year, I’ll be trying to resolve them one way or another.
(I’ll post the next of my 2014 questions tomorrow. Maybe.)

Project ThoughtStorms Wikish Plugin Update

There’s a quick update to the wikish plugin for Project ThoughtStorms (that’s the plugin type that offers UseMode / SdiDesk style wiki markup for the Smallest Federated Wiki). I’ve now added the double-comma defined tables.

oranges,, apples,, pears
43,, 55,, 22

Now becomes

oranges apples pears
43 55 22

Although not nearly as elegant. The simple CSS styling I was trying to use seems to clash with SFW’s style sheet in a way I haven’t debugged yet. So I’ve stuck with good old-fashioned table borders. (Ouch!)
Given the SFW’s narrow columns, I wasn’t originally going to support tables. But I have a couple of things for which it makes sense. So if you’re using Project ThoughtStorms, the repo is already updated. ThoughtStorms itself I haven’t updated to the most recent SFW codebase. That’s coming ASAP.