Tag: python

Microsoft adds Python to Excel

Several years late, it seems M$ is considering making Python native to Excel.

I wrote a … you guessed it … Quora answer as to why this makes sense for Microsoft.

—-

Python has become massively popular with the data and machine learning communities in the last few years.

Tools like JuPyter are increasingly popular and serious interfaces for data-modellers who previously would have used Excel.

It absolutely makes sense for Microsoft try to embed Excel and itself into that emerging Python data ecosystem by making Python a first-class citizen (ie. default, guaranteed to be there) of Excel.

Not just a third-party add on for those who know about it and can make the effort to install it.

Not only does Python need to be standard within Excel, but access to pip and all the Python libraries needs to be there too. So that Excel becomes the equivalent of Anaconda

That’s the way that M$ can keep Excel relevant in the new data age.

This is not only a good idea for Microsoft. It’s the difference between Excel remaining a major player in data modelling and analysis tools, vs. declining into obscurity.

Working with the file system is too verbose. Let’s make it more like JQuery!

Working with the file system in Python is too verbose.

Seriously.

Every time I want to do something with files I need to remember whether it’s the `os` or `sys` library that I need to import. I need to remember or look up half a dozen other functions to grab particular bits of metadata from those files.

I’ve written countless nested loops or recursive functions to walk the tree of files over the years.

And often I want to do something quick to a bunch of files, start writing shell-script. Then realize I hate and can’t remember shell-script and think it would be so much easier in Python. Then I open the editor to try to write my code in python, and I realize it’s too much trouble and go back to mashing around in bash again.

And then I remembered JQuery with its refreshingly easy abstractions for maneuvering in, and manipulating a tagged tree-shaped data-structure. And the file-system is just a tree, right? So why does it have to be hard? Why shouldn’t working with files just be like working with JQuery?

This is such an obvious idea, I’m sure someone must have done it.

But I couldn’t find it. So I had to write it myself a couple of months ago.

So, let me present … FSQuery . Now available on PyPI and GitHub.

Here’s what it looks like :

from fsquery import FSQuery

fsq = FSQuery(path).Match(".js$").NoFollow("vendor").FileOnly()  
for n in fsq :
    print n.abs

You create an FSQuery object with a path to the root of the files you’re interested in. You then load up extra filters / queries on the query by chaining them together.

Finally you can treat the whole thing as an iterable collection and loop through its results.

It returns object of class FSNode, which represent nodes in the file-system, either files or directories. The modifier FileOnly(), restricts the query to only return files. You only need to add this once to the query.

The NoFollow() method tells the query to avoid directories that match the name. But has no effect on file names. In the above example, “vendor.txt” would still be included in the results if that file is anywhere other than under the vendor directory. (This beats just piping find through grep in the terminal.) You can add as many NoFollow filters as you like to a query.

On the other hand, Match() is an inclusive filter. Only files whose names are explicitly matched end up in the results. However, this filter isn’t applied to directories. FQuery will still explore and return directories whether they match this or not. You will usually want to combine a Match with a FilesOnly to get the effect you want (eg. in this case, all the .js files anywhere except under the vendor directory)

We can even look inside files with the Contains filter, eg.

fsq = FSQuery(path).NoFollow("vendor").Ext("py").Contains("GNU Lesser General Public License").FileOnly()

Note that this is implemented purely in Python in a very non efficient way. (Ie. I just open up each file and run through it looking for the string.) It can be slow with large chunks of the file system.

Note also the Ext() filter for file-name extensions. This is easier than regexing the whole file-name if you’re just looking for files of type “py”. Be aware that if you try to have two Ext() filters on the same query, you will get no files returned. No file can have two different extensions at the same time.

More documentation, and more advanced tricks can be seen on the GitHub site. This is also the first library I’ve put on PyPI. So installing in your own project is as simple as

pip install fsquery

Languages for 2018

Bit early. But I answered a question on Quora about languages to learn for 2018.
Here’s what’s interesting me for 2018 :
I want to continue getting more experienced and better with Clojure. No language is perfect, but for me Clojure is the best language I’ve ever used. And I want to use it for more projects and in more different situations. I want Clojure to be my default / “workhorse” language for server-side, browser-based UI, Android apps. etc. Clojure is not just a great language but a practical language. And I’m expecting there to be more jobs / contracts available with it, going forward.
I’m intrigued by Rust. I haven’t even installed it yet. But I want to try it as a low-level C alternative. I have an idea it might be suitable for.
I admit that Richard Kenneth Eng and Peter Fisk are getting to me. I’d quite like to go back and have another look at / play with Smalltalk. I loved Smalltalk when I used it a bit in the late 80s / early 90s. But I now understand much more about programming than I did then. I want to compare it to what I now know about Lisp. Does Smalltalks’s simple consistent syntax / semantics actually offer the same kind of elegance, expressivity and power that I now see in Lisp? Plus, how are the modern Smalltalk environments / frameworks for useful application development?
I’m a big Python fan. I’ve written a lot of it over the last 15 years or so. However, everything is Python 2.7. I think it’s time to bite the bullet and get to terms with (and translate my outstanding code into) Python 3. Also, just learn more about some of the Python machine-learning / AI / big-data frameworks.
This year, as every year, I think I’ll finally sit down and do something with Prolog or more likely miniKanren / core.logic. The language is less important here. It’s about understanding how to work with the logic / relational paradigm.

Travelling without a laptop

Why, yes. This is, indeed, OWL running on the PocketCHIP.

Here’s the story.
I spent 12 days recently, travelling in the south of Brazil and Uruguay border. And I decided, as an experiment, to see if I could live without my laptop. I wasn’t meant to be working, but I always like to keep some kind of SDI / MTC style software on me to make notes and generally think about my insanely long queue of tasks and all my other half-baked ideas. I feel lost without having access to these. And would usually take a laptop, just to keep them near to me.
This time, though, I decided to see how well I could cope with a “device swarm” of very small (and cheapish) tech.
Here’s what I took :

  • The PocketCHIP
  • My Android phone
  • A small, portable USB mouse
  • The rubber-tipped stylus.
  • A 5V mini USB charger that plugs into the wall.
  • An A5 sized paper notebook and pen.

On my trip, I also acquired a small bluetooth keyboard / mousepad that’s actually just a capacitative touch surface with letters marked on it. This impressed me a lot when I bought it at a Duty Free shopping outlet just on the Uruguay side of the Yaguarón River in Rio Branco for under $8. It charges via USB and feels pleasantly heavy and solid, while still pretty small.
Here are some things I found out.
I should note, first, that I’m also pretty much against keeping things on public clouds and other people’s servers. I increasingly want my data to be private, on systems that I control, and mainly synced between my own machines rather than using, say, my hosting provider.
Although I’m far from a satisfactory solution to that, it turned out that the discipline was useful in the sense that I wasn’t as persistently connected to the internet as I’d hoped. And so if I had depended on the cloud I would have been without access to my data more often than I had it.
I also, of course, want to use my own software. I currently have three distinct trajectories of development in this area.

  • Mind Traffic Control, racket-version.
  • The new engine behind ThoughtStorms wiki which is written in Python and uses the light-weight Bottle web-framework. I use this for both ThoughtStorms and a second personal wiki-notebook which is descended from my old SdiDesk notebook.
  • OWL

All these projects are still alive. And my notes and todos (and some more significant chunks of writing) are somewhat haphazardly scattered between them. So I wanted to see how well I could take them on the road with me with such minimal hardware.
Before leaving I updated the Debian on the PocketCHIP and installed the relevant libraries / environments. I already had MTC-racket running on it. And Emacs. Python was there too, but I needed to pip install a couple of standard libraries. Then I could install both Project ThoughtStorms and the Python-served version of OWL. (Spoiler alert : 2017  is going to be a year of consolidation between all these projects, particularly I’m aiming to unify the Python-OWL and Project ThoughtStorms servers into a single code-base.)
I didn’t try installing the Android version of OWL on my mobile, because of … er … reasons … which currently prevent me having a working dev / deployment environment for Android on my laptop. I hoped, though, that I’d be able to access the OWL server on the PocketCHIP from the phone.
Results
The PocketCHIP is a wonderful machine. (And, seriously, a Debian box for $9 just insanely amazing.) It seemed to cope just fine with simultaneously running three Python servers (2 copies of the Project ThoughtStorms wiki, and the OWL server) + MTC. (I just ran unix “screen” on the terminal and ran each in a separate screen.)
Obviously these were not being heavily accessed. (I was the only user). But I’m still impressed.
The weakness of the CHIP is its WiFi. It is very weak. My original thought was to run the servers on the CHIP and then access them from either my phone, or other computers in the places I was staying. But even where WiFi was available, the CHIP typically failed to establish a usable connection with the router.
The only time I could get anything else talking to the CHIP was by turning the phone into a hotspot and then placing that within 10 to 20 centimetres of the PocketCHIP.
This way I was able to access both wikis and OWL from the phone. I didn’t get to remotely ssh-ing into the CHIP to see MTC in a terminal from the phone, but this would have been a particular fiddle and it’s not clear that it would have been worth it. In fact, nothing worked particularly well. OWL’s web-interface is pretty much impossible to work with on a small phone screen. It’s OK on a 7″ tablet, but the phone is way smaller and too difficult to manipulate. And the HTML UI doesn’t zoom in any effective way.
Reading the wiki pages was slightly better. I was surprised, though, how badly the phone handled the fairly simple, static html. I accept that there’s very little fancy “responsiveness” in the TS wiki at the moment. I hadn’t realized how little Chrome would help. On my laptop, ctrl + and ctrl – work beautifully to scale text up and down, reflowing and refitting the text. I have no hard settings for font-size or spacing. The page ought to be easy to automatically resize to convenience.
But pinch zooming a TS page on Android Chrome is diabolical. Not only does it not scale and reflow the text in a useful way. It also seems to remember (or guess) arbitrary different zoom levels for different pages, so you jump from a readable page to another page with illegibly tiny letters to another with enormous text, most of which is off screen. And zooming the menu seems to be independent of zooming the main page text. The whole thing is horrible.
Firefox on the phone is a bit better. You can go into “accessibility” and turn up the text size to full. And then the defaults are reasonably readable and consistent on all pages.
Editing is trickier. And this is somewhat my fault, I have set a fixed number of columns in the text-box in TS wiki which is too large for a phone with either browser.
Overall, I’m disappointed with the phone experience. TS wiki is just about readable on Firefox. But it’s useless for doing any kind of work. And horrible on Chrome. OWL looks OK on both browsers, but is too fiddly to actually edit.
I need to radically rework the UI for both these projects.
So I went back to see if I could actually look at these web-served applications on the PocketCHIP itself.
It turns out that OWL works surprisingly well with the surf browser which comes pre-installed … as long as you use a mouse!
The touch screen even with a rubber-tipped stylus isn’t viable for navigating and editing an outline. But attach an external mouse to the USB and it’s surprisingly usable. The PocketCHIP keyboard isn’t great for a lot of writing, but for short items in an outline it’s viable.
TS Wiki is also fine to read. But for some reason the text-area is coming out black, with black text. (I’m pretty sure I’m not setting this explicitly, so I assume it’s a bug in surf.) You, therefore, can’t edit the wiki with it. But reading is an acceptable experience. Once again, a mouse helps, but you can just about get away with the stylus.
The bluetooth keyboard I bought paired fine with the phone. But with the PocketCHIP there were some issues with the mapping between some symbols. I couldn’t find any combination of keys to make a backslash for instance. And sometimes the shift wouldn’t work. Also the bluetooth connection kept dropping. I found myself continually swapping between the PocketCHIP’s own keyboard for typing short commands with a lot of non-alphabetic symbols, and the bluetooth keyboard to type paragraphs of text. It was just enough to get some work done, and the bluetooth keyboard was just better enough to make it worthwhile, but it wasn’t really a viable solution.
Aside
This trip I also got interested in Logic Programming and tried out the Python Minikanren library, logpy, on the PocketCHIP. Unsurprisingly, it worked as expected. But, again, that is kind of startling when you think of it.
Conclusion
Well, this stuff works. If the PocketCHIP just had better wifi / bluetooth connections, then it would be a serious possibility to do some work on. It might be that a software update fixes the power-saving that may be weakening the wifi.
But it’s not something that even I can use yet.
Some of that is in my hands, of course, one task is to tweak the TS Wiki UI to ensure that the text-area is readable in surf. That would at least mean that PocketCHIP could be used (if uncomfortably) for working on my notebooks.
It’s not all working yet … but it’s getting closer.

Project ThoughtStorms Update

Been working quite a lot on Project ThoughtStorms, the new Python / Bottle based wiki engine behind ThoughtStorms in the last few days.
I can’t believe that I’ve spent 15 years building a wiki without creating my own software.
That error is going to be rectified from now on.
Sure, most of what I’m doing is adding the kind of functionality that most wikis already have : RecentChanges, Search, Sister Sites etc. But it’s pretty quick to write. And as I bring these functions back, ThoughtStorms becomes more subtle and alive.
The truth is that ThoughtStorms’ content is woefully outdated. There are a tonne of broken links, and news stories that were current in 2005, but equivalent stories between about 2008 and 2016 are no-where.
It needs a jolly good clean up / clean out / refactoring. But I’m starting to have some ideas about how to write some more innovative custom code to help with that.
It’s very much work in progress. And there’s still plenty that’s half-baked and unfinished.
But I’m starting to feel that ThoughtStorms is coming back to life.

Where I'm At, March 2017

A quick update (for me, as much as anyone else) of where I’m at and what I’m currently thinking about … March 2017:

ClojureScript and Reagent

I want to get up to speed on this as my standard way of writing browser based software. React seems to have conquered the world as browser based, reactive framework. And with React Native it’s also come to mobiles. It seems that the ClojureScript / React Native story is becoming compelling too.
So, yes, this is mainly how I want to do GUIs moving forward. I’m doing some experiments. Expect, eventually, Patterning and some of my other projects to move to this.

WebAudio

A friend of mine has sold me on the idea of going back to audio in the web-browser. Which is obviously a great way to make little musical apps that are easy to share with people. He’s not so keen on ClojureScript but does like Javascript in a functional style with Redux and Immutable.js so we’ll probably be using them on a collaboration. I’m considering porting versions of Gbloink! and Zewp! to it. Along with more of my current Processing based “ArtToys” Library.

Lisps on Android

It may be that ClojureScript + ReactNative is the right way to go. But I’m also looking at Clojure on Android. And Racket on Android.
Reading online it seems that Clojure on Android has been improving, but is maybe not as efficient as ClojureScript with ReactNative. And, also obliges you to deal more directly with the Android Java frameworks. Maybe something that’s useful for larger, more framework intense, applications.
I was originally tempted by LambdaNative, a Scheme for Android app. development. But it now looks like Racket’s Android story has improved considerably and is quite similar. Especially sharing the focus on using OpenGL for visuals. Racket on Android basically compiles to C and uses the NDK, which might be a faff to get working, but might also be more efficient. One to have in the arsenal.
Also, of course, Racket on Android would let me keep a common Racket code-base for both the command-line and an Android version of Mind Traffic Control.

Rosette

I’m kind of intrigued by Rosette and the idea of “solver aided” programming. And being a Racket DSL is a bonus. I want to get my head around this, this year.

Music Languages

Last year. I did a lot on Sonic-Pi. This year I thought I’d go back and take another look at Overtone. Which is, in many ways, the predecessor of Sonic-Pi done as a Clojure library.
It’s not quite as simple as Sonic Pi with its integrated development environment but Clojure is
more powerful than Ruby.
While I still think that Faust looks cool. I’ve also been tempted by the lower-level simplicity of CSOUND … wondering if it might make sense to write a CSOUND generator in Clojure. I’m slightly pulled backwards and forwards on this question. Plus, Overtone provides its own synth-design functions.

Project ThoughtStorms

So, I haven’t trumpeted it here. But last year I moved away from the Smallest Federated Wiki for ThoughtStorms. See ThoughtStorms:LeavingTheSFW for the full story.
Here I’ll just note … I’m very happy with the decision, and I have lots of ideas for how to develop my currently very minimal wiki engine to do more of the things I want to do. In many ways, this is like the MTC reboot of last year.

Small Boards

I’ve become a bit of a collector of small hobbyist / maker boards, from Arduino to last year’s CHIP. I’ve just got a couple of BBC Micro:Bits which are great fun. These are a really smart and original idea in what’s becoming a crowded and competitive market. The computing power is pretty minimal. But what gives them an edge is that they’re packed with sensors. And come with their own primitive display.
I’m looking forward to playing with and giving these a good run for their money.
Interestingly they run MicroPython. Something I hadn’t really looked at before.

Python

All of which means, despite last year’s thinking, that I’m actually renewing my relationship with Python. I really thought I was deprecating it in favour of Javascript. This year, that seems fairly unlikely. Python just is nicer than Javascript in so many ways. It’s the back-end of my new wiki, which I’m planning to extend. It’s the language of the Microbit and potentially useful on other boards. It’s getting ported to the GoLang infrastructure and perhaps has a great future in front of it there. Meanwhile, even a tiny serverside JS program with node.js seems to install a million dependencies.
So Python is definitely here to stay in my near future.

QaSaC

A rather crazy art-project I dabbled with last year spawned an interesting toy / esolang : QaSaC. This is little experimental “concatenative” language like Forth and Joy cross-pollinated with a visual data-flow language like Pure Data.
I think there’s some potential here. And I’ve really enjoyed the concatenative / stack-based thinking and elegance of this kind of programming. So I am working (albeit slowly and interspersed with everything else) on getting an actual viable version of this out soon.
OK. I think that’s a pretty comprehensive list of my obsessions for 2017. Let’s see how much progress I make with them all. This year, I promise, I’ll be trying to release, and blog, more often about them.

Colourize Clojure Errors

Clojure is a wonderful language. The more I use it, the more I like it.
But the default error reporting is catastrophically bad, forcing you to look through screensful of Java stack-trace for the needle of a bit of code that’s actually yours.
I’m sure there are better solutions, but today I just hacked a very quick and dirty python program to filter this result, highlighting the lines that I care about.



#!/usr/bin/python

import sys

p = sys.argv[1]

for line in sys.stdin :

    if p in line :

        print '33[92m' + line + '33[0m',

    else :

        print line,

I made this code in an executable file called “colourit” in my bin directory. Now I can do this :

lein test |& colourit patterning

to, for example, highlight lines containing the word “patterning”.

Conceptual Refactoring : Project GeekWeaver

A few years ago I looked at some of the programs I was writing, both for public release and libraries / scripts for internal consumption, and realized that even I was confused.
I needed a new map to understand what I was doing. Hence I came up with Project ThoughtStorms. A quick and dirty overview of what I was up to in this space of wikis, personal information and knowledge management.
I think that helps. I know what I’m up to, with the legacy of my ThoughtStorms UseMod wiki and SdiDesk, my experiments with Smallest Federated Wiki, the new OWL stuff, etc. It’s still a mess, but it’s conceptually “encapsulated” in a single place.
Now I find myself with a similar situation with a number of different tries at some tools to make it easier for me to produce static web-sites. Originally there was GeekWeaver, a way for me to generate static sites quickly and easily from an OPML outline. That was a great idea … except … it got mixed up with my n00bie enthusiasm for Lisp and desire to create a powerful Lisp-like programming language.
GeekWeaver also foundered on two other issues : soon after I wrote it, there were no popular OPML outliner tools. Although Dave Winer has resolved that issue and I have, indeed, used Fargo to author GeekWeaver programs.
The other, more subtle and pernicious issue is that few people want to develop a site from scratch these days. Most people use huge frameworks with not just Javascript, but CSS, LESS etc. It really doesn’t make sense to try to use a tool which denies the file-system or the fact that you’ll want to be working with a large number of external files.
So the next time I needed to produce some quick static pages I went back to the drawing-board and came up with BootDown : a very simple Python script that lets you write text in Markdown and wraps it in a BootStrap / BootSwatch template. It’s the opposite of GeekWeaver, it isn’t trying to be a rich and clever language that lets you define your own templates, it’s just a quick way to take advantage of a lot of templates that already exist.
BootDown adds two things to conventional Markdown. A shorthand for declaring div tags. And a page separator. So you can still make a multi-page site inside a single file. And you can explicitly reference the div classes and ids you need.
More recently still, I’ve been getting into a real Lisp : Clojure. Today, that’s the language I really want to be working with. And I have a current project that again involves generating a multi-page, fairly static site. But the inputs are a little bit more complicated : some more structured data, not just text. And it makes sense to store it in an outline. So I’ve started writing Clojure code to read OPML and spit out a BootDown Markdown file, to wrap in HTML.
I can see this project starting to grow too.
And so here I am … three Python code-bases : GeekWeaver, BootDown and the OWL back-end server. Two Clojure projects : the beginnings of a library to work on OPML outlines (OPMLKit) and my unreleased code I’m writing for this current project. And some clunky pipelines to use them together. It’s starting to be another mess. And that’s why I need Project GeekWeaver : an umbrella to pull together these different strands together, at least conceptually.
There are some thoughts … would it make sense to pull at least some of the original GeekWeaver code INTO the OWL back-end server? So that “run transformations on this outline” is a standard OWL feature, and OWL becomes GeekWeaver’s built-in IDE? But what about the OWLdroid version that doesn’t have Python? Should I rewrite that code in CoffeeScript. (Or ClojureScript … a working but still fairly slow option at the moment.) BootDown “just works”. Am I better off scrapping the GeekWeaver idea altogether and just focusing on such practical solutions?
Some of this will become clearer as I work through this current site I’m making. meanwhile Project GeekWeaver is the category and (shortly) the site for the high-level overview.

Clojure

OK. Consider me won over.
Lisp is great to work with. The things that grabbed me about FP in Erlang and Haskell (pattern-matching arguments, partial application, lazy evaluation) are all here.
My code is as concise as Erlang and damned nearly as concise as Haskell (I think the line count is similar though the number of characters per line is about twice as high)
Although I respect Haskell’s type system, Clojure’s dynamic typing holds me up less. I *am*, admittedly, hitting more problems that a type system would have picked up. (Far more than I typically hit in Python) but I think as I get more used to the language this will go down.
I’m not being particularly demanding in terms of libraries. (Quil is almost all the libraries I need at the moment.) But I’m finding that there are all these handy things like spit which make me smile. (Remember what this was like in Java?)
Things that would still be a bit fiddly in Python / CoffeeScript continue to turn out to be easier than I imagined they would be when I implement them in Lisp. That’s partly because I default to approaching them as “how do I write a small lambda that can then map across this list” and mostly, by the time I wrote that function, all I have to do is … er … map it across the list.
Lisp is extra-ordinarily compressible. I keep finding ways to refactor and fold up things I did earlier to make them shorter and shorter. And the more functionality I add to my program the more compressed it seems to get.
I am grinning to myself at the thought that my code might be compilable to both Android and into Javascript. I haven’t tried either yet, but if I manage it then Clojure is now my new favourite language.
CoffeeScript isn’t at all bad, but if ClojureScript works out, then I’ll probably start to move aggressively to use it as my default in-browser language. (And yes, I guess that may mean rewriting OWL in it. OWL is still short enough that I think I could sprint it in a couple of days.)
Similarly, if I can compile Clojure libraries that can be called from Java projects then you’ll start to see me more productive on Android. (Beyond OWLdroid I have a couple of other bits and pieces of apps. written, but not taken them to completion. If Clojure slots painlessly into the workflow, it will be a lot more tempting to dive back in.)
So … yeah … Clojure rocks!