Why is my phone so useless?

Why is my phone so useless?

I mean I’m a programmer. I can make it do what I want, can’t I?

And yet, actually, I’m as bad as anyone. Allowing my phone and its apps. to define my activities.

I still depend on the laptop for all the things that are for me. My phone makes me a consumer of other people’s services and channels me in its own direction.

I don’t have an answer to this. So I’m just making a ThoughtStorms page to track it.

Microsoft adds Python to Excel

Several years late, it seems M$ is considering making Python native to Excel.

I wrote a … you guessed it … Quora answer as to why this makes sense for Microsoft.

—-

Python has become massively popular with the data and machine learning communities in the last few years.

Tools like JuPyter are increasingly popular and serious interfaces for data-modellers who previously would have used Excel.

It absolutely makes sense for Microsoft try to embed Excel and itself into that emerging Python data ecosystem by making Python a first-class citizen (ie. default, guaranteed to be there) of Excel.

Not just a third-party add on for those who know about it and can make the effort to install it.

Not only does Python need to be standard within Excel, but access to pip and all the Python libraries needs to be there too. So that Excel becomes the equivalent of Anaconda

That’s the way that M$ can keep Excel relevant in the new data age.

This is not only a good idea for Microsoft. It’s the difference between Excel remaining a major player in data modelling and analysis tools, vs. declining into obscurity.

Gmail Snooze

Is GMail Snooze basically building Mind Traffic Control into your email client?

Well, it’s about time Google did something radical to improve GMail. There’s still so much untapped potential in the mail-box. And at least email is an open protocol that we should defend against moving to walled rivers like Facebook and Slack etc.

Working with the file system is too verbose. Let’s make it more like JQuery!

Working with the file system in Python is too verbose.

Seriously.

Every time I want to do something with files I need to remember whether it’s the `os` or `sys` library that I need to import. I need to remember or look up half a dozen other functions to grab particular bits of metadata from those files.

I’ve written countless nested loops or recursive functions to walk the tree of files over the years.

And often I want to do something quick to a bunch of files, start writing shell-script. Then realize I hate and can’t remember shell-script and think it would be so much easier in Python. Then I open the editor to try to write my code in python, and I realize it’s too much trouble and go back to mashing around in bash again.

And then I remembered JQuery with its refreshingly easy abstractions for maneuvering in, and manipulating a tagged tree-shaped data-structure. And the file-system is just a tree, right? So why does it have to be hard? Why shouldn’t working with files just be like working with JQuery?

This is such an obvious idea, I’m sure someone must have done it.

But I couldn’t find it. So I had to write it myself a couple of months ago.

So, let me present … FSQuery . Now available on PyPI and GitHub.

Here’s what it looks like :

from fsquery import FSQuery

fsq = FSQuery(path).Match(".js$").NoFollow("vendor").FileOnly()  
for n in fsq :
    print n.abs

You create an FSQuery object with a path to the root of the files you’re interested in. You then load up extra filters / queries on the query by chaining them together.

Finally you can treat the whole thing as an iterable collection and loop through its results.

It returns object of class FSNode, which represent nodes in the file-system, either files or directories. The modifier FileOnly(), restricts the query to only return files. You only need to add this once to the query.

The NoFollow() method tells the query to avoid directories that match the name. But has no effect on file names. In the above example, “vendor.txt” would still be included in the results if that file is anywhere other than under the vendor directory. (This beats just piping find through grep in the terminal.) You can add as many NoFollow filters as you like to a query.

On the other hand, Match() is an inclusive filter. Only files whose names are explicitly matched end up in the results. However, this filter isn’t applied to directories. FQuery will still explore and return directories whether they match this or not. You will usually want to combine a Match with a FilesOnly to get the effect you want (eg. in this case, all the .js files anywhere except under the vendor directory)

We can even look inside files with the Contains filter, eg.

fsq = FSQuery(path).NoFollow("vendor").Ext("py").Contains("GNU Lesser General Public License").FileOnly()

Note that this is implemented purely in Python in a very non efficient way. (Ie. I just open up each file and run through it looking for the string.) It can be slow with large chunks of the file system.

Note also the Ext() filter for file-name extensions. This is easier than regexing the whole file-name if you’re just looking for files of type “py”. Be aware that if you try to have two Ext() filters on the same query, you will get no files returned. No file can have two different extensions at the same time.

More documentation, and more advanced tricks can be seen on the GitHub site. This is also the first library I’ve put on PyPI. So installing in your own project is as simple as

pip install fsquery

A Replacement for PHP

I asked Quora for ideas for how a replacement PHP might look.

On the whole people are not enthusiastic. Alexander Tchitchigin had an interesting answer, but which focused on the basic theme of “once we move away from PHP’s weaknesses, we might as well use any language.

Which prompted me to write this comment elaborating what I was interested in. Plus some ideas of my own.

I agree that there are downsides to “embedded in HTML”.

But I think we can also see various what I call “pendulums” in computer science. For example between centralization and decentralization. Centralization gives you economies of scale, eliminates redundancy and makes it easier to see the wood for the trees. Decentralization makes it easier to modularize (or divide and conquer), easier to test and find bugs, easier to scale, easier to improve individual modules etc.

The pendulum oscillates because whenever one of these principles becomes more dominant, everyone starts to feel the pain and see the attraction of the other pole. And then stories start proliferating of the virtues of shifting the other way. Once everyone does, of course, there’s a pull back to the first way again. And so the pendulum continues to swing.

Right now I’m seeing this centralized / decentralized tension in ClojureScript web-frameworks. Comparing using Reagent directly with devcards vs. using Re-frame. Trying to decide whether the convenience of the modularity of keeping state decentralized in individual reagent components and being able to use devcards, outweighs the extra transparency of centralizing state in the re-frame db.

Now I think this thing about “MVC vs. templates with code” is another pendulum rather than an absolute principle.

At the end of the day, HTML is the data-structure for the application’s GUI. And the GUI data-structure does need to be fairly tightly integrated with the code. It doesn’t make sense to try to decouple them too much. You need a button attached to a handler attached to some transformation in your business logic. There’s no point trying to keep these things apart. A button without functionality makes no sense. Nor does functionality that can’t be accessed through the UI.

I’m now using Hiccup to generate HTML. And the place to do it is obviously tightly integrated with the actual functionality of the app in the code itself.

Yes, there are still some MVCish intuitions at work. But I don’t need or want a language to try to hard enforce that separation between UI and functionality either in separate files or with separate languages, when a simple DSL in the main programming language is sufficient.

But when you look at a Reagent (ie. React) component it’s basically a Huccup template with some extra code in it.

Now the brilliance of PHP, the reason it’s so popular is that :

a) it’s just there, pretty much always.

b) it simplifies simple sites considerably by automatically mapping the routing onto the directory structure of the file-system. There are many cases when that is fine. Why should I have to hand code a whole layer of routing inside my code when the file system already provides me with a logical hierarchical layout?

I think there’s still value in the “map files to pages” part of PHP. And that’s what I’d expect a “new PHP” to keep. Along with the “available everywhere” bit.

Of course, how it might look, might be more like Hiccup, a light-weight DSL rather than the verbosity of HTML. With each file implicitly mapped to a React component. Perhaps something like I started describing here : Phil Jones’ answer to What’s the best programming language for applications and GUIs?

In that other answer, I talk about what I found interesting in Eve : the event-handling within the language through “when” clauses and an implicit underlying data-structure.

And I tried to sketch what a language that brought events to Hiccup might look like :

 <pre>
    (defcomponent click-counter
        {:localstate ['counter 0]
         :when (on-click "the-button") #(reset! counter (+ @counter 1))
         :view
          [:div
             [:p "Counter clicked " @counter " times"]
             [:button {:id "the-button"} "Click Me"]
          ]
        }
    )
</pre>

Many frameworks encourage you to put components etc. in different files and directories anyway. Why not make this “official” in the same way that Python makes indentation official. And use the directory structure to infer the program structure?

A powerful modern language with the easy accessibility defaults of PHP would be a powerful combination.

August is Patterning Month

August is Patterning month again.

I’m back to work on the Patterning library. And, in particular, getting it working properly in the ClojureScript, in-browser version. I’m going to be using devcards, figwheel, spec and other good tools in the Clojure community.

I’ll be revamping the site and new versions of the code.

Watch this space …

Expression in Programming Languages

Source: My Quora Answer :

If Clojure is so expressive, and is basically Lisp, hasn’t there been any progress in expressivity in the last 50 years?
Well, as Paul Graham put it quite well, Lisp started as a kind of maths. That’s why it doesn’t go out of date. Or not in the short term.
You should probably judge the evolution of expressivity in programming languages by comparing them to the inventions of new concepts / theories / notations in maths (or even science), not by comparison to other kinds of technological development like faster processors or bigger hard drives.
What I’d suggest is that it turns out that function application is an amazingly powerful and general way to express “tell the computer to calculate something”.
Once you have a notation for defining, composing and applying functions (which is what Lisp is), then you already have an extraordinarily powerful toolkit for expressing programs. Add in Lisp’s macros which let you deconstruct, manipulate and reconstruct the definitions of functions, programmatically, and you basically have so much expressive power that it’s hard to see how to improve on it.
You can argue in the case of macros that, in fact, while the idea has been in Lisp for a while, the notation and semantics is still being actively fiddled with. We know that macros in principle are a good idea. But we’re still working on the right language to express them.
Perhaps we can also argue that there’s still room for evolving some other bits of expressivity. Eg. how to best express Communicating Sequential Processes or similar structures for concurrency etc. In Lisp all these things look like functions (or forms) because that’s the nature of Lisp. But often within the particular form we are still working out how best to express these higher level ideas.
Now, the fact that functions are more or less great for expressing computation doesn’t mean that the search for expressivity in programming languages has stopped. But it’s moved its focus elsewhere.
So there are three places where we’ve tried to go beyond function application (which Lisp serves admirably) and improve expression :

  • expressing constraints on our programs through types.
  • expressing data-structures
  • expressing large-scale architecture

These are somewhat intertwined, but let’s separate them.
Types
Types are the big one. Especially in languages like Haskell and the more exotic derivatives (Idris, Agda etc.) Types don’t tell the computer to DO anything more that you can tell it to do in Lisp. But they tell it what can / can’t be done in general. Which sometimes lets the compiler infer other things. But largely stops the programmer shooting themselves in the foot. Many programmers find this a great boost to productivity as it prevents many unnecessary errors during development.
Clearly, the type declarations in languages like Haskell or Agda are powerfully expressive. But I, personally. have yet to see a notation for expressing types that I really like or find intuitive and readable. So I believe that there is scope for improving the expressivity of type declarations. Now, sure some of that is adding power to existing notations like Hindley-Milner type systems. But I wouldn’t rule out something dramatically different in this area.
One big challenge is this : by definition, types cut across particular bits of computation. They are general / global / operating “at a distance”. One question is where to write this kind of information. Gather it together in standard “header” files? Or spread across the code, where it’s closest to where its used? What are the scoping rules for types? Can you have local or “inner” types? Or are all types global? What happens when data which is typed locally leaks into a wider context?
Data Structures
Lisp’s lists are incredibly flexible, general purpose data-structures. But also very low-level / “primitive”.
I don’t really know other Lisps. But it seems to me that Clojure’s introduction of both a { } notation for dictionaries / maps. And a [ ] notation for data vectors has been useful. Complex data literals can now be constructed out of these by following the EDN format. And it’s given rise to things like Hiccup for representing HTML and equivalents for other user-interfaces or configuration data. EDN is pretty similar to the way you define data in other contemporary languages like Javascript’s JSON etc. So it’s not that “radical”. But it’s nice to have these data structures completely integrated with the Clojure code representation.
Can we improve expressivity for this kind of data representation language?
I’m inclined to say that things like Markdown or YAML, that bring in white-space, make complex data-structures even more human readable and writable and therefore “expressive” than even JSON / EDN.
In most Lisps, but not Clojure, you can define reader-macros to embed DSLs of this form within programs.
So Lisps have highish expressivity in this area of declaring data. In Clojure through extending S-expressions into EDN and in other Lisps through applying reader-macros to make data DSLs.
Can we go further?
By collapsing data-structure into algebraic types, Haskell also comes up with a neat way of expressing data. With the added power of recursion and or-ed alternatives.
This leads us to imagine another line of developments for expression of data structures that brings these features. Perhaps ending up like regular or context free grammars.
Of course, you can write parser combinators in any functional language. Which gives you a reasonable way to represent such grammars. But ideally you want your grammar definition language sufficiently integrated with your programming language that you can use this knowledge of data-structure everywhere, such as pattern-matching arguments to functions.
Haskell, Clojure’s map representation, and perhaps Spec are moves in this direction.
But for real expressivity about data-structures, we’d have a full declarative / pattern-matching grammar-defining sub-language integrated with our function application language, for things like pattern matching, searching and even transformations. Think somewhere between BNF and JQuery selectors.
Shen’s “Sequent Calculus” might be giving us that. If I understand it correctly.
A third direction to increase expressivity in defining data-structures is to go beyond custom languages, and go for custom interactive editors (think things like spreadsheet grids or drawing applications for graphics) which manipulate particular recognised data types. These increase expressivity even further, but are very domain / application specific.
Architecture
“Architecture” is everywhere. It describes how different modules relate. How different services on different machines can be tied together. It defines the components of a user-interface and how they’re wired up to call-back handlers or streams of event processors. “Config files” are architecture. Architecture is what we’re trying to capture in “dependency injection”. And class hierarchies.
We need ways to express architecture, but mainly we rely either on code (programmatically constructing UIs), or more general data-structures. (Dreaded XML files.)
Or specific language features for specific architectural concerns (eg. the explicit “extends” keyword to describe inheritance in Java.)
OO / message passing languages like Smalltalk and IO do push you into thinking more architecturally than many FP languages do. Even “class” is an architectural term. OO languages push you towards thinking about OO design, and ideas like roles / responsibilities of various components or actors within the system.
Types are also in this story. To Haskell programmers, type-declarations are like UML diagrams are to Java programmers. They express large scale architecture of how all the components fit together. People skilled in Haskell and in reading type declarations probably read a great deal of the architecture of a large system just by looking at type declarations.
However, the problem with types, and OO classes etc. is that they are basically about … er … “types”. They’re great at expressing “this kind of function handles that kind of data”. Or “this kind of thing is like that kind of thing except different in these ways”.
But they aren’t particularly good to express relations between “tokens” or “concrete individuals”. For example, if you want to say “this server sits at address W.X.Y.Z” and uses a database which lives at “A.B.C.D:e”, then you’re back to config. files, dependency injection problems and the standard resources of describing data-structures. Architecture is treated as just another kind of data-structure.
Yes, good expressivity in data-structure helps a lot. So EDN or JSON beats out XML.
But, really, it feels like there’s still scope for a lot of improvement in the way we talk about these concrete relationships in (or alongside) our programs.
OK, I’m rambling …
tl;dr : function definition / application is a great way to express computation. Lisp got that right in the 1960s, and combined with macros is about as good as it gets to express computation. All the other improvements in expressivity have been developed to express other things : types and constraints, data-structures and architecture. In these areas, we’ve already been able to improve on Lisp. And can probably do even better than we’ve done so far.

Logic Programming is Overrated

An interesting argument that logic programming is overrated

The reason why logic programming is so rarely useful is that, essentially, core.logic is just a complex DSL for doing exhaustive search. Clojure already has an elegant, compact DSL for doing exhaustive search — it is called the for comprehension.

Could the internet disappear within 20 years?

From a Quora answer I just made :
The protocol will still exist.
But there is now a real danger that “the internet” will have been successfully enclosed by private corporations.
What will that look like?
1. the end of net neutrality means that the phone companies / network providers can choose to prioritise packets from certain companies / services. They start to demand a premium from popular services.
2. most people don’t notice or care because they only use a few popular services anyway : Facebook, Google, Microsoft, Twitter, Pinterest, Netflix etc. These large corporations complain but start to pay the premium to be first class citizens. Packets from other services / web-addresses get deprioritized.
3. the commercial Content Delivery Networks probably play a role here. Basically anyone who wants to provide content to the public will need to go via a CDN because they’ll be the ones cutting the deals between the phone companies and any company smaller than Facebook and Google. That’s probably how Quora and Pinterest etc. will keep going.
4. Meanwhile, governments will continue to try to reassert control over internet content. In the name of clamping down on piracy, pornography, fake news, terrorism, online harassment, illegal crypto-currencies etc. they’ll start putting more pressure on online services to police their content. Facebook and Google will be increasingly required to co-operate. Smaller services will find they need to be vetted to get onto the CDN.
5. Also networking “hardware” is increasingly moved into software. Routers become generic boxes whose protocols are just software configuration.
6. Finally, phone companies, CDNs and router companies figure out some “improved” protocols which make the shaping and inspection of traffic more efficient. Stripped of the need to route generic packets from anywhere to anywhere, this new infrastructure, brought through a software update, is sold to the consumers as an “upgrade”.
7. By now, consumer IP is dead. Email protocols stop working. (Google and other “web-mail” providers arrange an alternative that looks the same but works over the new protocol. Or perhaps integrates directly with other messaging providers.) Communication is via Facebook and Twitter which work on the new protocol. Other P2P protocols that rely on IP also stop working. No more bittorrent, btsync, syncthing, webrtc etc.
IP survives as a “legacy system” for corporate users of course. Too many large companies still depend on it. But it will now be expensive, like most “enterprise” kit.
Within 20 years, we’ll find that the remaining “free” internet, where any ordinary person can hire a web-server and put up their own content independent of any of the major corporations or systematic government oversight is reduced to a tiny minority of hobbyists like amateur radio. Most people never access it or even know of its existence.
If you don’t like this scenario, NOW is the time to do something about it.
Not just do what you can to defend net-neutrality, but start to use the free / peer-to-peer, open protocols that we still have. Run your own web-site or blog. Use RSS to share content. Use webrtc to do your video conferencing, and bittorrent to share legitimate files. Do what you can to make sure that there is just too much demand and usage of the open protocols and internet that the large corporations can’t afford to try to switch it off because they know they’ll be losing too many customers.
Once most people aren’t using the internet, taking it away becomes easy.

Languages for 2018

Bit early. But I answered a question on Quora about languages to learn for 2018.
Here’s what’s interesting me for 2018 :
I want to continue getting more experienced and better with Clojure. No language is perfect, but for me Clojure is the best language I’ve ever used. And I want to use it for more projects and in more different situations. I want Clojure to be my default / “workhorse” language for server-side, browser-based UI, Android apps. etc. Clojure is not just a great language but a practical language. And I’m expecting there to be more jobs / contracts available with it, going forward.
I’m intrigued by Rust. I haven’t even installed it yet. But I want to try it as a low-level C alternative. I have an idea it might be suitable for.
I admit that Richard Kenneth Eng and Peter Fisk are getting to me. I’d quite like to go back and have another look at / play with Smalltalk. I loved Smalltalk when I used it a bit in the late 80s / early 90s. But I now understand much more about programming than I did then. I want to compare it to what I now know about Lisp. Does Smalltalks’s simple consistent syntax / semantics actually offer the same kind of elegance, expressivity and power that I now see in Lisp? Plus, how are the modern Smalltalk environments / frameworks for useful application development?
I’m a big Python fan. I’ve written a lot of it over the last 15 years or so. However, everything is Python 2.7. I think it’s time to bite the bullet and get to terms with (and translate my outstanding code into) Python 3. Also, just learn more about some of the Python machine-learning / AI / big-data frameworks.
This year, as every year, I think I’ll finally sit down and do something with Prolog or more likely miniKanren / core.logic. The language is less important here. It’s about understanding how to work with the logic / relational paradigm.