Clojurians Log v2

Thanks all! Why does Chris Nuernberger say that one should use containers when using clojure and clj-python?

@metasoarous I have to read up on the GIL. As for parallelization, doesn't python have a library for that? I know distributed computing is not the same as parallel computing, but doesn't Spark solve part of their speed problems?

@daslu so the main take away from the long text are fun, easy to extend, high productivity, better REPL, right?

@aaelony I'm afraid "one language to rule them all" will just make pythonistas suspicious. For the package-mangement problems, could you please tell me more? I used pip before, and it worked.

@nick.romer I think that is a nice way to summarize it! I like the point it makes about the REPL in the last paragraph.

Because of this visibility advantage a common way to use Clojure is to model your problem as a transformation from datastructure to datastructure, testing each stage of this transformation in the repl and just letting the REPL printing show you the next move. 

@metasoarous typing races: are you saying that one can type vectors faster in Clojure? (Sorry, my command of English is mediocre, I don't have access to hidden meanings).

@daslu I don't know these python functional pipelines that @metasoarous talks about. Can they be used for transforming datastructure to datastructure? Background info: I've just been accepted as junior data scientist and will have to survive in python until I have gained enough credibility to start clojure evangelism. Worse, I'm newbie to both clojure and python.

... I've used transducers with core.async, so I have a vague idea what these transformations might look like.

I don't know so much about the different possibilities of doing Functional Programming in Python. When I have to, I like to compose things with toolz.curried.pipe as demonstrated here: https://toolz.readthedocs.io/en/latest/streaming-analytics.html but probably there are other options. Hoping your time with Python will not be so bad : ) .. till your time of evangelism comes.

Yes, one can work with datastructures in Python in a somewhat functional fashion. But I think the Clojure REPL experience make it clearer and simpler. Probably one reason for that is that things are Printed in the same notations they are Read. So the Read and Print parts of the Read-Eval-Print-Loop speak the same language. Does it make sense?

Yes, that makes sense.

user=> ;; Evaluate something:
user=> (update {:x 9} :x inc)
{:x 10}
user=> ;; Take the printed result and pass it to be read for the next evaluation:
user=> (update {:x 10} :x inc)
{:x 11}

That's clear. Different question. Are neanderthal and numpy overlapping in functionality? Would you mix them?

Afaik they overlap. Never tried to mix them.

and thanks for the toolz link.

@chris441's tech.dataset has some level of interop with numpy matrices, but I don't think we get this with neanderthal.

toolz looks cool; I've used some thing s like that. I just hate that I have to go to a second hand library to get all the Functional Programming goodies. Basically every python program I've written since learning Clojure results in me implementing a few clojure.core functions. I'd rather not have to do that work if I don't have to.

And yes, Python is a fairly functional language as far as OOP languages go. In fact, I'd argue that at a core design level, it's actually functional first! The reason is that in Python, Objects & Methods are built using basic functional primitives. So it's not like Ruby, which for all the hype over having lambdas, is different in that in Ruby methods are not functions. It's possible to get to the function (lambda, really) for a given method, but it's an extra step. By contrast, in python, every object has a "magic" o.__dict__ attribute which points to all of the Object attributes and methods (as functions!) that the Object (and class) knows about. Which is hella elegant in my opinion. For an OOP language, Python is pretty nice (again, IMHO).

However... And this is a big "however": Functional programming is deeply limited in power (compared to its full potential) when you don't bake persistent/immutable data structures into the language. JS has this problem too; It has first class functions, and at least used to require you to use protyping to build objects (which is also a functional pattern, if not sometimes a messy one). But the lack of persistent data structures means you can't squeeze every bit of juice out of those FP patterns. And this isn't just academic: Perhaps the single strongest part of the JS ecosystem right now is React, which is fundamentally a Functional Reactive Programming paradigm. Time and time again I have found that vanilla JS React is significantly limited relative to ClojureScript+React due to this limitation. As with toolz in python, there are libraries for adding persistent data structures to JS. In fact, mori just rips off ClojureScript's persistent data structures as a JS lib! Which is great, but also not the same as being a pervasive and fundamental assumption in the language. In Clojure everything is build up around these ideas and so they carry much more heft and power.

Taking comments/questions in turn... <deep-breath>

Next up for package management (now let the real grilling begin).

Yes, pip works fine for the first few pacckages you install.

Until you end up needing to install a package that depends on a different version of a package than the one you already have installed. In python you cannot install more than one version of a package in a given python environment! Which, to be honest, is fucking lunacy. Ruby, JS, Java and by implication Clojure all dodge this bullet. Each project specifies which version it needs, but you can have multiple versions installed along side each other.

Python does not allow this, and thus there have been scores of projects which try to patch over the fundamental failing, and none of them have really done a good job of it (again, IMHO).

In virtualenv you actually create separate environments for each project, and have to install things separately in each one of them. Managing and switching between these becomes a pain.

Pip has finally embraced this pipenv project that gives you something close to what we have had forever in a lot of languages, which is a single tool for specifying packages and versions in a file, and managing virtualenvs based on those: https://pypi.org/project/pipenv/

Hopefully that project is going well. It's very new, and I haven't looked at it since maybe 1.5yr ago. When I did, I had some issues with it, but it seems to be doing the right thing by copying "bundler, composer, npm, cargo, yarn, etc." (aka all the things that other languages have had for years, and Pythonistas are only now coming around to...).

To be fair, I've found Python libraries to be pretty good about not breaking by themselves, probably because of lived experience with the pains of package management in this kind of system. But also because of the Python etiquette for simplicity (as they see it, at least: "there should be one obvious right way to do things"; to be contrasted with the ruby philosophy of "monkeypatching goes vrrrrrrmmmm!"), I think there's a natural inclination towards some of the Clojure philosophy of simplicity, which may rub off in part as "try not to break things". In Ruby, any upgrade of any major Rails or Active whatever infrastructure meant a big refactor. Not as frequently the case in Python, in my experience. So credit where due.

However, and we all knew this was coming. Drullroll please!

Py3k.

The language itself likes to break things! 🎉 (Actually I think it may be worse than Ruby in this ironically, though I really haven't used Ruby in years)

It has taken Python something like 15 years to transition to Python 3. People are still running Python 2! It's getting more and more uncommon, but contrast the situation with that in pretty much any other language.

A lot of Pythonistas hate this, and I think it will prime them nicely for Clojure, which in language and community has an exceptional ethic of not breaking things.

I have seen such an ethic in exactly 0 other languages I have spent time with. Everyone seems to think that by incrementing a version number it's safe break things (semvar). Wrong! It still causes pain! https://www.youtube.com/watch?v=oyLBGkS5ICk

Anyway, the part that pipenv (see above) doesn't solve is the rest of your system (native dependencies, etc). This is more in the realm of conda, which to some extent does a decent job of aiming at that problem, but also (to my knowledge) doesn't interoperate with pipenv 😂🔫

Clojure has a sort of unfair advantage here, which is that it's hosted on the JVM, and so there generally aren't as many system-level libraries and such needed.

This is all important context for understanding how Clojure could potentially help, and is where we dovetail with @chris441's point about containers.

Because Clojure also doesn't really solve the problem of native/system dependencies (again, because it mostly doesn't have the problem, except when doing things like this: trying to interop with python or low level computational libraries), there's a space here where if we build good tooling (ideally as an extension (or at least in compatibility with) the deps.edn config) we could potentially solve some of the combined problems of conda+pipenv :thinking_face: 🏗️

Imagine a world where pythonists would use this cool new tool for solving their combined conda+pipenv needs, that lets them write in python! 🌈 :unicorn_face:

And where that tool just happens to be implemented in Clojure and comes preconfigured with libpython-clj so that curious Pythonistas can dabble in the divine art of true parallelization and immutable data! 🕍 🎆 🙏

I get that none of this is easy, but all of this brings me to my last point in response to @nick.romer's questions/comments: Parallelism!

Python does have some libraries for parallel computation, but they are nothing like the support we have in Clojure. A lot of this (to be honest) comes from being on the JVM. The JVM is awesome for threading. Clojure takes it to the next level by providing pervasive persistent & immutable data, which solves a lot of the problems one gets with place oriented (and Object Oriented) programming in particular (see https://www.infoq.com/presentations/Are-We-There-Yet-Rich-Hickey/).

But it goes even further than that! Clojure provides state management primatives like atoms, refs, agents, futures, and abstract/custom dereffables (like Reagent's ratoms)!

It also comes with concurrency tooling like core.async, which, it's worth mentioning, is a testament to Clojure's macro capabilities as a lisp! 💪 With macros, lisps allow users (programmers, libraries) to extend the syntax of the language, and so we were able to copy design features from the Go programming language 🎉

What python has (the multiprocessing library) only allows you to spawn new processes from a parent python process.

This is different than being able to spawn new threads in that processes can't share data in memory like threads can. You're stuck with message passing which requires data serialization/deserialization, and can severely constrain the kinds of things you can do.

In python, the GIL means that only one thread can ever be running at time.

So you can share memory between "threads", but they're not really threads, because you can't get them to run in parallel.

Spark is basically just a layer of abstraction over lots of separate processes, which might be python or other (in fact its jvm, so we win there as well cause it's easier to interop there, and we have some very snappy spark wrappers, or you can interact with the jvm code directly).

But you're still going to have the same constraints around processes passing messages, and will have added overhead of... Spark as infrastucture.

Bottom line: There's a lot that Pythonistas stand to gain by working with the Clojure community, but the reverse is also true. We should be thinking about how our communities can benefit each other.

The idea of "one language to rule them all" is in some ways a sort of lisp pipe dream, but also one which Clojure has sort of come closest to with it's hosted philosophy & design.

But it's not really "one language to rule them all", it's "one language to connect them all" 🌈 :unicorn_face:❇️

Oh; looked up and saw two more things to respond to: > typing races: are you saying that one can type vectors faster in Clojure? (Sorry, my command of English is mediocre, I don't have access to hidden meanings). Yes, not as in static typing but as in literally keyboard typing. Try it:

[1 2 3 4 5 6 7 8 9]
;vs
[1, 2, 3, 4, 5, 6, 7, 8, 9]

Obviously, it's a little bit silly calling out little things like commas, but that's exactly the point! Anyone who says "BuT thE COmMaS!?!" is a) focused on minutia, and b) not seeing the whole picture.

I would take Clojure's parens over the commas and semicolons and syntactic conflation/complexion of code blocks with data structures (looking at you { ... }) of other languages any day.

Last thing: functional pipelines. @nick.romer You are already familiar with transducers, so you more or less know what I'm talking about. But also, transducers are the more decomplected (simpler/more-general/more-powerful) versions of a thing we have long done with the -> and ->> (and cond->/`cond->>`) macros, which more or less do the same thing but greadily. (-> x (f :arg) (g 1 2)) is the same as

(g (f x :arg))  

(filter g
        (map f xs))

Basically just a nice way of taking (certain) deeply nested Cojure expressions and rewriting them as a sequece of operations, much like with transducer composition. I guess the above exampole looks better like:

(->> xs
     (map f)
     (filter g))

I actually usually end up writing threadf and threadl functions in python projects now, where I pass vectors of [fn, arg1, arg2], because I find it easier to read.

OK; Sorry for the long rant. Hope that was helpful.

Wow, this thread is crazy long. Even so, I think a few points of clarification might help. Neanderthal is analogous to Numpy while http://tech.ml.dataset (TMD) covers more the Pandas, data.table http://et.al. space. So, Neanderthal is for real number crunching while TMD is for dataframe style slicing and dicing. Python doesn't really have any parallelism capabilities (due to the GIL issues) - it does have concurrency. Sort of. Anyone who has ever tried to do concurrency in Python knows it is just awful. There have been loads of libs over the decades trying to 'fix' this in various ways. Asyncio was supposed to be the final true way and was 'blessed' by Guido. But it is ludicrously complex and error prone. Trio is a lib that actually finally does (mostly) work for concurrent / async work. Thankfully. I threw away a bunch of brittle horrible asyncio code and replaced with trio code which actually worked w/o random exceptions and other idiocies. The python2 vs python3 fiasco isn't as bad as perl 5 / 6 mess, but it is closer than you might think. There is good evidence (and reasons) that py2 will be around more or less forever. IMO, Python is very non functional in almost all aspects. You can beat on it to some extent to use it in a broken functional manner, but it fights you all way. This isn't really surprising as Guido is on record dissing functional programming. Indeed, he tried hard to get rid of the simple functional constructs that it does have but failed.

Thanks for the clarifications and additional context @jsa-aerial.

My point about http://tech.ml.dataset is that IIRC libpython-clj does a bit of work to give you zero-copy mappings to/from http://tech.ml datasetructures and what I thought were numpy data structures. I could have gotten that wrong, and that mapping is more directly to pandas. In any case, you're point is well taken that conceptually http://tech.ml.dataset & pandas serve the same role.

The only real way that python is functional is in embracing first class functions as values. Which in my book is sort of the minimal sufficient condition. And again, their OOP approach leans heavily on that assumption, hence my perspective on it. But you're absolutely right @jsa-aerial that this is pretty weak sauce relative to what you can do with first-class functions when you build around a core of immutable data structures & functions, and provide utilities for managing state separate from value (atoms, software transactional memory, etc).

BTW, not having commas is actually important in making code-editing easier. It is not just about aesthetics and space-saving. In languages that require commas, whenever I have to comment out some part of a big data structure (say, a long list, or some nested dict, or something), it becomes quite annoying. Lots of editing is required to take care of the commas. In clojure, you just comment out (`#_`) the relevant inner form, without any such bother.

@metasoarous I don't think you are wrong about that bit (as Pandas sits on top of Numpy to various extents), but TMD is focused on the Pandas columnar data sliceing and dicing stuff - Clojisr also uses it to map to/from R dataframe things (dplyr / data.type / http://et.al.). TMD also tries (like Pandas on top Numpy) to utilize efficient native memory.

Re: Python func stuff: Yeah, I think that is a reasonable point Chris. And following that line of thought, (it hurts to say this but...) JS is much more functional that Python

How do you quote if F-ing slack anyway??? Daniel: "BTW, not having commas is actually important in making code-editing easier. " This is definitely a VERY big deal. This has made Saite code transformation (at the editor level) far more simple. Having extraneous, and totally irrelevant, syntax is an f-ing disaster.

Geez as long as I am on this, another thing that sucks like the tar pit from hell, is Python (and R just as much - maybe even worse) scoping 'rules'. Yeah, if it weren't such a geyser of bugs, this would be a LOL statement.

You can quote with > followed by the quote

@metasoarous wow, you should write a blog post about this 😉

Yeah... I kinda started thinking that about half way through

I'm going to assume that since this is a more or less public forum that it would be fine for me to use folk's handle's or names (happy to link out to folk's twitter's as well), but if you've contributed to this thread please let me know if you'd like yours to be elided.

+1 for the blog post.

I don't have much to add here as @metasoarous really did sum things up really well. 1. I think containers are important because I don't have the time to dive into everyone's individual machine configuration and figure out why the thing does not work. If you come to me with an issue that seems to be hardware or machine config related then my first response will be to ask you to reproduce it in a container. You need containers anyway to move to production and conda+docker gives you a completely reproduceable pathway for a lot of things. We have put effort into making a container development flow as painless as possible like for instance we have a container that mounts the local directory and runs as your logged in user so it reads/writes files as your logged in user. 2. We do have support for zerocopy to/from neanderthal. In my talk at time around 14:38 (https://www.youtube.com/watch?v=vQPW16_jixs) I show zerocopy from neanderthal to numpy via the tech platform. I haven't kept up the neaderthal bindings mainly because it is tough to dig through the class hierarchy that Dragan uses but the bindings used in the talk are at https://github.com/techascent/tech.neanderthal. They are somewhat out of date.

Wonderful; Thanks @chris441!