Clojurians Log v2

Hi there! Here is a blog about some of the stuff re Clojure and data science that @daslu , @konrad.kuehne and myself discussed in Helsinki after ClojuTRE, kindly summarised by Daniel. We would love to hear your thoughts about this! https://scicloj.github.io/posts/2019-10-18-data-wishes/

@neo2551 I have not tried pyodide. I would like to see the jvm implement wasm such that it performed as it should.

Maybe we could leverage one of the runtimes listed here: https://github.com/appcypher/awesome-wasm-runtimes https://github.com/fastly/lucet.

With Rust it is relatively easy to add a C-layer that we can use via JNA. Lucet may already have that layer in fact as it is designed to be embedded.

Two things: 1. New version of https://github.com/techascent/tvm-clj/

2. TVM now has an IR layer that supports autodifferentiation: https://docs.tvm.ai/dev/relay_intro.html

New version of TVM works with latest tech.datatype, tech.ml.dataset, etc.

What is TVM?

Never mind :)

It is my favorite toy 🙂.

I just want to have pandas in CLJS

I can’t find any library that provides the data alignment functionality

https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.align.html

Is that the align function you want?

For timeseries: you want to concat two ts, but they might not have the same time stamps. So you would need to make an outer join

Yeah that one.

When you have a sorted index, I can’t solve the problem with an efficient algorithm.

Do you mean you 'can' solve the problem with sorted index?

These functionalities I am missing https://jakevdp.github.io/PythonDataScienceHandbook/03.11-working-with-time-series.html

Well, what I am doing now is I am representing my timeseries as sorted-map

And have date->vector map

I get super fast slicing thanks to subseq

But this representation is inefficient for computation (so I would need to go for date->index and manage my slicing accordingly and keep the data as a matrix).

The biggest trick is whenever I have two timeseries that might potentially have different time index

But some of the times align and some don't

Then I would need to merge them (the issue with merge is I can’t do outer join)

Exactly

IMO, the biggest feature of pandas is to solve this problem extremely efficiently.

And all the handling of time as well.

tablesaw does not even care about that.

Tablesaw (and tech.ml.dataset) does not have the concept of an index across the table. You could create brand new tables that looked the same as the two tables in the documentation though, they would not share backing store data.

Views are doable in tech.ml.dataset but I would first get the functionality and tests working correctly for the 'align' function and then worry about views when someone runs out of RAM.

Agree.

But then I would need it on CLJS xD

I am at the point of thinking to use tensorflow-js for performing my linear algebra operations.

You could rely on WebGL whenever available xD

lol, you are better off with tensorflow-js than waiting for me to port the tech platform to js. Why CLJS? Just for kicks?

My company forbid me to use Clojure (to import jar file more precisely) whereas they have now restrictions for JS files

haha 🙂

So I code my tools outside the company networks and download the JS files from Github

Plus my velocity in developing UI has been amazing the last year (I am UI newbee)

So they let me play with it xD

Actually, they have been so amazed that we probably are going to stick with ClojureScript for any official web interface

I hope I can hack Clojure in the backend soon. I want to play with Neanderthal xD

I also suspect that tfjs WebGL is faster than most of our internal data science tools (we use Matlab and R)

Maybe. The internal stuff should use system blas libraries so it depends on which system blas they have installed. They could have mkl installed or something like that.

Yeah, that would be tricky to beat

That being said I could mimic tech.ml.dataset with tfjs

You have my attention now :) Guess I have to check Saite now

One of my best shot would still be to force them use GraalVM for the R<->Java interop and hope I can sneak all my Clojure dependencies as well