data-science

Data science, data analysis, and machine learning in Clojure https://scicloj.github.io/pages/chat_streams/ for additional discussions
2020-08-16T00:32:48.470700Z

Hi @smith.adriane! I'd highly recommend Oz, but as the author, am highly biased 🙂

🙂 1
2020-08-16T00:33:18.471500Z

Quil is really cool, and you have a lot of fine grained control with it if you want to do something wild.

2020-08-16T00:33:49.472200Z

And manually creating SVGs can also be a good way to go if you want something really custom, though may not scale if you have a lot of data.

2020-08-16T00:34:36.473400Z

Oz uses Vega-Lite & Vega for visualization, and there are a number of other tools in the ecosystem that are going that way now (Saite, Hanami, Gorilla, Pink-Gorilla, darkstar, etc). So you can't go wrong with VL & V specifically.

2020-08-16T00:35:42.475600Z

VL & V have very "data-driven" declarative approaches to specifying a data visualization as a mapping from data to visual aesthetics (like ggplot, Grammar of Graphics, etc), which I think fits really well with the Clojure model of programming (philosophically).

phronmophobic 2020-08-16T00:35:47.475700Z

how much does oz know about the vega data viz model? how crazy of an idea would it be to implement a graphics backend for the vega descriptions in clojure?

2020-08-16T00:36:08.476300Z

I think some folks have looked at that with clj-plot

2020-08-16T00:36:21.476600Z

Which is javafx based IIRC

2020-08-16T00:36:38.477Z

It's a little crazy, but depends on how much of the spec you want to cover

2020-08-16T00:36:52.477400Z

I've been wanting to build a simple ascii render for scatterplots and such

2020-08-16T00:37:39.478100Z

The other thing you get from Vega / Vega-Lite is a grammar of interaction which allows you to build interactivity into your graphs in a very simple but powerful way

2020-08-16T00:38:20.478900Z

As far as Oz itself, it goes beyond just data visualization and is really a complete scientific document processing toolkit

2020-08-16T00:38:41.479500Z

So if you want to render markdown or hiccup with embedded vega to html or (soon) pdf, you can do that too

2020-08-16T00:38:49.479800Z

Has live-view mode that will watch a file for changes and update as you save

2020-08-16T00:39:27.481Z

And doing that with clj files even tries to be smart about how it evaluates forms, so you can use it as a notebook-like environment from the comfort of your vim editor

2020-08-16T00:40:36.482700Z

Also reagent components if you want to build a full-on app using VL/V

2020-08-16T00:41:11.484Z

Stepping away now, but feel free to ask more questions, and I'll circle back around if other folks haven't answered already

phronmophobic 2020-08-16T00:41:27.484300Z

my other side project is a platform agnostic graphics and ui library (linux, mac, and webgl support and experimental support for html, terminal, java2d). ideally, I'd like a library that helps with the data analytics side and if there's a model for visualizations that's already well designed, then I can just start implementing the parts of the data viz model that I care about.

2020-08-16T01:27:24.485400Z

Nice!

2020-08-16T01:27:43.485800Z

I'll not that there is a webgl vega renderer already: https://github.com/vega/vega-webgl-renderer

👍 2
2020-08-16T01:27:53.486100Z

Would be cool to integrate that in oz

jsa-aerial 2020-08-16T20:23:31.488100Z

@smith.adriane I think what you are looking for is @tsulej’s CljPlot: https://github.com/generateme/cljplot

👍 1
👀 1
genmeblog 2020-08-17T09:28:17.004100Z

I'm happy to answer your questions

phronmophobic 2020-08-17T14:41:46.004900Z

@tsulej , are the plots themselves reified?

genmeblog 2020-08-17T15:08:42.005100Z

@smith.adriane can you elaborate a little bit what do you mean by reified plots?

genmeblog 2020-08-17T15:13:19.005300Z

Generally plots are represented by nested maps consisting data, scales, aesthetics, configuration + some functions (this means that these are not pure data). Based on above you render chart on canvas as a separate step.

phronmophobic 2020-08-17T16:16:28.005700Z

basically just wondering if there is a data model for the charts/graphs/etc.

genmeblog 2020-08-17T19:48:53.006800Z

kind of, but rather generated and not simple to write by hand. I'm going to change it though.

phronmophobic 2020-08-17T19:57:28.007100Z

what I would be looking for is some sort of data model that does a good job describing different charts. it doesn't really matter if the descriptions are meant to be produced by hand.

phronmophobic 2020-08-17T19:58:54.007400Z

eg. • here's all the type charts • here's the data they can expect • here's all the different ways to customize them • maybe some specs for the chart descriptions and the acceptable input data

genmeblog 2020-08-17T20:00:14.007600Z

vega/vega-lite does it perfectly well

phronmophobic 2020-08-17T20:00:21.007800Z

It matters less to me if there are actually implementations of drawing all of the charts. I've been working on this graphics library for a while, https://github.com/phronmophobic/membrane and would love to just implement the graphics for a given chart spec library

genmeblog 2020-08-17T20:01:54.008200Z

There was discussion to introduce vega/vega-lite spec to cljplot, but first I need to make rendering backend closer to this (if it's possible, I don't know)

phronmophobic 2020-08-17T20:02:29.008400Z

vega* requires a whole ecosystem of non jvm stuff and it's unclear how extensible it is from clojure land

genmeblog 2020-08-17T20:03:00.008600Z

but vega contains well defined specification which can be adopted to other backends

phronmophobic 2020-08-17T20:04:41.008800Z

how many backends do they have?

genmeblog 2020-08-17T20:05:01.009Z

I focused mostly on rendering at the beginning and stopped there since I hit the dead end with data manipulation and data dimensionality representation (called aesthetics in ggplot2 or enconding in vega*)

genmeblog 2020-08-17T20:05:12.009200Z

they have one backend 🙂

genmeblog 2020-08-17T20:06:13.009400Z

Vega advertises itself as a grammar or language, the rendering engine comes at the end of the desctiption

phronmophobic 2020-08-17T20:06:22.009600Z

i actually have a copy of the grammar of graphics that I haven't read yet

genmeblog 2020-08-17T20:08:32.009800Z

me too

phronmophobic 2020-08-17T20:08:34.010Z

yea, it definitely seems like a good starting point, but I am wary of large specifications that only have one implementation. it makes it very easy for the spec and the implementation to start to bleed into one another

genmeblog 2020-08-17T20:08:48.010400Z

true

genmeblog 2020-08-17T20:09:17.010600Z

and it's huge

genmeblog 2020-08-17T20:10:23.010800Z

personally I prefer ggplot2 approach but nothing stops one to build another, grammar level on the top of rendering library to store/exchange/reuse chart description

phronmophobic 2020-08-17T20:11:33.011200Z

hmm, https://github.com/rstudio/cheatsheets/blob/master/data-visualization-2.1.pdf seems cool

genmeblog 2020-08-17T20:12:40.011400Z

it is, but it's highly declarative: map data and add this, add that etc.

phronmophobic 2020-08-17T20:13:09.011600Z

declarative sounds great. in clojure, that's just data

jsa-aerial 2020-08-17T20:14:55.011800Z

The key thing for Vega and Vega-Lite is these things are interactive - they are not simple static things lke ggplot2. If all you want is ggplot2 like stuff, then a good way now may be to just use ggplot2 via Clojisr

phronmophobic 2020-08-17T20:15:42.012Z

what's the support like for adding interactivity to a vega-lite plot via clojure?

jsa-aerial 2020-08-17T20:15:56.012200Z

It comes for free

phronmophobic 2020-08-17T20:16:27.012400Z

like if I wanted to add a custom interaction?

genmeblog 2020-08-17T20:17:07.012600Z

there is something like signals: https://vega.github.io/vega/docs/signals/

jsa-aerial 2020-08-17T20:17:15.012900Z

Depends on what it is. There is a lot of support for that in Vega-Lite

genmeblog 2020-08-17T20:17:42.013100Z

(and event streams)

jsa-aerial 2020-08-17T20:17:45.013300Z

And in Vega as well (since Vega-Lite compiles down to Vega)

jsa-aerial 2020-08-17T20:19:06.013500Z

It doesn't cover all aspects - for example, various external controls that depend on recomputing the input data are not typically covered

phronmophobic 2020-08-17T20:19:52.013700Z

I will have to take some time to digest all this

genmeblog 2020-08-17T20:20:29.013900Z

if you want to see reimplementation of vega-lite or lattice (R) examples in cljplot- take a look here https://github.com/generateme/cljplot/tree/master/sketches - I made it to test my concepts (even with kind of interactivity through java2d)

👍 1
phronmophobic 2020-08-17T20:20:56.014200Z

yea, I've been poking around with that

genmeblog 2020-08-17T20:21:03.014400Z

but as I said cljplot shouts for deep refactoring

jsa-aerial 2020-08-17T20:21:12.014600Z

That sort of 'interactive' control requires some form of reactive components coupled to the data streams TBH, it is really difficult to 'compete' with what the IDL folks are doing. They have a lot of resources and a boatload of talent

genmeblog 2020-08-17T20:22:29.014800Z

this is deep true

phronmophobic 2020-08-17T20:24:51.015Z

that's true, but I feel like I would also be giving up a lot of control. Like I don't think it would have been difficult (not impossible) to make this project using vega-lite, https://blog.phronemophobic.com/treemap/treemaps-are-awesome.html

jsa-aerial 2020-08-17T20:28:55.015200Z

Any time you opt for some form of 'declarative' coding, you give up control. Vega sits on top of D3 as its primary backend, It gives up the total control you have in D3, but you get back a great deal in the form of consistency simplicity (I know - Vega does not seem simple, but compared to D3? yeah)

jsa-aerial 2020-08-17T20:30:12.015400Z

Vega-Lite also give up some control as it is layered on top of Vega and targets simplifying a lot of the nuts and bolts of generating a simple chart (kind of like ggplot2 in that regard)

phronmophobic 2020-08-17T20:32:26.015600Z

I think you can do what looks like declarative coding in clojure that is really just manipulating data. my experience is that you get a lot of the benefits of declarative coding, but you still retain the flexibility

jsa-aerial 2020-08-17T20:32:38.015800Z

I don't think tree maps are available in Vega-Lite (yet), but they are definitely in Vega: https://vega.github.io/vega/examples/treemap/

jsa-aerial 2020-08-17T20:33:49.016100Z

Only up to a point. Otherwise you wouldn't 'need' things like core.logic

phronmophobic 2020-08-17T20:34:32.016300Z

the treemap project wasn't about drawing a traditional treemap, it was about augmenting the treemaps in different ways, including interactivity

jsa-aerial 2020-08-17T20:34:39.016500Z

Functional stuff often looks declarative but it really isn't. There is still an explicit flow

jsa-aerial 2020-08-17T20:35:02.016700Z

well, you have interactivity for free in vega

phronmophobic 2020-08-17T20:37:09.016900Z

we might be talking about different things with respective to writing declarative code in clojure. what I'm referring to is when you're working with just data. data has no explicit flow and hopefully, the what is decoupled from how

phronmophobic 2020-08-16T20:24:25.488600Z

neat! will check it out!

jsa-aerial 2020-08-16T20:30:16.492600Z

With respect to "platform agnostic graphics and ui" you may also want to take a look at Saite https://github.com/jsa-aerial/saite which is a "turnkey" data exploration and visualization system. Just need Java8 and it works on linux, mac, and windows. But it is Vega/Vega-Lite based. A new release featuring Shiny like dash construction (with R if you like as well) is coming. I am giving a demo of this new stuff at the upcoming SclCloj virtual meeting: https://twitter.com/scicloj/status/1291845872884625408

jsa-aerial 2020-08-16T20:36:17.496700Z

One last thing to know about would be hanami https://github.com/jsa-aerial/hanami (and bare bones example: https://github.com/jsa-aerial/bbhanami) which is 'just' a library - does not provide complete document like capabilities that Saite and Oz do. It is agnostic on that and only provides a the 'kit' components and capabilities for you to build your own vis apps.

phronmophobic 2020-08-16T20:42:45.497600Z

really great options!