Clojurians Log v2
data-science
maybe a github repo that folks could commit/push links to current ds projects might be in order? or any other mechanism. A long, long, long time ago clojure toolbox was just this kind of thing. I'm not sure if it still is updated (https://www.clojure-toolbox.com/)
Oh hey, I'm building such a thing I guess, it's work in progress
I sometimes have very wide data structures that are clunky to look at if you just pprint it. This is sort of the next thing I'm working on — adding support for custom visualizations, such as tables with limited height, that make it much more convenient for looking at the current level, for example
for what I’m interested in. there are two aspects: 1. exploration - given some data, can I navigate through it and look at all its parts 2. summary - given some data, what can I know about it without looking at every single detail it has to offer I think exploration is important, but more straightforward. The problem of summarizing medium size, heterogeneous data is much more interesting to me. We have lots of ways to summarize large sequences of numbers (mean, median, mode, histograms, etc). We have fewer tools for summarizing something like a json blob that comes from an API call.
@smith.adriane I've been using malli.provider/provide for exploring JSON structure. See #malli
Caveat: you'll have to figure out where your sequences are, then you can run provide on those.
@nickstares0 - perhaps you're interested in this 🙂
@smith.adriane your points (1) and (2) sound like automl to me, e.g. https://arxiv.org/pdf/1810.13306.pdf
@aaelony, I’m not sure I follow. It looks like automl is meant to take people out of the loop rather than a tool for people to quickly form an intuition about some data
depends on the intuition, I suppose.
using something like automl sounds cool. I just have zero experience with machine learning so I’m not sure how I would use it
it's an active topic with aspirational goals riffing off of exploration and summary as you had mentioned (esp for predictive models)
it’s still being updated. I had a repo added within the last month
I don't see DS categories there though. Perhaps that's one idea
e.g. Python Integration lists pickler but no libpython-clj
those seem like good additions
I don't see http://tech.ml.dataset either
I was able to get my library added by making a pull request, https://github.com/weavejester/clojure-toolbox.com/pulls
perhaps the word can get out to the lib creators to register there if desired
I think anyone can make a pull request
Oz, Saite, etc..
Oz is under data exploration
I did put together this a long time ago; Idea was supposed to be the same as far as PRing libraries: https://github.com/metasoarous/clojure-datascience
well, it used to be the case that a few libs fell under >1 category
Yeah, it would be nice to have a little database of these things, which can be submitted and approved dynamically
maybe that's cleaned up, but still possible to have multiple uses
Searchable UI; blahblah
yep, exactly that, metasoarous
BERT enabled search would be awesome 😉
@aaelony @metasoarous @smith.adriane
Hi. :)
We maintain some relevant lists at the scicloj website.
https://github.com/scicloj/scicloj/blob/master/resources/templates/md/pages/libraries.md
https://github.com/scicloj/scicloj/blob/master/resources/templates/md/pages/reading.md
https://github.com/scicloj/scicloj/blob/master/resources/templates/md/pages/chat_streams.md
If anyone wants to have push permissions -- please tell.
Our current method is that people can push changes to a draft branch, and one person (the "editor") merges them to master and tidies up.
Any thoughts?
@aaelony @metasoarous Searchable UI is a great idea. We have been thinking for some time about migrating the website from Cryogen to something hiccup-driven such as Oz. Then (I think) it will be more fun to create some interactive views.
Yes! That would be awesome. I've wanted to do that; Build viz tools for library discovery/evaluation.
On a related thread, @teodorlu is exploring some ideas of knowledge organization using Roam, that should be exportable to some comfortable data format.
🙂
Roam seems to be everywhere these days doesn't it...
Your tweet about it was inspiring.
Regarding the scicloj website -- it gets too little attention these days. If anyone wants to take over and make it fresh and beautiful, that would be more than welcome of course. 🙃
fyi, the links to vega and vega lite in libraries.md are broken
thanks!
is there a library for exploring a giant edn blob? like if you want to just open up a visualizer for your app state or a response from an API?
curious too!
i guess giant is the wrong adjective. anyway, there’s lots of tools for wide data sets (eg. this db table a million rows), but I’m interested in a generic data viewer for data sets that are 5-20 levels deep and maybe up to 100s of levels wide under certain branches
REBL is in the same concept space, but I’m wondering if there are others
Maybe @vlaaad may comment about Reveal, as a REBL-related UI. https://github.com/vlaaad/reveal
oh yea, i’ve been meaning to look into that