data-science

Data science, data analysis, and machine learning in Clojure https://scicloj.github.io/pages/chat_streams/ for additional discussions
agigao 2020-02-05T16:32:39.028600Z

Hey there, I’m kinda having hard time using kixi.stats core functions median, summary etc without transducers and can some help me figure out - why?

...
(:require [kixi.stats.core :refer [median]])

(median (repeatedly 100 rand))
REPL:
1. Unhandled java.lang.ClassCastException
   class clojure.lang.LazySeq cannot be cast to class
   com.tdunning.math.stats.TDigest (clojure.lang.LazySeq and
   com.tdunning.math.stats.TDigest are in unnamed module of loader 'app')

2020-02-05T17:43:41.029800Z

@chokheli I think you do need to use them with transducers. This works:

(transduce (map identity) kixi.stats.core/median (repeatedly 100 rand))

2020-02-05T17:45:10.030200Z

and if you really hate transducers this works (but is ugly)

(kixi.stats.core/median (reduce kixi.stats.core/median (kixi.stats.core/median) (repeatedly 100 rand)))

2020-02-05T17:45:41.030600Z

as you need an inital value and the completion step to get the median out rather than the t-digest

bocaj 2020-02-05T23:25:38.031500Z

I'm hunting for a standard way or a library to select out of (e.g Oracle) into avro, and avro into another db (e.g. Sql Server).

bocaj 2020-02-05T23:27:38.031600Z

posting here, b/c of the avro case. Maybe others would find a library like this useful? Or, are there other reliable intermediary formats that keep types around?

jsa-aerial 2020-02-05T23:41:36.031800Z

Spark SQL??