Clojurians Log v2

By “sketching” I meant toying with an impl https://gist.github.com/cgrand/9e15363ef790471a9f6f1364d79cc6c8

hmm, could this be on a branch with Clojure 1.9?

alternatively, when Spark 2.0 support is merged, we hop to Clojure 1.8 and then we could use the spec backport for 1.8 https://github.com/tonsky/clojure-future-spec

this would allow to jump to 1.9 just by changing required namespaces

for what it's worth, I could live with 1.9 directly, but staying on 1.8 could make it more easy to link with currently built apps

well I guess the presence of spec could be sniffed so that it could be mainline

yup

actually, about helping with Spark 2.0, you could probably try out cgore/newer-spark to see if changes in it work :)

having said that, I'm wondering how to also support 1.x

macros 🙂

and reflection

http://spark.apache.org/docs/2.1.0/api/java/org/apache/spark/SparkContext.html#version()

though Scala version also matters

@alan.wischmeyer

So we just checked on that weirdness, it was a lack-of-coffee bug, I was misreading the output on one of the examples 😄

The examples all work on my end.

https://github.com/HCADatalab/powderkeg/pull/19

lack of coffee 🙂

PR 👍

was wondering if rebase with upstream is needed

I’m not sure how we’d go about adding tests for this, standing up a spark instance in the CI stuff somehow. That would be nice.

docker comes to mind

I’ve used Travis on some of the other stuff on github, it’s running in some sort of containers.

I’m just not sure how much you can make the container do.

this seems to use spark 2.1.0: https://github.com/gettyimages/docker-spark

I’ve just recently run Docker in CircleCI and plenty in inhouse Jenkins

consuming ready made good containers is neat, haven’t build one myself yet

I'm not too intimate with Spark master-slave networking, just remembering that when we had a stand-alone EC2 cluster, needed to open quite many ports, but thinking that the networking side is somewhat doable

now that I remember, I’ve written some unit tests in scala that just do

val conf: SparkConf = new SparkConf()
      .setMaster("local[2]")
      .setAppName(”app”)

around try (do the test) catch (sc.stop())

so using local mode for running

hmm so maybe just (deftest keg-works (try (keg/connect! “local[2]”) (is (= (keg…) expected)) (finally (keg/disconnect!)))

waving hands here 🙂

Yeah, not sure. I’d definitely like some real tests other than “it worked on my box”.

off topic (maybe) but haven’t looked into Onyx for real, but how do they update the apps running in different machines? Maybe ouroboros could be of use there too?

safety net at least 🙂

so maybe tests after this PR? 🙂

Sounds good to me.

but have to go get some sleep, small kids and late night hacking make for amplified sleep deprivation 🙂

Yeah, good night, you’re a few hours forward of me.

2:18pm here.

22:18 here

world wide around the clock duty 🙂