powderkeg

viesti 2017-05-09T03:38:07.351430Z

@kenny could you post your project.clj, looks like kryo version issue

kenny 2017-05-09T03:41:16.365824Z

@viesti Here are my deps:

[org.clojure/clojure "1.8.0"]
[hcadatalab/powderkeg "0.5.1"]
[org.apache.spark/spark-core_2.11 "2.1.1"]
[org.apache.spark/spark-streaming_2.11 "2.1.1"]

kenny 2017-05-09T03:43:02.374045Z

I have also tried with these deps:

[org.clojure/clojure "1.8.0"]
[hcadatalab/powderkeg "0.5.1"]
[org.apache.spark/spark-core_2.11 "2.1.0"]
[org.apache.spark/spark-streaming_2.11 "2.1.0"]
and running the master and slave with 2.1.0 as well.

viesti 2017-05-09T05:48:17.957464Z

hmm, we have a problem with version of kryo used

viesti 2017-05-09T05:48:47.959724Z

@kenny as a workaround, could you add [com.esotericsoftware/kryo-shaded "4.0.0"] explicitly

kenny 2017-05-09T06:16:48.106699Z

@viesti That did it! I tried updating kyro myself before but I must've used the wrong artifact or version.

viesti 2017-05-09T09:22:06.700726Z

so I remember that powderkeg/spark 1.5 works with kryo 3.0.3

viesti 2017-05-09T09:22:21.703127Z

powderkeg/spark 2.x works with kryo 4.0.0

viesti 2017-05-09T09:22:52.708523Z

either we ship two jars (powderkeg-kryo3/powderkeg-kryo4) or instruct to add direct dependency

cgrand 2017-05-09T11:12:03.735318Z

shipping both makes building a classpath management harder no?

viesti 2017-05-09T11:38:48.961314Z

and one still has to add dependency with different name actually 🙂

viesti 2017-05-09T11:38:50.961523Z

hmm