@kenny could you post your project.clj, looks like kryo version issue
@viesti Here are my deps:
[org.clojure/clojure "1.8.0"]
[hcadatalab/powderkeg "0.5.1"]
[org.apache.spark/spark-core_2.11 "2.1.1"]
[org.apache.spark/spark-streaming_2.11 "2.1.1"]
I have also tried with these deps:
[org.clojure/clojure "1.8.0"]
[hcadatalab/powderkeg "0.5.1"]
[org.apache.spark/spark-core_2.11 "2.1.0"]
[org.apache.spark/spark-streaming_2.11 "2.1.0"]
and running the master and slave with 2.1.0 as well.hmm, we have a problem with version of kryo used
@kenny as a workaround, could you add [com.esotericsoftware/kryo-shaded "4.0.0"]
explicitly
@viesti That did it! I tried updating kyro myself before but I must've used the wrong artifact or version.
so I remember that powderkeg/spark 1.5 works with kryo 3.0.3
powderkeg/spark 2.x works with kryo 4.0.0
either we ship two jars (powderkeg-kryo3/powderkeg-kryo4) or instruct to add direct dependency
shipping both makes building a classpath management harder no?
and one still has to add dependency with different name actually 🙂
hmm