Clojurians Log v2
Clojure programming
Channels
Apps
flambo
sorenmacbeth 2016-09-01T01:07:58.000009Z
2016-09-01T10:38:02.000010Z
Hi guys, I have a problem
2016-09-01T10:38:23.000011Z
When using mllib, I run into incmpatibilities with kryo serialization
2016-09-01T10:38:43.000012Z
Seems it’s a known bug, and I need to be going with spark 2.X
2016-09-01T10:39:11.000013Z
problem is, When I reaquire spark 2.X, it tells me that it can’t find logger class...
2016-09-01T10:39:14.000014Z
any idea ?
2016-09-01T10:43:49.000015Z
here is the relevant trace:
2016-09-01T10:43:51.000016Z
com.esotericsoftware.kryo.KryoException: java.lang.IllegalArgumentException: Can not set final scala.collection.mutable.ListBuffer field org.apache.spark.mllib.fpm.FPTree$Summary.nodes to scala.collection.mutable.ArrayBuffer Serialization trace: nodes (org.apache.spark.mllib.fpm.FPTree$Summary) org$apache$spark$mllib$fpm$FPTree$$summaries (org.apache.spark.mllib.fpm.FPTree) at com.esotericsoftware.kryo.serializers.FieldSerializer$Obje
2016-09-01T10:44:02.000017Z
seems it does not like fptree
2016-09-01T10:52:45.000018Z
seems to have been better if I register the missing classes at kryo’s level :
2016-09-01T10:53:03.000019Z
… (conf/set "spark.kryo.classesToRegister" "scala.collection.mutable.ArrayBuffer,scala.collection.mutable.ListBuffer")
2016-09-01T10:53:39.000020Z
But still waiting for it to finish ...
2016-09-01T11:01:34.000021Z
Yes.
2016-09-01T11:01:40.000022Z
This definitely worked !