Clojurians Log v2

running against a standalone spark 1.5.2 taken off from a downloaded package even gives trouble, but chugging along, since this doesn’t block others 🙂

quite a beast this

what should I run to experience the error?

downloaded http://d3kbcqa49mib13.cloudfront.net/spark-1.5.2-bin-hadoop2.6.tgz and ran ./sbin/start-all.sh. then on feature/circleci branch of my fork I run lein with-profile spark1.5 repl. In the repl, in user namespace I run (require '[powderkeg.core :as keg]) followed by (keg/connect! “<spark://taskunauris.tiuhtiwlan.net:7077”>) (using hostname of my laptop), then I do (into [] (keg/rdd (range 10))) but get

17/03/14 14:33:27 WARN TaskSetManager: Lost task 0.0 in stage 0.0 (TID 0, 172.16.0.71): java.io.EOFException
	at org.apache.spark.serializer.KryoDeserializationStream.readObject(KryoSerializer.scala:196)
	at org.apache.spark.broadcast.TorrentBroadcast$.unBlockifyObject(TorrentBroadcast.scala:217)
	at org.apache.spark.broadcast.TorrentBroadcast$$anonfun$readBroadcastBlock$1.apply(TorrentBroadcast.scala:178)
	at org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:1175)
	at org.apache.spark.broadcast.TorrentBroadcast.readBroadcastBlock(TorrentBroadcast.scala:165)
	at org.apache.spark.broadcast.TorrentBroadcast._value$lzycompute(TorrentBroadcast.scala:64)
	at org.apache.spark.broadcast.TorrentBroadcast._value(TorrentBroadcast.scala:64)
	at org.apache.spark.broadcast.TorrentBroadcast.getValue(TorrentBroadcast.scala:88)
	at org.apache.spark.broadcast.Broadcast.value(Broadcast.scala:70)
	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:62)
	at org.apache.spark.scheduler.Task.run(Task.scala:88)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

It’s the kryo version

if you change it back to 3.0.3 it works

ah, so older kryo with 1.5.2

will try a bit later

hum, moved kryo-shaded spark2-deps/spark1.5-deps, 3.0.3 on the latter, still same

show me your project.clj

(I just replaced 4.0.0 by 3.0.3, didn’t move things around)

(defproject hcadatalab/powderkeg "0.4.7"
  :description "Live-coding Spark clusters!"
  :url "<https://github.com/HCADatalab/powderkeg>"
  :license {:name "Eclipse Public License"
            :url "<http://www.eclipse.org/legal/epl-v10.html>"}

  :aliases {"test-all" ["with-profile" "spark1.5:spark2" "test"]}
  :test-selectors {:default (complement :integration)
                   :integration :integration}
  :profiles {:default [:spark2]
             :spark2 [:leiningen/default :spark2-deps]
             :spark1.5 [:leiningen/default :spark1.5-deps]
             :spark2-deps ^{:pom-scope :provided}
             {:dependencies [[org.apache.spark/spark-core_2.11 "2.1.0"]
                             [org.apache.spark/spark-streaming_2.11 "2.1.0"]
                             [com.esotericsoftware/kryo-shaded "4.0.0"]]}
             :spark1.5-deps ^{:pom-scope :provided}
             {:dependencies [[org.apache.spark/spark-core_2.10 "1.5.2"]
                             [org.apache.spark/spark-streaming_2.10 "1.5.2"]
                             [com.esotericsoftware/kryo-shaded "3.0.3"]]}}
  :dependencies [[org.clojure/clojure "1.8.0"]
                 [net.cgrand/xforms "0.5.1"]
                 [com.twitter/carbonite "1.4.0"
                  :exclusions [com.twitter/chill-java]]
                 [com.cemerick/pomegranate "0.3.0"]]
  :aot [powderkeg.repl]
  :java-source-paths ["src/main/java"]
  :source-paths [”src/main/clojure”])

hum

apropo, :target-path "target/%s/"

comes to my mind

since we have a few profiles with different deps

lein clean helped

should have tried putting “target/%s/” first into :target-path though

lein help sample says:

;; All generated files will be placed in :target-path. In order to avoid
  ;; cross-profile contamination (for instance, uberjar classes interfering
  ;; with development), it's recommended to include %s in in your custom
  ;; :target-path, which will splice in names of the currently active profiles.
  :target-path “target/%s/”

:spark1.5-deps [^{:pom-scope :provided}
             {:dependencies [[org.apache.spark/spark-core_2.10 "1.5.2"]
                             [org.apache.spark/spark-streaming_2.10 "1.5.2"]]}
             {:dependencies [[com.esotericsoftware/kryo-shaded “3.0.3”]]}]

works

but now don’t we need to push 2 jars?

ah, true

It’s the provided scope that caused you some issues

ah, true, didn’t notice

hmm, so we have problem I guess

spark 1.5.2 works with kryo 3.0.3, spark 2 with kryo 4.0.0

on the docker side for 1.5.2 in my branch, need to figure out how to specify suitable hostname for the spark running inside docker, now I need to edit /etc/hosts to contain "127.0.0.1 master"

two builds then. keg-kryo3 and keg-kryo4

yep, tracing root cause and circumventing might be difficult

yei, first build to succeed that runs a simple (keg/rdd (range 10)) against spark 1.5 and spark 2, and runs lein test-all too

still thinking that docker is troublesome. To be able to connect to spark 1.5 started with docker using default bridged networking, I had to resolve “master” to 127.0.0.1 in /etc/hosts. Funny thing is that using <spark://localhost:7077> works for spark 2 running in docker.

another option would be require people to download spark binary distribution and make the test call sbin/start-all.sh (sbin/stop-all.sh for stopping)

for CircleCI, the downloaded/unpacked binary could be stored in a cache (https://circleci.com/docs/1.0/how-cache-works/#nav-button) for later builds

and seems that even docker caching is troublesome in CircleCI… https://circleci.com/docs/1.0/docker/#caching-docker-layers

then again, just being able to run lein with-profile spark1.5 test :only powderkeg.integration-test/rdd-spark-1.5.2 is neat too, if one has docker