Clojurians Log v2

I just asked a question in #clojure that probably should have been here, but there's too much content to paste this time.

It's an issue I'm having with next-jdbc and streaming.

I'm dipping my toes in the 'next-jdbc' waters, and ran into a small issue. It's probably a gap in my knowledge of streaming, but here goes.... [9:01 AM] I have a query using 'plan' that then can be reduced 'into []' with no issue.. it's a huge result set, which is what I expect. [9:02 AM] But when I try to reduce that into something streamable, I'm getting an empty result.. [9:02 AM] So...

[9:03 AM] (ring-io/piped-input-stream
                                (fn [ostream]
                                  (let [^java.io.Writer w (<http://clojure.java.io/make-writer|clojure.java.io/make-writer> ostream {})]
                                    (get-big-data-streaming w))
                                  (.close ostream)))

[9:04 AM] (reduce (fn [^java.io.Writer w record]
              (let [m (datafiable-row record *db* {})]
                (.write w m)
                ))
            writer
            (jdbc/plan *db* sv))

Well, the copy/paste was easier than I thought it would be.

@jmckitrick the function f supplied to reduce has to return a value to be passed on, but I have a feeling this bit here

(fn [^java.io.Writer w record]
              (let [m (datafiable-row record *db* {})]
                (.write w m)
                ))

You're right, I missed that. But once I fixed it, I'm still getting the empty result....

(fn [^java.io.Writer w record]
              (let [m (datafiable-row record *db* {})]
                (.write w m))
              w)

I think you already know this, but if reduce is working fine in the non-stream version, then if you manually write something to the stream, does it work as intended?

Well, when I output a debugging line after the .write it is apparently only reached once.

@jmckitrick your original call to .close is redundant fwiw, that's done automatically.

@jmckitrick What does datafiable-row return? The fn name makes it sound like a map or something, but it should be a string/char/int

I think that's my issue. I wrapped it in 'str' in the call to '.write' and that seems to have done the trick....

@jmckitrick consider setting up https://stuartsierra.com/2015/05/27/clojure-uncaught-exceptions as it will help reveal these errors

Ah, great! Thanks@

Actually, in this case, I'm lying. ring-io uses future for it's threads, that means they will hold onto their exception until deref'd. A little annoying.

https://docs.oracle.com/javase/7/docs/api/java/util/concurrent/Future.html#get() docs this behaviour. You'll have to do your own try/catch to make sure you have visibility into these fns, or rewrite the piped-input-stream to just use a threadpool instead.