Clojurians Log v2

RELEASE dependencies are not officially supported (because they are pseudo versions and break the cache)

You can add -Sforce to force the cp to be recomputed

you mentioned Ripley, I wrote about that a year ago https://dev.solita.fi/2020/06/01/rethinking-the-frontend.html I took the approach of server side apps that are enhanced with live components… you can probably get even better patch granularity and smaller websocket traffic with full “vdom on the backend”

but I find that sending html fragments is good enough, tho I don’t have any real world benchmarks to give

and of course, I didn’t even try to solve the “no server” scenario

anybody know off-hand where Ring adds a session token (cookie GUID) to the response? I can't find it in wrap-session

doing some security auditing and need to confirm it's securely generated

it's called ring-session

after some more digging it seems to be in ring.middleware.session.memory, where a UUID/randomUUIDis added to Ring's memory store as a map key

I'm trying to track down a memory leak using VisualVM and have a few questions 1) Any hints what it might mean when there are a lot of java.lang.reflect.Methods allocated? 2) Other than that, the memory sampler is just showing me Doubles and PersistentVectors and Objects. Is there a good way to narrow this down further? I do know I'm creating a lot of these, I'm rendering a visual that adds a lot of points to several time series every second. 3) Any other VisualVM tips welcome

Seems likely you have a reflective call due to 1, but that itself is unlikely to be a memory leak

Tools like yourkit let you do things like “find paths to root” which can tell you reference path from an object back to how it’s being held, not sure if visualvm has that

Is there a reason a reflective call might not be detected with (set! *warn-on-reflection* true) ?

@jjttjj it can happen when the code isn't loaded at all if it's already AOT-ed for example

e.g.:

$ clj
Clojure 1.10.1
user=> (set! *warn-on-reflection* true)
true
user=> (require 'clojure.core :reload)
nil
user=&gt; (load-string (slurp (<http://clojure.java.io/resource|clojure.java.io/resource> "clojure/core.clj")))
Reflection warning, null:1717:7 - call to java.lang.IllegalArgumentException ctor can't be resolved.
Reflection warning, null:3631:22 - call to java.math.BigInteger ctor can't be resolved.
Reflection warning, null:3645:14 - call to java.math.BigInteger ctor can't be resolved.
Reflection warning, null:3659:14 - call to java.math.BigDecimal ctor can't be resolved.
...

there might be other things that create Method objects too, just seems like the most likely candidate

next.jdbc states this in their docs: > Note: you need to be careful when using stateful transducers, such as partition-by, when reducing over the result of plan. Since plan returns an IReduceInit, the resource management (around the ResultSet) only applies to the reduce operation: many stateful transducers have a completing function that will access elements of the result sequence -- and this will usually fail after the reduction has cleaned up the resources. This is an inherent problem with stateful transducers over resource-managing reductions with no good solution. https://github.com/seancorfield/next-jdbc/blob/v1.2.659/doc/getting-started.md Is there a more detailed explanation of this problem; and perhaps a minimal example that illustrates it?

Reproducing this is a bit tricky: you generally need a resource that is batched/buffered under the hood and to trigger the issue you need a large enough input set that it will cross that batch/buffer boundary — and the way that the completing function processes elements needs to actually trigger an error if performed on a resource that has already been “closed” (not all access will trigger an error — so it very much depends on the system that is fetching data from that resource). So it’s only going to happen with a resource that is lazily computed (with side effects) and can fail if you attempt to realize items after turning the resource “off”.

If you can reduce the whole thing first, and then run the stateful transducer over that result, you’re safe.

that doesn't really make sense to me -- is the "connection/resource" not closed as the cleanup procedure for the IReduceInit operation?

Yes it is: but the completing function only runs after the reduction is completed.

I see

So if the completing function does anything that would cause realization of elements it has in hand, it can fail.

got a concrete example?

As I said above, it’s hard to repro because of the conditions under which it fails.

(transduce (partition-all 500) conj [] (sql query.......)) wouldn't be an example of this, right?

Using plan, yes, maybe it would fail. On some databases, with a sufficiently large query result, and certain fetch size (batching) settings.

It’s because in the stateful transducers, you can have elements stored (in the state) that have not yet been fully-realized, then the reduction completes (and closes the resource), and then the completion function processes those elements and attempts to realize them.

is the "elements not fully-realized" some optimization that avoids touching the ResultSet?

fake maps

Yes.

seems like you have to ensure that anything in the state does not require an active connection

or you could defer the transducer calling into the completion of the rs until after it's processed? I mean, it's a stack and the transducer surrounds the rs and controls the completion call right?

maybe the rs is being closed on last iteration instead of on completion? (sorry, haven't looked at any code)

Well, transduce itself does the full reduction (which closes the resource at the end) and then it runs the completion after that:

([xform f init coll]
     (let [f (xform f)
           ret (if (instance? clojure.lang.IReduceInit coll)
                 (.reduce ^clojure.lang.IReduceInit coll f init)
                 (clojure.core.protocols/coll-reduce coll f init))]
       (f ret))))

is there a good resource for uberjar creation? Specifically how to deal with multiple licenses from deps. Our uberjar seems to have random dist/LICENSE and LICENSE and LICENSE.txt from deps. I want to be a good jvm citizen and ensure that a) our license it top level and in a canonical spot, and that all of our deps licenses are in an appropriate spot

it is because the "completing" is part of transduction, and not part of the IReduceInit interface, so the IReduceInit that is backed by the ResultSet closes things down and cleans things up when the "reduce" is done, because it can't clean up at a point later than that, and then transducer comes along and calls the completing arity

yeah, what @seancorfield said

i think racket has a reduce type function which offers a completion step which seems quite nice

@seancorfield IMHO it's because the funny maps returned are entangled with the live resources, not because of how the completing step works

wouldn't it would still be a problem if there was no such thing as completing?

there are N tools for this and they make N choices afaict

the problem with completing is a transducer with a partition-by step in the middle, everything after the partition-by might run as part of the completion

yeah. and i think its the defaults of lein uberjar that i need to be fighting. But i would like to find out the guidelines for what to do so I know a goal to shoot for

the proper place (imo N+1 opinion) is that META-INF is the explicit place marked out to put descriptive meta stuff orthogonal to the jar itself

depstar concatenates all the license files, as I recall, assuming they’re recognizable as such in the input files.

Maven will for example put stuff under META-INF/maven/<index by group/artifact>/stuff

so your pipeline that only pulls what it needs from the database as it needs and it produces a nice concrete result will still fail

that is imo a sane strategy

yeah i looked at depstar and uberjar. and they look for license files at certain spots and not root level ones

so i'm not sure if that's a missing part of theirs or if the ibm/icu jar is being naughty with a top level LICENSE file about unicode

yeah, there's really two questions here where do I find stuff in jars where do I put stuff in uberjars

https://github.com/seancorfield/depstar/issues/41 is the issue where I tried to deal with this — see my comment about “I couldn’t find much information on how Maven shade handles various license files and I examined the source and could not deduce what it does from that either” 😐

exactly. what should a jar look like, what should i do when creating my uberjars, what surgery should i do to correct misbehaved dep jars

because there is no standard, there will likely be conflicts, particularly at jar root

The practical answer is that there’s only so much you can do, and depstar mostly “does the right thing” (now).

yeah. i'm bundling all licenses up by inspecting all jars on the classpath, so i think i'm comfortable nuking all licenses in the resulting jar but ours and placing those rollups in there

I think it makes sense to grab stuff at jar META-INF and put it under uberjar META-INF/<indexing-strategy>/...

depstar does some of that too, yes.

“It’s complicated”.

i've got to run to a meeting. thank you both for your experiences

I have not finalized a strategy/process for this in tools.build yet, but I would like it to be a pluggable/directable thing

I’m working with Selmer and I have a hash map that I want to loop over the keys of in a particular order — I have a vector with the keys in order but it’s not clear to me from the Selmer docs whether it’s possible to do that sort of get-in style variable reference where a key in the middle of a path is a variable, rather than hardcoded? i.e., something like {{foo.bar.@k.quux}} where k is the for loop var — the equivalent of (get-in foo [:bar k :quux]) /cc @yogthos

racket has so many helpful functions, love that language

I don't know the answer for sure, but I assume it's best to do some preprocessing on the data before feeding it into the template