Clojurians Log v2

hello! today i notice a weird interaction between datomic client api and djblue/portal (https://github.com/djblue/portal): when this last one is not on the classpath, then i can obtain a connection to my (local, datomic pro) database; when portal is on the classpath, connecting to the database fails with the following error:

Exception in thread "async-dispatch-1" java.lang.RuntimeException: java.lang.NoClassDefFoundError: org/msgpack/MessagePack
	at com.cognitect.transit.TransitFactory.writer(TransitFactory.java:104)
	at cognitect.transit$writer.invokeStatic(transit.clj:161)
	at cognitect.transit$writer.invoke(transit.clj:139)
	at <http://datomic.client.impl.shared.io|datomic.client.impl.shared.io>$marshal.invokeStatic(io.clj:48)
	at <http://datomic.client.impl.shared.io|datomic.client.impl.shared.io>$marshal.invoke(io.clj:38)
	at <http://datomic.client.impl.shared.io|datomic.client.impl.shared.io>$client_req__GT_http_req.invokeStatic(io.clj:76)
	at <http://datomic.client.impl.shared.io|datomic.client.impl.shared.io>$client_req__GT_http_req.invoke(io.clj:73)
	at datomic.client.impl.shared.Client._async_op(shared.clj:398)
	at datomic.client.impl.shared.Client$fn__34578$state_machine__5717__auto____34593$fn__34595.invoke(shared.clj:423)
	at datomic.client.impl.shared.Client$fn__34578$state_machine__5717__auto____34593.invoke(shared.clj:422)
	at clojure.core.async.impl.ioc_macros$run_state_machine.invokeStatic(ioc_macros.clj:973)
	at clojure.core.async.impl.ioc_macros$run_state_machine.invoke(ioc_macros.clj:972)
	at clojure.core.async.impl.ioc_macros$run_state_machine_wrapped.invokeStatic(ioc_macros.clj:977)
	at clojure.core.async.impl.ioc_macros$run_state_machine_wrapped.invoke(ioc_macros.clj:975)
	at datomic.client.impl.shared.Client$fn__34578.invoke(shared.clj:422)
	at clojure.lang.AFn.run(AFn.java:22)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at clojure.core.async.impl.concurrent$counted_thread_factory$reify__469$fn__470.invoke(concurrent.clj:29)
	at clojure.lang.AFn.run(AFn.java:22)
	at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.lang.NoClassDefFoundError: org/msgpack/MessagePack
	at com.cognitect.transit.impl.WriterFactory.getMsgpackInstance(WriterFactory.java:77)
	at com.cognitect.transit.TransitFactory.writer(TransitFactory.java:95)
	... 20 more
Caused by: java.lang.ClassNotFoundException: org.msgpack.MessagePack
	at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:581)
	at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:521)
	... 22 more

i noticed that portal has com.cognitect/- transit-cljs, transit-js, transit-clj, and transit-java as dependencies. why would these dependencies interfere with datomic obtaining a connection?

(there are other dependencies but i don’t believe they have anything to do with the case: cheshire 5.10.0 and http-kit 2.5.0).

a) what are you using to make the classpath b) if clj, what version? (prob best to upgrade to latest if you haven’t) c) please provide the set of project deps that repro this

i am using clojure tools (clojure 1.10.1) and i open a repl using cider-jack-in-clj. then i start my system via integrant.repl.

this is my deps.edn. when connecting to the repl, i add -A:dev to the clj command.

Portal requires transit but excludes msgpack

and datomic requires msgpack?

if this is the case, is it a classpath resolution conflict/problem, i.e. msgpack should be in the final classpath because datomic requires it, even if portal excludes it?

what version of the clojure tools? clj -Sdescribe

there were issues with this kind of scenario that were fixed a few versions ago

release info here https://clojure.org/releases/tools - latest is 1.10.1.727

The client apparently requires msgpack but doesn’t depend on it directly, expecting it via transit

Yes

Portal uses transit but not the msgpack encoding option, so it doesn’t want to bring it in

Datomic client uses transit with msgpack encoding but doesn’t require it directly

I wonder what maven would compute in this case

I'm stepping away till this afternoon but I'd be happy to look at this in depth when I get back. My recommendation would be to move to latest clj if you haven't already as there have been fixes in this area.

Hello @joshkh, @jaret! Thank you for explanation. In the end we decided to go with event bridge or sns topic directly. It is problematic to route events from cloudwatch logs to aws lambda functions. As the only option there is subscription filter (max 2 per log group). Maybe I missing something, but haven't found solution where I am able to get events submitted using cast/event to lambda functions.

thanks, @alexmiller and @favila. i’ll update clj tools if mine is not up-to-date.

https://clojurians.slack.com/archives/C03RZMDSH/p1603978063215500?thread_ts=1603974900.204700&cid=C03RZMDSH mine was 1.10.1.716

(Sorry if I’m confusing, slack must have barfed on my messages because they’re all out of order and 20 min late)

i don't know if this is useful to you, but we cast/alert exceptions to CloudWatch, and then use a CLJS Lambda to send them to Slack to torture our developers. cast/events are not really any different, except for maybe the frequency at which they appear, but i think log streams are batched. (sorry for the lack of a proper README, it was a hobby project) https://github.com/joshkh/datomic-alerts-to-slack > It is problematic to route events from cloudwatch logs to aws lambda functions. As the only option there is subscription filter (max 2 per log group). for application level logs and alerts, we tend to the use a common message throughout the application (e.g. {:msg "<app-name>-application-log"} ) , and then we attach other keys such as :env and :query-group . this provides us different levels of filtering while keeping our logs all in one place

but if SNS works for you then go for it! 🙂 i just prefer CloudWatch because cast serialises Clojure to JSON very well, and being able to add arbitrary keys at any level is useful for filtering

@alexmiller fyi i updated clj tools and the error still happens.

I do actually see msgpack on the classpath with these deps. Can you run clj -Sforce -Stree -A:dev and grep the output for msgpack? force will force recomputing the classpath - it's possible you are seeing an older cached classpath

also if you're still seeing it after that, do clj -Strace -A:dev and attach the trace.edn file it emits here

when untruthifying™ a boolean value of a schema attribute, is there an advantage to choosing one method over the other?

[:db/retract :some/ident :db/isComponent true]

[:db/add :some/ident :db/isComponent false]

fwiw, they are not equivalent, right? The first removes the fact about being a component altogether, and the second asserts it as false

do folks compose pull patterns? suppose i have an entity and for one use case, i need a set of attrs; for another, i need a non-overlapping different set of attrs. since it's all data, maybe they can be defined separately but then composed so i can make one db call instead of n

i probably want to use sets instead of vectors in the initial representation? merge those, then swap the sets for vectors before use with datomic

handling the vector syntax vs the map syntax might be tricky

@michael740 We use the https://github.com/edn-query-language/eql for this.

https://github.com/edn-query-language/eql#unions ?

We go pull pattern -> ast -> merge -> pull pattern.

The merge is typically very simple since it’s in the ast format.

thanks!

this looks cool as all get out

We also use pathom so this is a very natural lib for us to use. You could probably write a smaller version to just do what you need if you don’t want to bring on a new dep.

yup. in this case the resulting behaviours are the same (:some/ident is no longer a component), but the resulting annotations of the ident are different (false vs missing). i'm just wondering why someone might choose one over the other.