hello! today i notice a weird interaction between datomic client api and djblue/portal (https://github.com/djblue/portal): when this last one is not on the classpath, then i can obtain a connection to my (local, datomic pro) database; when portal is on the classpath, connecting to the database fails with the following error:
Exception in thread "async-dispatch-1" java.lang.RuntimeException: java.lang.NoClassDefFoundError: org/msgpack/MessagePack
at com.cognitect.transit.TransitFactory.writer(TransitFactory.java:104)
at cognitect.transit$writer.invokeStatic(transit.clj:161)
at cognitect.transit$writer.invoke(transit.clj:139)
at <http://datomic.client.impl.shared.io|datomic.client.impl.shared.io>$marshal.invokeStatic(io.clj:48)
at <http://datomic.client.impl.shared.io|datomic.client.impl.shared.io>$marshal.invoke(io.clj:38)
at <http://datomic.client.impl.shared.io|datomic.client.impl.shared.io>$client_req__GT_http_req.invokeStatic(io.clj:76)
at <http://datomic.client.impl.shared.io|datomic.client.impl.shared.io>$client_req__GT_http_req.invoke(io.clj:73)
at datomic.client.impl.shared.Client._async_op(shared.clj:398)
at datomic.client.impl.shared.Client$fn__34578$state_machine__5717__auto____34593$fn__34595.invoke(shared.clj:423)
at datomic.client.impl.shared.Client$fn__34578$state_machine__5717__auto____34593.invoke(shared.clj:422)
at clojure.core.async.impl.ioc_macros$run_state_machine.invokeStatic(ioc_macros.clj:973)
at clojure.core.async.impl.ioc_macros$run_state_machine.invoke(ioc_macros.clj:972)
at clojure.core.async.impl.ioc_macros$run_state_machine_wrapped.invokeStatic(ioc_macros.clj:977)
at clojure.core.async.impl.ioc_macros$run_state_machine_wrapped.invoke(ioc_macros.clj:975)
at datomic.client.impl.shared.Client$fn__34578.invoke(shared.clj:422)
at clojure.lang.AFn.run(AFn.java:22)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at clojure.core.async.impl.concurrent$counted_thread_factory$reify__469$fn__470.invoke(concurrent.clj:29)
at clojure.lang.AFn.run(AFn.java:22)
at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.lang.NoClassDefFoundError: org/msgpack/MessagePack
at com.cognitect.transit.impl.WriterFactory.getMsgpackInstance(WriterFactory.java:77)
at com.cognitect.transit.TransitFactory.writer(TransitFactory.java:95)
... 20 more
Caused by: java.lang.ClassNotFoundException: org.msgpack.MessagePack
at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:581)
at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:521)
... 22 more
clj -Sforce -Stree -A:dev
returns nothing. djblue/portal was commented out in deps.edn, maybe that is why you see it in your output.
@alexmiller sorry for the delay in my reply…
i noticed that portal has com.cognitect/- transit-cljs, transit-js, transit-clj, and transit-java as dependencies. why would these dependencies interfere with datomic obtaining a connection?
(there are other dependencies but i don’t believe they have anything to do with the case: cheshire 5.10.0 and http-kit 2.5.0).
a) what are you using to make the classpath b) if clj, what version? (prob best to upgrade to latest if you haven’t) c) please provide the set of project deps that repro this
i am using clojure tools (clojure 1.10.1) and i open a repl using cider-jack-in-clj. then i start my system via integrant.repl.
this is my deps.edn. when connecting to the repl, i add -A:dev to the clj command.
Portal requires transit but excludes msgpack
and datomic requires msgpack?
if this is the case, is it a classpath resolution conflict/problem, i.e. msgpack should be in the final classpath because datomic requires it, even if portal excludes it?
what version of the clojure tools? clj -Sdescribe
there were issues with this kind of scenario that were fixed a few versions ago
release info here https://clojure.org/releases/tools - latest is 1.10.1.727
The client apparently requires msgpack but doesn’t depend on it directly, expecting it via transit
Yes
Portal uses transit but not the msgpack encoding option, so it doesn’t want to bring it in
Datomic client uses transit with msgpack encoding but doesn’t require it directly
I wonder what maven would compute in this case
I'm stepping away till this afternoon but I'd be happy to look at this in depth when I get back. My recommendation would be to move to latest clj if you haven't already as there have been fixes in this area.
Hello @joshkh, @jaret! Thank you for explanation. In the end we decided to go with event bridge or sns topic directly. It is problematic to route events from cloudwatch logs to aws lambda functions. As the only option there is subscription filter (max 2 per log group). Maybe I missing something, but haven't found solution where I am able to get events submitted using cast/event to lambda functions.
thanks, @alexmiller and @favila. i’ll update clj tools if mine is not up-to-date.
https://clojurians.slack.com/archives/C03RZMDSH/p1603978063215500?thread_ts=1603974900.204700&cid=C03RZMDSH mine was 1.10.1.716
(Sorry if I’m confusing, slack must have barfed on my messages because they’re all out of order and 20 min late)
1😂i don't know if this is useful to you, but we cast/alert exceptions to CloudWatch, and then use a CLJS Lambda to send them to Slack to torture our developers. cast/events are not really any different, except for maybe the frequency at which they appear, but i think log streams are batched. (sorry for the lack of a proper README, it was a hobby project)
https://github.com/joshkh/datomic-alerts-to-slack
> It is problematic to route events from cloudwatch logs to aws lambda functions. As the only option there is subscription filter (max 2 per log group).
for application level logs and alerts, we tend to the use a common message throughout the application (e.g. {:msg "<app-name>-application-log"}
) , and then we attach other keys such as :env
and :query-group
. this provides us different levels of filtering while keeping our logs all in one place
but if SNS works for you then go for it! 🙂 i just prefer CloudWatch because cast
serialises Clojure to JSON very well, and being able to add arbitrary keys at any level is useful for filtering
@alexmiller fyi i updated clj tools and the error still happens.
I do actually see msgpack on the classpath with these deps. Can you run clj -Sforce -Stree -A:dev
and grep the output for msgpack? force will force recomputing the classpath - it's possible you are seeing an older cached classpath
also if you're still seeing it after that, do clj -Strace -A:dev
and attach the trace.edn file it emits here
when untruthifying™ a boolean value of a schema attribute, is there an advantage to choosing one method over the other?
[:db/retract :some/ident :db/isComponent true]
vs
[:db/add :some/ident :db/isComponent false]
just curiousfwiw, they are not equivalent, right? The first removes the fact about being a component altogether, and the second asserts it as false
do folks compose pull patterns? suppose i have an entity and for one use case, i need a set of attrs; for another, i need a non-overlapping different set of attrs. since it's all data, maybe they can be defined separately but then composed so i can make one db call instead of n
@michael740 I do this a lot. I use the apply-template fn in clojure core to inject pull vectors into other pull vectors or queries. it’s simple and it works well.
1🙏even better if you match a transform fn for the results to each pull expr. then you can compose them to process the results as well.
do you mean apply-template
in clojure.template
?
sorry, yes, that’s the one. in my case, I generate the pull expressions and post query transform fns from a domain model. something like https://github.com/stevebuik/clj-code-gen-hodur
i probably want to use sets instead of vectors in the initial representation? merge those, then swap the sets for vectors before use with datomic
handling the vector syntax vs the map syntax might be tricky
@michael740 We use the https://github.com/edn-query-language/eql for this.
We go pull pattern -> ast -> merge -> pull pattern.
The merge is typically very simple since it’s in the ast format.
thanks!
this looks cool as all get out
We also use pathom so this is a very natural lib for us to use. You could probably write a smaller version to just do what you need if you don’t want to bring on a new dep.
yup. in this case the resulting behaviours are the same (:some/ident is no longer a component), but the resulting annotations of the ident are different (false vs missing). i'm just wondering why someone might choose one over the other.