datomic

Ask questions on the official Q&A site at https://ask.datomic.com!
Matheus Moreira 2020-10-29T12:35:00.204700Z

hello! today i notice a weird interaction between datomic client api and djblue/portal (https://github.com/djblue/portal): when this last one is not on the classpath, then i can obtain a connection to my (local, datomic pro) database; when portal is on the classpath, connecting to the database fails with the following error:

Exception in thread "async-dispatch-1" java.lang.RuntimeException: java.lang.NoClassDefFoundError: org/msgpack/MessagePack
	at com.cognitect.transit.TransitFactory.writer(TransitFactory.java:104)
	at cognitect.transit$writer.invokeStatic(transit.clj:161)
	at cognitect.transit$writer.invoke(transit.clj:139)
	at <http://datomic.client.impl.shared.io|datomic.client.impl.shared.io>$marshal.invokeStatic(io.clj:48)
	at <http://datomic.client.impl.shared.io|datomic.client.impl.shared.io>$marshal.invoke(io.clj:38)
	at <http://datomic.client.impl.shared.io|datomic.client.impl.shared.io>$client_req__GT_http_req.invokeStatic(io.clj:76)
	at <http://datomic.client.impl.shared.io|datomic.client.impl.shared.io>$client_req__GT_http_req.invoke(io.clj:73)
	at datomic.client.impl.shared.Client._async_op(shared.clj:398)
	at datomic.client.impl.shared.Client$fn__34578$state_machine__5717__auto____34593$fn__34595.invoke(shared.clj:423)
	at datomic.client.impl.shared.Client$fn__34578$state_machine__5717__auto____34593.invoke(shared.clj:422)
	at clojure.core.async.impl.ioc_macros$run_state_machine.invokeStatic(ioc_macros.clj:973)
	at clojure.core.async.impl.ioc_macros$run_state_machine.invoke(ioc_macros.clj:972)
	at clojure.core.async.impl.ioc_macros$run_state_machine_wrapped.invokeStatic(ioc_macros.clj:977)
	at clojure.core.async.impl.ioc_macros$run_state_machine_wrapped.invoke(ioc_macros.clj:975)
	at datomic.client.impl.shared.Client$fn__34578.invoke(shared.clj:422)
	at clojure.lang.AFn.run(AFn.java:22)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at clojure.core.async.impl.concurrent$counted_thread_factory$reify__469$fn__470.invoke(concurrent.clj:29)
	at clojure.lang.AFn.run(AFn.java:22)
	at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.lang.NoClassDefFoundError: org/msgpack/MessagePack
	at com.cognitect.transit.impl.WriterFactory.getMsgpackInstance(WriterFactory.java:77)
	at com.cognitect.transit.TransitFactory.writer(TransitFactory.java:95)
	... 20 more
Caused by: java.lang.ClassNotFoundException: org.msgpack.MessagePack
	at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:581)
	at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:521)
	... 22 more

Matheus Moreira 2020-11-03T11:39:45.280100Z

clj -Sforce -Stree -A:dev returns nothing. djblue/portal was commented out in deps.edn, maybe that is why you see it in your output.

Matheus Moreira 2020-11-03T11:40:33.280300Z

@alexmiller sorry for the delay in my reply…

Matheus Moreira 2020-10-29T12:37:45.204900Z

i noticed that portal has com.cognitect/- transit-cljs, transit-js, transit-clj, and transit-java as dependencies. why would these dependencies interfere with datomic obtaining a connection?

Matheus Moreira 2020-10-29T12:38:25.205100Z

(there are other dependencies but i don’t believe they have anything to do with the case: cheshire 5.10.0 and http-kit 2.5.0).

alexmiller 2020-10-29T12:42:12.207500Z

a) what are you using to make the classpath b) if clj, what version? (prob best to upgrade to latest if you haven’t) c) please provide the set of project deps that repro this

Matheus Moreira 2020-10-29T12:53:22.207700Z

i am using clojure tools (clojure 1.10.1) and i open a repl using cider-jack-in-clj. then i start my system via integrant.repl.

Matheus Moreira 2020-10-29T12:54:58.207900Z

this is my deps.edn. when connecting to the repl, i add -A:dev to the clj command.

favila 2020-10-29T13:04:52.209Z

Portal requires transit but excludes msgpack

Matheus Moreira 2020-10-29T13:06:12.210400Z

and datomic requires msgpack?

Matheus Moreira 2020-10-29T13:07:05.211600Z

if this is the case, is it a classpath resolution conflict/problem, i.e. msgpack should be in the final classpath because datomic requires it, even if portal excludes it?

alexmiller 2020-10-29T13:26:30.215100Z

what version of the clojure tools? clj -Sdescribe

alexmiller 2020-10-29T13:27:06.215300Z

there were issues with this kind of scenario that were fixed a few versions ago

alexmiller 2020-10-29T13:27:43.215500Z

release info here https://clojure.org/releases/tools - latest is 1.10.1.727

favila 2020-10-29T13:33:38.215700Z

The client apparently requires msgpack but doesn’t depend on it directly, expecting it via transit

favila 2020-10-29T13:33:46.215900Z

Yes

favila 2020-10-29T13:33:52.216100Z

Portal uses transit but not the msgpack encoding option, so it doesn’t want to bring it in

favila 2020-10-29T13:33:57.216300Z

Datomic client uses transit with msgpack encoding but doesn’t require it directly

favila 2020-10-29T13:34:04.216500Z

I wonder what maven would compute in this case

alexmiller 2020-10-29T13:38:31.217500Z

I'm stepping away till this afternoon but I'd be happy to look at this in depth when I get back. My recommendation would be to move to latest clj if you haven't already as there have been fixes in this area.

Aleh Atsman 2020-10-29T14:08:59.217700Z

Hello @joshkh, @jaret! Thank you for explanation. In the end we decided to go with event bridge or sns topic directly. It is problematic to route events from cloudwatch logs to aws lambda functions. As the only option there is subscription filter (max 2 per log group). Maybe I missing something, but haven't found solution where I am able to get events submitted using cast/event to lambda functions.

Matheus Moreira 2020-10-29T14:14:39.217900Z

thanks, @alexmiller and @favila. i’ll update clj tools if mine is not up-to-date.

favila 2020-10-29T14:20:01.220100Z

(Sorry if I’m confusing, slack must have barfed on my messages because they’re all out of order and 20 min late)

1😂
joshkh 2020-10-29T14:20:31.220300Z

i don't know if this is useful to you, but we cast/alert exceptions to CloudWatch, and then use a CLJS Lambda to send them to Slack to torture our developers. cast/events are not really any different, except for maybe the frequency at which they appear, but i think log streams are batched. (sorry for the lack of a proper README, it was a hobby project) https://github.com/joshkh/datomic-alerts-to-slack > It is problematic to route events from cloudwatch logs to aws lambda functions. As the only option there is subscription filter (max 2 per log group). for application level logs and alerts, we tend to the use a common message throughout the application (e.g. {:msg "&lt;app-name&gt;-application-log"} ) , and then we attach other keys such as :env and :query-group . this provides us different levels of filtering while keeping our logs all in one place

joshkh 2020-10-29T14:22:23.220600Z

but if SNS works for you then go for it! 🙂 i just prefer CloudWatch because cast serialises Clojure to JSON very well, and being able to add arbitrary keys at any level is useful for filtering

Matheus Moreira 2020-10-29T14:31:52.221100Z

@alexmiller fyi i updated clj tools and the error still happens.

alexmiller 2020-10-29T18:57:57.226400Z

I do actually see msgpack on the classpath with these deps. Can you run clj -Sforce -Stree -A:dev and grep the output for msgpack? force will force recomputing the classpath - it's possible you are seeing an older cached classpath

alexmiller 2020-10-29T18:59:17.227400Z

also if you're still seeing it after that, do clj -Strace -A:dev and attach the trace.edn file it emits here

joshkh 2020-10-29T19:04:18.230800Z

when untruthifying™ a boolean value of a schema attribute, is there an advantage to choosing one method over the other?

[:db/retract :some/ident :db/isComponent true]
vs
[:db/add :some/ident :db/isComponent false]
just curious

2020-10-29T19:46:15.231400Z

fwiw, they are not equivalent, right? The first removes the fact about being a component altogether, and the second asserts it as false

2020-10-29T20:09:38.233100Z

do folks compose pull patterns? suppose i have an entity and for one use case, i need a set of attrs; for another, i need a non-overlapping different set of attrs. since it's all data, maybe they can be defined separately but then composed so i can make one db call instead of n

steveb8n 2020-11-01T03:21:02.252200Z

@michael740 I do this a lot. I use the apply-template fn in clojure core to inject pull vectors into other pull vectors or queries. it’s simple and it works well.

1🙏
steveb8n 2020-11-01T03:21:37.252400Z

even better if you match a transform fn for the results to each pull expr. then you can compose them to process the results as well.

2020-11-02T23:47:46.279500Z

do you mean apply-template in clojure.template?

steveb8n 2020-11-03T21:57:40.300Z

sorry, yes, that’s the one. in my case, I generate the pull expressions and post query transform fns from a domain model. something like https://github.com/stevebuik/clj-code-gen-hodur

2020-10-29T20:10:28.233800Z

i probably want to use sets instead of vectors in the initial representation? merge those, then swap the sets for vectors before use with datomic

2020-10-29T20:12:18.234400Z

handling the vector syntax vs the map syntax might be tricky

kenny 2020-10-29T20:14:14.235Z

@michael740 We use the https://github.com/edn-query-language/eql for this.

kenny 2020-10-29T20:17:38.236300Z

We go pull pattern -> ast -> merge -> pull pattern.

kenny 2020-10-29T20:17:56.237Z

The merge is typically very simple since it’s in the ast format.

2020-10-29T20:19:25.237600Z

thanks!

2020-10-29T20:19:33.237800Z

this looks cool as all get out

kenny 2020-10-29T20:20:30.239200Z

We also use pathom so this is a very natural lib for us to use. You could probably write a smaller version to just do what you need if you don’t want to bring on a new dep.

joshkh 2020-10-29T21:59:29.244100Z

yup. in this case the resulting behaviours are the same (:some/ident is no longer a component), but the resulting annotations of the ident are different (false vs missing). i'm just wondering why someone might choose one over the other.