datomic

Ask questions on the official Q&A site at https://ask.datomic.com!
2020-09-04T05:15:56.234900Z

With Datomic on-prem with PostgreSQL, would it be possible to increase the read capacity of datomic by adding Postgres recplicas of the database? I did not see any documention on whether it was feasible in the docs. I guess it would not make sense because of transactors.

cmdrdats 2020-09-04T09:10:11.237400Z

I'm finding using enum's as refs in datomic (on-prem) extremely tedious - in order to find the enum keyword, I need to pull or query/join the :db/ident every time... writing and explicitly querying is simple enough, but where I want to do something like (case (:message/status m) :message.status/read ...) it's a pain.. is there something I'm missing? Currently, I'm heavily leaning toward just using the :db.type/keyword

2020-09-04T09:22:20.240900Z

Hi. I've setup a Datomic Cloud in us-east-1. I'm going through the ion tutorial at https://docs.datomic.com/cloud/ions/ions-tutorial.html. Everything is working until I try to clojure -A:ion-dev '{:op :push}' when I get a 403 AccessDenied trying to download the ion-dev pom/jars. I have an AWS_PROFILE environment exported, which seems to be working enough to get ion jars downloaded and the samples working up until this point (i.e. I can load the sample db, run queries etc). Any thoughts on this would be much appreciated.

2020-09-04T09:24:07.241400Z

Could not transfer artifact com.datomic:ion-dev:pom:0.9.265 from/to datomic-cloud (<s3://datomic-releases-1fc2183a/maven/releases>): Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: XXX; S3 Extended Request ID: XXX)

2020-09-07T15:32:05.291300Z

Thanks - yeah, must have been that. I'm not seeing issues doing this now at all. It didn't make sense that some artefacts could be downloaded, but not others.

2020-09-04T09:32:24.241500Z

Incidentally, I'm curious as to why these are authenticated maven repos in the first place. Anyone know why this would be?

pithyless 2020-09-04T09:40:58.241700Z

One advantage is you can hang other attributes off of an :db/ident enum (e.g. docs, etc.). This may or may not be useful to you. If it's just a question of dealing with the pull result - and all I care about in a specific context are the keywords - I find it useful to have a helper function that will postwalk the pull result and replace all the enum maps (cardinality one-or-many) with the :db/ident keyword.

2020-09-04T11:14:40.241900Z

This is now working, but intermittently. Unfortunately I can't say what fixed it as I've made no changes - I've just tried again. Perhaps an AWS issue?

cmdrdats 2020-09-04T11:34:23.242100Z

ye, I'm just not sure it's worth it when I can just keep a map about the enum's around (most might have a displayname of some sort, but that's really it).. the postwalk sounds too fiddly for my liking, and I'd much rather have us not have to think about it 😕 Trying to find out if there's another reason for using refs instead of keywords? is the indexing more efficient?

favila 2020-09-04T12:11:40.242900Z

Pull with xform is another option

favila 2020-09-04T12:13:19.245100Z

{[:ident-Val-attr :xform lift-ident] [:db/ident]}

favila 2020-09-04T12:14:23.246700Z

Define lift-ident as :db/ident

Jake Shelby 2020-09-04T15:41:38.247Z

there were pretty big IAM issues yesterday - so it's possibly an AWS thing, depending on when you were getting the issue

Jake Shelby 2020-09-04T16:06:15.249100Z

I just noticed that https://docs.datomic.com/cloud/releases.html#current shows older versions for ion and ion-dev than is listed in the https://docs.datomic.com/cloud/releases.html#ion-dev-276

Jake Shelby 2020-09-04T16:09:12.251Z

(in case anyone else is scratching their heads wondering why it looks as if there are conflicts for the latest Datomic Cloud versions not using the latest client libs ... you just have to use the actual latest version of ion-dev)

cmdrdats 2020-09-04T16:49:28.255600Z

But why do all that instead of just keyword? Besides being able to attach other metadata to the enum value, of course? Surely there's another tradeoff?

jaret 2020-09-04T17:04:26.256400Z

@jake.shelby Thanks for the report! I just fixed this …an artifact in our docs build failed to get updated.

1👍
Matheus Serpellone Nash 2020-09-04T19:07:44.261800Z

Hello! We noticed our application tends to crash after a few weeks of no restarts (due to no deploys or cycles), we see increased GC activity and CPU usage until it becomes unhealthy and we have to cycle it. I used jmap to extract a thread dump of a recent pod (24h), and ran it through eclipse memory analyzer, to see if i can find any leaks. It’s telling me that there is an object datomic.cache.WrappedGCache retaining 9+GB of heap. I have attached the stack from gcroot I tried to search for this class to understand what it does, but nothing shows up. Do you guys have any advice on how to debug this? Is this Datomic’s peer getting too bloated from all the queries? Can I tune this to not take too much space? I’m a bit uncertain on how to proceed. Also, sorry if this is not the right channel.

csm 2020-09-04T19:28:06.262500Z

I think https://docs.datomic.com/on-prem/caching.html#object-cache is what you’re looking for?

1👍
Matheus Serpellone Nash 2020-09-04T19:29:39.264500Z

Ah ok! I just also realized that the “95% usage” im getting is probably from the used heap, not total heap

csm 2020-09-04T19:30:09.265200Z

Though we also see our peer processes eventually start consuming a significant amount of CPU, and inspecting the process, it’s almost all GC scans, and almost all the memory is in datomic’s cache. I haven’t figured out if there is a good way to tune the GC to handle a large in-memory cache, to avoid this CPU usage

Matheus Serpellone Nash 2020-09-04T19:41:33.265700Z

What are you using to inspect the process? visualvm?

csm 2020-09-04T19:58:25.266Z

top -H -p &lt;pid&gt;

csm 2020-09-04T20:01:57.266600Z

(and jmap)

marshall 2020-09-04T20:21:13.267400Z

what JVM gc flags are you using and what heap size

marshall 2020-09-04T20:22:01.268100Z

you mostly likely want to use the recommended flags from our docs: -XX:+UseG1GC and -XX:MaxGCPauseMillis=50

Matheus Serpellone Nash 2020-09-04T20:22:55.268900Z

Let me confirm. I’m sure we use G1. Not sure about gc pause

marshall 2020-09-04T20:22:58.269Z

@m.serpellone this is a peer application, yes?

Matheus Serpellone Nash 2020-09-04T20:23:29.269300Z

yes

marshall 2020-09-04T20:23:52.269800Z

keep in mind the peer is your app so you need to be aware of things like head holding/etc

marshall 2020-09-04T20:24:27.270500Z

if looking for memory leaks/etc

Nassin 2020-09-04T20:25:27.270600Z

enums gives you more constraints (if you need that), a keyword type would accept any keyword, with refs, only already asserted enums/entities are valid

cmdrdats 2020-09-04T20:32:01.270800Z

That's ok, I can work around that easily enough, even with the constraint functions.. i forget what they're called.. surely there's a more fundamental reason?

Nassin 2020-09-04T20:56:49.271300Z

Using keywords directly will require more storage I guess(duplicates), instead of pointing always to the same enum.

Nassin 2020-09-04T20:57:41.271700Z

you could create refs to keywords but no idea if that is less or more efficient than refs to enums

Nassin 2020-09-04T20:58:08.271900Z

docs don't say, datomic being a black box, only cognitect knows

2020-09-04T21:08:14.273200Z

Can I ask more lighthearted questions here? Stuart Halloway has this party trick of using a vector as vector as a database value, what makes that that doesn’t work for the client api? Sounded like a simple way to test rules ^^

marshall 2020-09-04T21:09:52.273800Z

@lennart.buit you can do that in the peer because the work of query happens in the peer

marshall 2020-09-04T21:10:12.274300Z

client sends the request over the wire to the peer server (on prem) or to a datomic cloud system (cloud)

2020-09-04T21:10:28.274700Z

Ah yes that makes sense 🙂! I should have thought of that

kenny 2020-09-04T21:14:43.277Z

It'd be great if ion-dev bumped its version of tools.deps. The version it is using still uses the s3-wagon-private lib which transitively brings in a logback.xml file. This causes the multiple slf4j bindings warning. I can manually bump tools.deps to 0.8.709 to work around the warning.

1👍
favila 2020-09-04T21:20:04.277300Z

This is really just the difference between a pure value and something you would assert things about

favila 2020-09-04T21:21:29.277500Z

if it’s really just a pure value: sure, keyword, fine; but if you want to assert anything about it (e.g. give it unique names, change its name, say it belongs to a set of other values in the same domain, give them aliases for different contexts, say who introduced them, give them sorting priorities, etc etc), then entities give you that flexibility