Clojurians Log v2

With Datomic on-prem with PostgreSQL, would it be possible to increase the read capacity of datomic by adding Postgres recplicas of the database? I did not see any documention on whether it was feasible in the docs. I guess it would not make sense because of transactors.

I'm finding using enum's as refs in datomic (on-prem) extremely tedious - in order to find the enum keyword, I need to pull or query/join the :db/ident every time... writing and explicitly querying is simple enough, but where I want to do something like (case (:message/status m) :message.status/read ...) it's a pain.. is there something I'm missing? Currently, I'm heavily leaning toward just using the :db.type/keyword

Hi. I've setup a Datomic Cloud in us-east-1. I'm going through the ion tutorial at https://docs.datomic.com/cloud/ions/ions-tutorial.html. Everything is working until I try to clojure -A:ion-dev '{:op :push}' when I get a 403 AccessDenied trying to download the ion-dev pom/jars. I have an AWS_PROFILE environment exported, which seems to be working enough to get ion jars downloaded and the samples working up until this point (i.e. I can load the sample db, run queries etc). Any thoughts on this would be much appreciated.

Could not transfer artifact com.datomic:ion-dev:pom:0.9.265 from/to datomic-cloud (<s3://datomic-releases-1fc2183a/maven/releases>): Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: XXX; S3 Extended Request ID: XXX)

Incidentally, I'm curious as to why these are authenticated maven repos in the first place. Anyone know why this would be?

One advantage is you can hang other attributes off of an :db/ident enum (e.g. docs, etc.). This may or may not be useful to you. If it's just a question of dealing with the pull result - and all I care about in a specific context are the keywords - I find it useful to have a helper function that will postwalk the pull result and replace all the enum maps (cardinality one-or-many) with the :db/ident keyword.

This is now working, but intermittently. Unfortunately I can't say what fixed it as I've made no changes - I've just tried again. Perhaps an AWS issue?

ye, I'm just not sure it's worth it when I can just keep a map about the enum's around (most might have a displayname of some sort, but that's really it).. the postwalk sounds too fiddly for my liking, and I'd much rather have us not have to think about it 😕 Trying to find out if there's another reason for using refs instead of keywords? is the indexing more efficient?

Pull with xform is another option

{[:ident-Val-attr :xform lift-ident] [:db/ident]}

Define lift-ident as :db/ident

there were pretty big IAM issues yesterday - so it's possibly an AWS thing, depending on when you were getting the issue

I just noticed that https://docs.datomic.com/cloud/releases.html#current shows older versions for ion and ion-dev than is listed in the https://docs.datomic.com/cloud/releases.html#ion-dev-276

(in case anyone else is scratching their heads wondering why it looks as if there are conflicts for the latest Datomic Cloud versions not using the latest client libs ... you just have to use the actual latest version of ion-dev)

But why do all that instead of just keyword? Besides being able to attach other metadata to the enum value, of course? Surely there's another tradeoff?

@jake.shelby Thanks for the report! I just fixed this …an artifact in our docs build failed to get updated.

Hello! We noticed our application tends to crash after a few weeks of no restarts (due to no deploys or cycles), we see increased GC activity and CPU usage until it becomes unhealthy and we have to cycle it. I used jmap to extract a thread dump of a recent pod (24h), and ran it through eclipse memory analyzer, to see if i can find any leaks. It’s telling me that there is an object datomic.cache.WrappedGCache retaining 9+GB of heap. I have attached the stack from gcroot I tried to search for this class to understand what it does, but nothing shows up. Do you guys have any advice on how to debug this? Is this Datomic’s peer getting too bloated from all the queries? Can I tune this to not take too much space? I’m a bit uncertain on how to proceed. Also, sorry if this is not the right channel.

I think https://docs.datomic.com/on-prem/caching.html#object-cache is what you’re looking for?

Ah ok! I just also realized that the “95% usage” im getting is probably from the used heap, not total heap

Though we also see our peer processes eventually start consuming a significant amount of CPU, and inspecting the process, it’s almost all GC scans, and almost all the memory is in datomic’s cache. I haven’t figured out if there is a good way to tune the GC to handle a large in-memory cache, to avoid this CPU usage

What are you using to inspect the process? visualvm?

top -H -p <pid>

(and jmap)

what JVM gc flags are you using and what heap size

you mostly likely want to use the recommended flags from our docs: -XX:+UseG1GC and -XX:MaxGCPauseMillis=50

Let me confirm. I’m sure we use G1. Not sure about gc pause

@m.serpellone this is a peer application, yes?

yes

keep in mind the peer is your app so you need to be aware of things like head holding/etc

if looking for memory leaks/etc

enums gives you more constraints (if you need that), a keyword type would accept any keyword, with refs, only already asserted enums/entities are valid

That's ok, I can work around that easily enough, even with the constraint functions.. i forget what they're called.. surely there's a more fundamental reason?

Using keywords directly will require more storage I guess(duplicates), instead of pointing always to the same enum.

you could create refs to keywords but no idea if that is less or more efficient than refs to enums

docs don't say, datomic being a black box, only cognitect knows

Can I ask more lighthearted questions here? Stuart Halloway has this party trick of using a vector as vector as a database value, what makes that that doesn’t work for the client api? Sounded like a simple way to test rules ^^

@lennart.buit you can do that in the peer because the work of query happens in the peer

client sends the request over the wire to the peer server (on prem) or to a datomic cloud system (cloud)

Ah yes that makes sense 🙂! I should have thought of that

It'd be great if ion-dev bumped its version of tools.deps. The version it is using still uses the s3-wagon-private lib which transitively brings in a logback.xml file. This causes the multiple slf4j bindings warning. I can manually bump tools.deps to 0.8.709 to work around the warning.

This is really just the difference between a pure value and something you would assert things about

if it’s really just a pure value: sure, keyword, fine; but if you want to assert anything about it (e.g. give it unique names, change its name, say it belongs to a set of other values in the same domain, give them aliases for different contexts, say who introduced them, give them sorting priorities, etc etc), then entities give you that flexibility