Clojurians Log v2

Does anyone know enough about upserts in mysql and postgresql to be able to review this PR from @niamu? I don’t have enough time to properly investigate at the moment https://github.com/onyx-platform/onyx-sql/pull/31

@lucasbradstreet I have done a little with PG upserts, I would want a few more tests around the different cases of executing the queries to make sure they are all handled correctly

But the PG one seems ok to me, and matches the kind of queries I have written in HugSQL

Documentation would still need to be improved before that gets accepted as well. I’ll expand on that a little more tomorrow.

It needs to be made clear that the :where map would need to have the primary key value present in order for the upserts to work.

Thanks guys

Much appreciated everyone.

I think when a duplicate key exception occurs, postgresql automatically aborts the current transaction

I see that multiple of these upserts are done in a single transaction tho

so I would like to take a better look at how hsql handles this

Hi @michaeldrogalis, I finally managed to put together a minimal reproducer to point out the problems I'm experiencing when using onyx with tags (wrote about this some days ago). You can find the example here: https://github.com/DiCanio/onyx-tag-usage-error I hope everything is covered by the README. If this is not the case for you please feel free to write me. Thanks in advance for your help.

Great, thanks @atwrdik! Ill try to get to this one tonight, or if not in the next few days.

Does anyone use Onyx for archiving Kafka topics and replaying the archive back to a Kafka topic? Can anyone recommend a process for Kafka archiving and replay-ability?

@niamu as long as you configure the topic to never delete stuff, you could always read from tree beginning of a topic. Depending on use case, you can use compaction, so messages with the same key eventually do get deleted. You could also put each message in a database, and thus be able to play it back, but seems to make things unnecessary complex.

@gklijs Thanks. I’m not sure if it’s easier for us to be able to scale storage to some separate object store for Kafka messages or to scale the retention of Kafka itself. So that’s where some of the decision making comes in as well.

@niamu I used to use Onyx as a Kafka store to S3 with an ordered date/time so you had system of a record. That’s kinda got replaced with Kafka Connect now. If you want to keep the storage down it makes sense to do it.

@atwrdik Pretty sure this is a bug in the scheduler. Thanks for the reproducer, makes my job a heck of a lot easier.

@atwrdik could you please give it a try with onyx 0.12.3? We just merged https://github.com/onyx-platform/onyx/pull/837

Oh, I hadn't even noticed that PR. Thanks @lucasbradstreet. I would have gone on a wild goose chase.

@lucasbradstreet @michaeldrogalis when using onyx 0.12.3 the job actually starts. However, it seems as if tasks are now assigned to peers that do not offer the required functionality/tags. Cannot estimate whether this has something to do with the changes from the PR or if the project is just misconfigured.

I can definitely make a test for it by only looking at the replica. When you strip away all of the messaging/concurrency code, these bugs are super straight forward to reproduce and observe

Fixing is sometimes hard because scheduling is just a tough subject, but this sounds plain broken

Ill investigate tonight

Thx for the insight - might come in handy in the future. Appreciate your effort 👍:skin-tone-2:

Anytime!

Anyone using (or planning) onyx-datomic with #datomic cloud? I see that https://github.com/onyx-platform/onyx-datomic/blob/0.12.x/src/onyx/plugin/datomic.clj is almost compatible with Client API L41 the use of d/ident can turn into d/pull L142 on client API, d/tx-range receives conn, not log. L5 sould require datomic.client.api, not datomic.api 😛

@souenzzo It sounds like with the exception of L142, we can make changes to support both?

@michaeldrogalis not sure how (different build params/artifacts, runtime arg). I will investigate.

Cool, thanks. I'd love to support both. Datomic Cloud looks wonderful.