Clojurians Log v2
onyx
Hi, I’m in early design phase for the next project, and looking at Datomic Cloud as the storage. I think onyx datomic plugin uses peer api datomic.api, which is not supported by Datomic Cloud. Do you have a plan to add support for client api to the plugin?
@kenji_signifier There was discussion of that just the other day: https://clojurians.slack.com/archives/C051WKSP3/p1516653946000087
onyx-sql was just bumped to version 1.9.0 recently, so yes.
Bring it up to 1.9.0 - yup
I spent some time to get familiar myself to datomic cloud and onyx. Here are the gaps I face;
the following fns don’t exist in client-api. entity, ident, log, next-t, and tempid. as @souenzzo suggested some fns may be re-written in client-api, but not sure for all of them. Majority of them are used in tests so re-writing tests might be an option but I’d like to minimize the deviations.
I managed to handle selective lib loading and abstraction by pushing down the impls under protocol.
Tomorrow I’m gonna try the approach to keep using peer api for tests, and incorporate on-prem client lib. The differences between on-prem and cloud client api are primarily in connection area so I hope it has good enough fidelity.
I made some progress and now can handle connections with peer, client via peer-server, and client via cloud. I’m going to spend some more time this weekend to convert tests to use client API so that I can run tests against Datomic cloud, too.
I managed to pass tx_output_test.clj in both peer and cloud. 8 more tests to go…
Nice work! I’m pretty busy at the moment but I will try to get a quick review in ASAP.
They were going to investigate further to see what changes were necessary to support it, but the initial reaction was that it would be a small change.
@niamu, thx, i searched in slack but didn’t find it. Looks promising!
Happy to take a patch if someone can make both version compatible, or add a parallel set of tasks
@michaeldrogalis I’m gonna give a shot this weekend.
@kenji_signifier Great! Any questions, hit us up
thx
@souenzzo Fantastic!
Do all Onyx plugins have to be utilized as an input or output task? I'm guessing that's the only usable interface for a plugin right?
output plugins can also be used on reduce tasks, but that’s almost the same thing as outputs
We have discussed being able to make intermediate tasks outputs, and then flowing the messages further down, but we haven’t been able to figure out how to make it make sense.
ah gotcha, that's kinda where I just got stuck. I was trying to make an output task flow into another output task and it didn't seem to work
What’s the use case?
just for dev experimentation. I wanted to save to Elasticsearch and make the result available in a channel as an async out task
K. Yeah, I think the general approach we’d recommend for now is to make an intermediate task that does the transformation, then have it flow to an elasticsearch task as well as a core async task
Right. Gotcha, yea that makes sense. Thanks!