Clojurians Log v2

Trying to get compojure-api + aleph + core.async combo to work. Unfortunately my channels end up into compojure render phase and there cause an exception

Trying to figure out which part am I missing

I have required compojure.api.async so that it extends the necessary protocols

https://github.com/metosin/compojure-api/wiki/Async says that aleph needs manual bootstrapping. Is this still the case? This page has been updated in 2017

And if so, what that would actually mean?

I'm having hard time to understand what could possibly be needed

I mean, I have a fully working aleph server running, everything else works except async stuff

I would prefer using core.async since I need to do relatively complex concurrent querying which then will translate into the actual response which I'll return from my API

Guess I can just read the channel in a blocking manner just before I return something from my handlers but that sounds so lame...

The more I've dug into this the harder this problem seems to be. Now I've dumped Aleph and taken Jetty. Now I manage to serialize channels but unfortunately not via compojure-api but via my own system. However, what I've realized here is that it isn't strictly obvious how one should serialize the channel. For example, the main reason why channels are nice as response medium is that they can be written "as they come". So when it comes to JSON one should have one put operation per JSON array element. However, what if one wants to send just a JSON object without array wrapper? Should we just write an object if all we get from the channel is just one element? The answer is of course "no" since in that case we couldn't have one element long arrays which are definitely needed.

@niklas.collin https://github.com/ztellman/aleph/pull/442

Yeah, I've seen that

have you checked https://github.com/metosin/compojure-api/tree/master/examples/async

Yes

Also, do you have an idea regarding the problem I described above? 🙂

Basically sounds to me like an unsolvable problem without some additional information

oh, you would also need a streaming json writer

One could use spec definitions to deduce if the result should be an array or not

do you have large data sets you need to return?

Yes and no. No right now, yes in the future

So I could just ignore it at this point

But that feels... Dirty 🙂

if you need to stream data, I would use sse or web sockets and push separate full json data blobs out

And basically if I did that then the easiest solution would be just to consume the whole thing at handler level, do the serialization there etc since that's where I know what kind of structure the data should have

writing a large json-array in parts would require to write the json internally in parts too.

if the data is not big, just collect the bits via core.async and when it’s done, flush it out?

Yeah, that is practically the only thing I can think of which would make this in any way reasonable

Without excessive amount of work

Practically need to skip all the nice automatic conversions of compojure-api then though

or wait. Not really

yeah, you can’t do things like validate the output if you stream out the data in parts etc.

Right. Need to push this to higher abstraction level and do the work there

but, returning a stream of json parts is also a different contract to a returning a full data

you can still validate the parts manually in the handler, if you would stream it out.

Yeah

I have streaming use case right from the bat but fortunately it's a bit of a special case where I do not need to care about this stuff

Cheers. Needed some brain racking.

(flu, not at my best right now)

hey @ikitommi do you have any pointers on this? Something I could investigate

I would try to run the coercion directly with spec-tools to isolate all c-api things. There is st/decode (runs both coerce and conform), st/coerce (best-effort coercion with form walking) and st/conform coercion via conforming. Sorry, really busy with a client project, hopefully time on Friday to help with all the piled issues. Ping @mping

Here's the c-api code for it https://github.com/metosin/compojure-api/blob/master/src/compojure/api/coercion/spec.clj#L120-L135

No worries, I appreciate your efforts

I'll dive and see if I can spot anything shady