Clojurians Log v2

Curious whether gfredricks’s idea might work. For literal check for fn, for var use metadata.

Hard to see what having to tag vars improves over the sam macro above, in both cases you have to specify the Sam type.

In theory, it gets vars to play too.

But they were talking about compiler changes, not a macro.

don't have to specify the method in the var tag case

which might be a bigger deal than just less typing -- the interface name might be a lot more memorable than the method name

I'd like to set a challenge to do it with neither method nor interface name

Thread.currentThread().setUncaughtExceptionHandler((t, ex) -> body...);

(.setUncaughtExceptionHandler (Thread/currentThread) __?__)

has there been any work done on making clojure objects (e.g. functions, anything reified) efficiently serializable between processes which are running the same code?

seems like this would need deep compiler integration in order to correctly account for closures

to what end?

like create an object in one process and send it to another process on another machine

that's running the same code

like CORBA?

not very familiar with that

or Java serialization?

specifically, I need to be able to take a function instance and send it to another process

send data and eval it

or an instance of an object defined via reify that has some random closure

actual data or a sexp

that seems too inefficient, and how would you send the closure?

but why?

@ghadi I’ve faced this a lot of times

seems there are plenty of reasons

distributed processing especially

yes, my case would be distributed processing

it’s not always super trivial to just eval it on the other side either

anyway, just wondering if there was some branch of clojure somewhere tackling this or any related work

1) have to setup the same context around it (maybe not trivial), 2) I’ve hit issue with max method size errors trying to eval across as well - depending on what you need to send.

there's some examples of serializing functions in portkey

using java serialization IIRC

I remember doing some extensions and improvements over this https://github.com/technomancy/serializable-fn in a private repo in the past

but it was based around just eval’ing forms later, but could get a bit more sophisticated with it.

I think the standard Serializable fails for certain closure situations.

(also, I’ve only did work on any of this for function instances, not things liek reify)

oh and lastly - concerning eval

i'll take a look at portkey

it is quite slow to eval a bunch of times. you want to “batch eval” but in a way that can not break method size limits

yea

We did some stuff with this on the clara-rules serialization stuff

the efficient way to do this would be to serialize an id for the class and the values in its closure

It was super slow to eval tons of “fn’s back again”, so we made a big data structure to hold them all in to do a “batch eval and then relink them to their appropriate places after”

Nope, kryo

data does not have edge cases: maps, sets, lists, scalars

👌:skin-tone-4:

sure, but there are useful applications …

you can send bytecode, you can have a classloader that is network-controlled

sending arbitrary things is an arbitrary requirement, and is going to be challenging

Sometimes, the customer really wants a machete, and all you can do is point out the sharp edges and hope they don't return it later with a bandage wrapped around their hand.

> Sometimes, the customer really wants a machete, and all you can do is point out the sharp edges and hope they don’t return it later with a bandage wrapped around their hand. eh, I don’t really consider this constructive here

distributing work across processes is common and this comes up

There are several clj libs out there doing things around this. Ones built around Hadoop stuff/Spark etc

But yeah, I’ve specifically targeted fn’s before, not just “any object”. I know there will be limitations

and I’m not even sure I think there is a “good solution” for clj alone. Basically I’ve always came back to the eval approach, but then have to do things like batching forms together etc

if you don’t want it to be extremely slow for larger things

http://paul.stadig.name/2009/03/clojure-terracotta-we-have-repl.html was a thing

networked style classloader etc, seems interesting indeed

which is an interesting question, if you want to send closures around, why not just turn all your machines in to a single shared memory space

even with the very general problem statement expressed, I still think you shouldn't pass around closures

if you want to send code, have a control plane be in charge of worker machines or classloaders from above

roll the code forwards or what not, but limit what's passed in between workers to be data

unless that's impossible for the SLA, in which case send the code to the data like Datomic Ions do

it works, and it's Really Fast

I was actually working at Terracotta at this time, helping Paul, before I was a Clojure user :)

working on sending function instances is not something we're likely to do in Clojure proper anytime soon

working on being able to send rehydratable var refs is something we're working on

vars are actually serializable now (since 1.9? 1.10? can't remember) and when deserialized, they will re-resolve in the target

very cool

what we're looking at is making the reader support that too

so can read a var and have it become a resolved var again

all seems reasonable. I just chimed in based on some past experiences anyways. fortunately for me, I haven’t been fighting with this situation in recent times.

this var addition seems interesting

the other place this pops up (thorny serialization issues) is image based development

the var serialization stuff is still a reference to the var, not the actual contents

https://clojure.atlassian.net/browse/CLJ-2165

that might have been my first intro to Clojure actually

Nice!

@nathanmarz I've been working on a similar thing in CLJS [tau.alpha](https://github.com/johnmn3/tau.alpha/tree/master/src/tau/alpha) that I've recently been working on porting to nodejs.

And I've been re-working the fn serialization stuff and it works pretty much like shipping around byte code

but javascript as bytecode 🙂

And in a web worker env or a tightly controlled cluster, I think it makes most sense to use a fully connected mesh so they appear to have a single shared memory space

like @hiredman was saying

the more recent version for node is aware of locals and grabs them too, if necessary, so the code looks a little more traditional. In the code above, that version uses a more explicit binding conveyance mechanism for the on macro. Anyway, ping me if you have any questions about it. I'll hopefully have a rough nodejs version out soon

@john cool, my case though is specifically that the code itself is shared between processes

so no need to send byetcode, since it's already there

just need to send an id for the class and whatever the fields are, which would be the closure

since you're not dynamically generating code, you don't need to do any of this: which ever class that generates the closures should be the target

ask instances of that class to generate the closures

(aside: this is no different semantically than passing around RPC-style maps with a target "op" key)

@nathanmarz that's sort of how I'm doing it though. The reason what I'm doing mostly works is because the same compiled code is on both sides

The byte code becomes mostly the serialized calling convention

You can just use tagged literals for the IDs and hydrate them with the read function on the other side

@ghadi how do you ask instance of a function to generate its closure?

you mean with reflection?

no i mean call a Regular Function that returns a closure

with normal arguments, return function that closes over arguments

do you mean to write or generate the functions that need to be serializable in a special way?

(defn callme
  [a b c]
  (fn [x]
    ...use a b c))

not sure what you mean by that example

the code I need to be able to write would be like this:

(defn foo [a] (fn [] a))
(def f (foo 1))
(defn bar [a] (reify SomeInterface (someMethod [this] a)))
(def r (bar 2))
...
(send-to-other-process (serialize f))
(send-to-other-process (serialize r))

I get the impression that you're focused on mechanism and might need to step back into the problem space

if you control code on all nodes, you don't need to send code

send data that can recreate the code

yes, that's been established

the question is how do I take an instance of an arbitrary function and send that data

the data would be the class and the fields that comprise that instance (the closure)

or an instance of some object made with reify

send data that can recreate the objects, you can use metadata on the objects

so how would you imagine my code sample working then?

foo has to annotate its return with metadata as to what is in its closure?

If foo and bar can be custom types, you can customize the print writer so that the other side knows to call foo and bar constructors on them on the other side. Assuming you control foo and bar.

no, that's not the case

this needs to work for any function instance or reify instance

it sounds like the answer to my question is there has been no experimental work on the clojure compiler for this

Shipping code between run times is a generally frowned upon practice. think there hasn't been a lot of interest in it until recently

it's one of those things that every generation re-learns is a bad idea

where is the #corba channel? 😉

must have died in the late 90's

Yeah but latency changes

At some point we'll have to reevaluate the tradeoffs