Clojurians Log v2

I am making a Java library that needs to be extended by having custom transit read/write. It works except for the mapBuilder and listBuilder . It works fine without them:

Reader reader = TransitFactory.reader(TransitFactory.Format.JSON, in, this.c.getCustomReadHandlers(), this.c.getCustomReadDefaultHandler());
T data = reader.read();

Reader reader = TransitFactory.reader(TransitFactory.Format.JSON, in, this.c.getCustomReadHandlers(), this.c.getCustomReadDefaultHandler());
ReaderSPI reader = reader.setBuilders(this.c.getMapBuilder(), this.c.getListBuilder());
T data = reader.read();

cannot find symbol
symbol:   method setBuilders(com.cognitect.transit.MapReader<capture#1 of ?,java.util.Map<java.lang.Object,java.lang.Object>,java.lang.Object,java.lang.Object>,com.cognitect.transit.ArrayReader<capture#2 of ?,java.util.List<java.lang.Object>,java.lang.Object>)
location: variable reader of type com.cognitect.transit.Reader

Why aren’t you using transit-java?

Or are you

I am using transit-java yes 🙂 I making a java lib that uses transit-java. And a Clj lib on top depends on my-java-lib + transit-clj.

@nha why getCustomReadHandlers?

if i had to guess you know the types statically

why not just

public static final Map<Class<?>, ReadHandler> TRANSIT_READ_HANDLERS = Map.ofEntries(...);

I don’t know the types statically - they are open

wait 1 sec

(dogs)

okay so what do you mean by open

do you mean your library handles the serialization and people can pass in custom handlers

correct, 1min I’ll expand on what I am doing

I am making a library let’s say “my-java-lib”, and a clojure library built on top of it “my-clojure-lib”. The dependency graph looks like: my-java-lib • transit-java my-clj-lib • my-java-lib • transit-clj These libraries are in the message queues domain. Ideally my-java-lib can be used as a base for my-clj-lib and other JVM languages. So I would like the transit handlers and builders to be “open”.

I am able to set the transit write and read handlers just fine. Something like this in my-java-lib :

public setCustomReadHandler(Map<Class<?>, ReadHandler> readHandler) { ... }

Then my-clj-lib calls setCustomReadHandler and can “teach” my-java-lib how to serialize/deserialize Clojure Objects. And that works great already…. BUT

instead of Clojure {} I am getting back Java Hashmap in the Clojure lib when reading transit data. I believe the way to fix this would be by calling .setBuilders in a similar way but I cannot seem to be able to write that call in Java

hmm

honestly all i would be able to do would be done putting the . at the end and letting IDE auto complete guide my hands

though i am curious in what context you do the encoding/decoding

context = domain? I am writing a message queue library

like

okay messages are going in the queue

are they always in the queue as transit?

like

.put(Object o) -> o -> transit -> [... transit1, transit2, ...] -> .get() -> transit -> o -> Object o

and if so, what is the utility of having them be in transit in the queue?

like - what does your library do that makes it make sense to pick a particular data format for being in transit (pun unavoidable)

> are they always in the queue as transit? Yes > like - what does your library do that makes it make sense to pick a particular data format for being in transit (pun unavoidable) I guess you could pick any format but transit makes it convenient to avoid repetitive serialization/deserialization without having to define a schema

why not raw bytes and let the user do their own encoding?

(Thanks for thinking about this with me btw 🙂 ) It could be an option to have bytes[], and let the user do it’s own. But I was thinking that by pushing a preferred unified encoding the user experience would be better (and it would avoid the classic JSON + custom ser/deserialize transparently) - with maybe an escape hatch for raw bytes.

i mean, if you want to let java programmers use it you also need to consider the culture

it will be easier for people to include this in a project if they don't also need to sell people on a data format most people havent hear of

it is fine if it gives some benefit - like introspection on data in the queue or whatever

I was actually hoping that it would actually be easier to get started with transit hidden underneath because then it is already supporting all the Java primitive types. And writing custom Json encoding/decoding is even worse in Java than in Clojure. I actually thought about custom introspection/filtering too which a unified format makes easier.

how would json be worse than transit?

That would be the benefit of a unified format, not specifically transit though. For instance Json could work - but only on types that Json knows ofc so no Dates for instance.

most messages aren't primitives

unrelated, i've been doodling out a java serde impl

copying the library for rust

public interface Serializable {
    <Ok, Err extends Exception> Ok serialize(Serializer<Ok, Err> serializer) throws Err;

    static Serializable fromBoolean(boolean b) {
        return new SerializableBoolean(b);
    }

    static Serializable fromChar(char c) {
        return new SerializableChar(c);
    }

    static Serializable fromByte(byte b) {
        return new SerializableByte(b);
    }

    static Serializable fromShort(short s) {
        return new SerializableShort(s);
    }

    static Serializable fromInt(int i) {
        return new SerializableInt(i);
    }

    static Serializable fromLong(long l) {
        return new SerializableLong(l);
    }

    static Serializable fromFloat(float f) {
        return new SerializableFloat(f);
    }

    static Serializable fromDouble(double d) {
        return new SerializableDouble(d);
    }

    static Serializable fromString(String s) {
        return new SerializableString(s);
    }

    static Serializable forNull() {
        return SerializableNull.INSTANCE;
    }

    static Serializable fromUnsignedByte(byte b) {
        return new SerializableUnsignedByte(b);
    }

    static Serializable fromUnsignedShort(short s) {
        return new SerializableUnsignedShort(s);
    }

    static Serializable fromUnsignedInt(int i) {
        return new SerializableUnsignedInt(i);
    }

    static Serializable fromUnsignedLong(long l) {
        return new SerializableUnsignedLong(l);
    }
}

/**
 * Serializes a True/False
 */
record SerializableBoolean(boolean b) implements Serializable {
    @Override
    public <Ok, Err extends Exception> Ok serialize(Serializer<Ok, Err> serializer) throws Err {
        return serializer.serializeBoolean(b);
    }
}

/**
 * Serializes a single character.
 */
record SerializableChar(char c) implements Serializable {
    @Override
    public <Ok, Err extends Exception> Ok serialize(Serializer<Ok, Err> serializer) throws Err {
        return serializer.serializeChar(c);
    }
}

/**
 * Serializes a signed byte. (i8)
 */
record SerializableByte(byte b) implements Serializable {
    @Override
    public <Ok, Err extends Exception> Ok serialize(Serializer<Ok, Err> serializer) throws Err {
        return serializer.serializeI8(b);
    }
}

public interface Serializer<Ok, Err extends Exception> {
    Ok serializeBoolean(boolean b) throws Err;

    Ok serializeU8(byte b) throws Err;
    Ok serializeU16(short s) throws Err;
    Ok serializeU32(int i) throws Err;
    Ok serializeU64(long l) throws Err;

    Ok serializeI8(byte b) throws Err;
    Ok serializeI16(short s) throws Err;

...and so on

SerializeSequence<Ok, Err> serializeSequence() throws Err;

    SerializeMap<Ok, Err> serializeMap() throws Err;

    SerializeObject<Ok, Err> serializeObject(String name) throws Err;
    SerializeObject<Ok, Err> serializeObjectVariant(String name, String variantName, int variantIndex) throws Err;
}

import ser.Serializable;
import ser.Serializer;

public record Apple(int size, String color) implements Serializable {
    @Override
    public <Ok, Err extends Exception> Ok serialize(Serializer<Ok, Err> serializer) throws Err {
        return serializer.serializeObject("Apple")
                .serializeField("size", Serializable.fromInt(size))
                .serializeField("color", color == null ? Serializable.forNull() : Serializable.fromString(color))
                .end();
    }
}

Oh cool. > most messages aren’t primitives You mean that in your experience they are mostly simple java objects like Apple in your example?

they are usually java objects

in what i've done they have sometimes been "flat", but most of the time they had at least one list of things attached

but its always been value classes/pojos/data carriers/dtos whatever

like, one use case i had was syncing full "articles" between services

and i wrote a few value classes using immutables in java for that to decode json that came on the wire

another was more simple "chat messages" for live delivery - that was just text with metadata

i am def. not the person to ask about that domain though

Funny that 😉 I definitely saw and wrote some code passing “articles” or “chat messages” through some queues before. Mostly in Clojure and JS though, not so much in Java

Alright thanks a lot for your input @emccue 🙂 Getting late here, I need to think about it more but will leave it for now

Hey all. I remember there being a website where you could write the input and the expected output of a function. It would then search the core library for any matching functions. Anyone know what I'm referring to? I'm looking for a function that does the following:

Input:  (1 2 3 4)
Output: '((1 2) (2 3) (3 4))

Not sure about the site, but that looks like :

(partition 2 1 [1 2 3 4])

Thanks, that's the one 🙂

The site was probably this one https://borkdude.github.io/re-find.web/

https://borkdude.github.io/re-find.web/?args=%5B1%202%203%204%5D%202&ret=%5B%5B1%202%5D%20%5B3%204%5D%5D&more=true :)

Bookmarked, thanks 😄

if something seems like an area for performance concern, posting a repro (as minimal as possible) on https://ask.clojure.org would be appreciated

we have found and fixed hot spots in macro expansion before

in thread, i'll put the function that had previously been a macro that is called 3k times

(defn click-prompt
  "Clicks a button in a prompt. {choice} is a string or map only, no numbers."
  [state side choice & args]
  (let [prompt (get-prompt state side)
        choices (:choices prompt)]
    (cond
      ;; Integer prompts
      (or (= choices :credit)
          (:counter choices)
          (:number choices))
      (when-not (core/process-action "choice" state side {:choice (Integer/parseInt choice)})
        (is (number? (Integer/parseInt choice))
            (expect-type "number string" choice)))

      (= :trace (:prompt-type prompt))
      (let [int-choice (Integer/parseInt choice)
            under (<= int-choice (:choices prompt))]
        (when-not (and under
                       (when under (core/process-action "choice" state side {:choice int-choice})))
          (is under (str (side-str side) " expected to click [ "
                         int-choice " ] but couldn't find it. Current prompt is: n" prompt))))

      ;; List of card titles for auto-completion
      (:card-title choices)
      (when-not (core/process-action "choice" state side {:choice choice})
        (is (or (map? choice)
                (string? choice))
            (expect-type "card string or map" choice)))

      ;; Default text prompt
      :else
      (let [choice-fn #(or (= choice (:value %))
                           (= choice (get-in % [:value :title]))
                           (same-card? choice (:value %)))
            idx (or (:idx (first args)) 0)
            chosen (nth (filter choice-fn choices) idx nil)]
        (when-not (and chosen (core/process-action "choice" state side {:choice {:uuid (:uuid chosen)}}))
          (is (= choice (first choices))
              (str (side-str side) " expected to click [ "
                   (if (string? choice) choice (:title choice ""))
                   " ] but couldn't find it. Current prompt is: n" prompt)))))))

i'm unsurprised it's slow and don't think it's worth pursuing a reproducible build because of the size/number of uses

how much was being emitted in the macro case? that whole thing?

yeah, change the defn to defmacro and pepper backtics and tildes where necessary lol

certainly seems like it should be a function then

If there was some way to have is re-throw the failure or something, I could wrap the function call in a macro that merely says (defn click-prompt [& args] '(is (click-prompt-impl ~@args))), so the error would be on the right line but expansion would be small

Couldn't you wrap it in try-catch and rethrow the exception from the right place with the code that was generated by a macro? Alternatively, as hiredman has suggested yesterday - just walk up the trace and report the location of interest.

i have is calls inside the function, making it hard to not immediately print the error. i could change those, you mean? have them throw instead, and then catch and report the error?

If you throw instead of is and move is to the caller of click-prompt, yes.

cool, i'll try that out

to follow up on the conversation about the macro, this is what I've ended up with:

(defmacro error-wrapper [form]
  `(try ~form
        (catch Exception ~'ex
          (let [msg# (.getMessage ^Throwable ~'ex)
                form# (:cause (ex-data ~'ex))]
            (try (assert-expr msg# (eval form#))
                 (catch Throwable t#
                   (do-report {:type :error, :message msg#,
                               :expected form#, :actual t#})))))))

(defmacro click-prompt
  [state side choice & args]
  `(error-wrapper (click-prompt-impl ~state ~side ~choice ~@args)))

which lets me write in the validation function click-prompt-imp:

(throw (ex-info (expect-type "number string" choice)
                {:cause `(number? (Integer/parseInt ~choice))}))

slightly wordier than the previous is expressions, but keeps the compilation time around 45 seconds, vs 35 seconds with no macros and 70 seconds with the whole validation function as a macro

thanks for the help, both of you!