Clojurians Log v2

I’m querying a mongodb database which is returning items of the following kind:

({:_id #object[org.bson.types.ObjectId 0x79b7fb5 "6011a0b428dacb0004d7475b"], :name nil, :uri "dzhqroksvtvveqqnxmrclebakmighdlmkccpkidefupibqifziezasljxqvjkrmxfzuydyoucxozsbqxmodpjvvuzzkxjgckrkmz.mov"} etc...)

(map id->str data)

(defn id->str [item]
  (assoc item :_id (str (:_id item))))

It isn't a tagged literal, that is just how prn prints objects is doesn't know a more specific way to print

Depending on how you serialize the data to send it to clients etc, that may be the best place to do any transformation, since serialization has to walk the whole structure anyway

@hiredman does that mean that I have to know beforehand what fields contain ObjectId’s before doing the transformation, or is there a way to do that automatically?

I can’t convert every field to a string either because some are numbers

https://github.com/clojurewerkz/monger.docs/blob/master/articles/integration.md#cheshire-or-clojuredatajson

For objects that print like #object[org.bson.types.ObjectId 0x79b7fb5 "6011a0b428dacb0004d7475b"] the format is like #object[JVM_CLASS SYSTEM_IDENTITY_HASH TO_STRING]

So you would walk the data structure, perhaps using something like clojure.walk, looking for instances of ObjectId

Or use a something in your serialization code that allows you to customize the serialization based on types

@hiredman what about the link that you sent? I required monger.json but that didn’t work

did you add cheshire?

> To use it, you need to add Chshire dependency to your project

yes that’s a dependency

and did you require monger.json?

(json/generate-string (mc/insert-and-return (mg/get-db conn "monger-test") "documents" {:name "John" :age 30}))
"{\"_id\":\"6049ca80360cf43286348620\",\"name\":\"John\",\"age\":30}"

user> (require '[monger.core :as mg])
nil
user> (def conn (mg/connect))
#'user/conn
user> (require '[monger.collection :as mc])
nil
user> (mc/insert-and-return (mg/get-db conn "monger-test") "documents" {:name "John" :age 30})
Mar 11, 2021 1:42:34 AM com.mongodb.diagnostics.logging.JULLogger log
INFO: Opened connection [connectionId{localValue:2, serverValue:3432}] to 127.0.0.1:27017
{:_id
 #object[org.bson.types.ObjectId 0x50f01cbb "6049c9ea360cf4328634861e"],
 :name "John",
 :age 30}
user> (require 'monger.json)
nil
user> (mc/insert-and-return (mg/get-db conn "monger-test") "documents" {:name "John" :age 30})
{:_id
 #object[org.bson.types.ObjectId 0x462cd22d "6049c9f0360cf4328634861f"],
 :name "John",
 :age 30}
user> (require '[cheshire.core :as json])
nil
user> (json/generate-string *2)
"{\"_id\":\"6049c9f0360cf4328634861f\",\"name\":\"John\",\"age\":30}"
user> 

some logging and other stuff omitted

@dpsutton however, this involves making changes in the client, because earlier I was sending an edn and now it’s a string.

albeit the edn gets converted into json, which is not the case here

so I resorted to doing: (-> data json/generate-string json/parse-string)

If you are sending edn I would likely use clojure.walk to transform the structure before printing. You could add a print-method for ObjectId, but that is a global thing (I try to avoid that kind of thing)

the idea was something like have the "queue" be a producer that yields when someone takes from it

Hello everyone! Don't know where to post this, can someone point me out to a clj-commons/manifold channel? Or can I ask help over here?

There doesn't seem to be such a channel, so here should be fine.

Thanks, I'll write my question then

I'm using manifold and I can't wrap my head around splitting a stream: here's an example code:

(let [pos-stream (ms/stream)
      neg-stream (ms/stream)]
  (->> value-generator 
       (ms/->source)
       (ms/map (fn [value]
                 (if (pos? value)
                   (ms/put! pos-stream value)
                   (ms/put! neg-stream value))))
       ; I need to fill streams, how should I do that? 
       (ms/consume identity)
       )
  
  (->> (ms/zip pos-stream neg-stream)
       (ms/map (fn [[pos-value neg-value]]
                 (process pos-value neg-value)))))

Or is this fine since you can't put more than one element in the stream until it's taken away?

You could try #aleph if you don't get a response here.

Thanks! I'll try there if no reply comes here

I'm assuming the answer is no, but worth a try: Is there a trick to "disable" an interface on an object instance after the fact? E.g. (def obj (reify Interface1)) (instance? Interface1 obj) ;; true (disable! obj Interface1) (instance? Interface1 obj) ;; false

No, I don’t even think that’s possible via java agent trickery

The problem I'm struggling with: I'm trying to support reify in a clojure interpreter that runs in an environment that doesn't allow you to define new classes. The trick I've used is to support reify with a fixed list of interfaces. A compile time function creates instances that implement all of the interfaces, but the instance checks at runtime for methods it has available provided by the caller of reify. This works well. Until... you pass those objects to core functions that have a series of instanceof checks. E.g. the objects always implement IFn so (fn? obj) always returns true, even if you didn't provide an implementation for IFn, since this is a compile time decision.

I'm inclined to roll back this idea, unless there is a way out.

The other option I've considered is to support groups of interfaces that are commonly used together and then you'll have to implement them all

The problem there is that if we add an interface to one of the groups, it will break existing callers.

I have also tried the "every subset" approach, but this blows up very quickly. For 10 classes this requires pre-generation of 1024 classes. For 100 classes it's already not feasible anymore.

Have you looked at java.lang.reflect.Proxy?

Probably it counts as defining a new class tho :thinking_face:

https://www.graalvm.org/reference-manual/native-image/DynamicProxy/

> Therefore all dynamic proxy classes need to be generated at native image build time. Same kind of problem, but there you compile the image yourself, so the classes are adapted to your specific use case. In the above case it's a general use case.

Seems so

So the main problem is false positives of instanceof towards the true side and runtime exceptions that result from it with "not implemented". E.g. when a data structure doesn't explicitly implement Counted you will get an exception for (count obj), because count thinks it's a Counted instance (it is, but it is not)

Clojure kind of relies on Java being Smalltalk in C++ clothing. When it becomes C++ with GC, you are going to have a bad time with reify etc.

for sure

I guess you could patch clojure to replace instanceofs with something that has enough indirection but that does not feel very promising either

oooh haha, I could patch the Clojure that goes into babashka's compiled artifact. Hmm, that would be a major hack.

but even if I would do that, I would have to patch any other library that is bundled with babashka as well, if they do manual instance checks. not going there

it may all not be worth supporting reify with interfaces in bb. Protocols work relatively well since they have a smaller surface area

I wonder if these issues would just go away with Truffle. Even if they did, it would be a lot of work as well

There is a Java on Truffle interpreter now that may allow for these things

joinr did some experiments on getting the clojure compiler running. It's terribly slow though

But this could change in the future maybe

https://epub.jku.at/obvulihs/download/pdf/501665?originalFilename=true

> TruffleClojure: A self-optimizingAST-Interpreter for Clojure > Master’s Thesis

yeah, I've seen that. the source code isn't available and it's based on a very old truffle version

Indeed

I've been thinking of writing my own, but this will be another major project and it's not certain that this will solve most of these problems

https://github.com/oracle/truffleruby/blob/master/doc/contributor/native-image.md seems promising

It has the same kind of limitation: > Java interoperability works in the native configuration but requires more setup. First, only for classes loaded in the image can be accessed.

I'm not an expert in manifold, but what I would think to try first is putting all values from the generator on one stream, and then filter the one stream

I assumed that making a Ruby class that extends a Java class (that is in the image) would work

You're assuming that they implemented Ruby classes using Java class hierarchies, which doesn't have to be the case

But maybe they do

The Ruby object model has more stuff but I it would not be very useful if you could not inherit from Java classes

Thanks for your help! I don't know if I'm doing it wrong but filtering doesn't work for me. The first filter who discards messages will not propagate to the other stream.

(defn split-stream [condition stream]
  [(ms/filter condition stream)
   (ms/filter (comp not condition) stream)])

I sometimes think of making a new TruffleClojure when I get ahead of myself

Generating Java Classes at Runtime JRuby supports converting a Ruby class to a concrete Java class using become_java!. TruffleRuby does not support this. We recommend using a proper Java interface as your interface between Java and Ruby. That's also interesting.

If they allow creating "reified" objects that implement interfaces at runtime though, then I want to know if that is possible at runtime in a native image.

You can see a full working example on #aleph a value generator can be as simple as rand

I posted a question in their channel: > Does TR allow creating objects that implement Java interfaces at runtime, in a native-image? How is this achieved?

when you say it doesn't work, what does that mean?

You could also just try it

if I use two filter with a single stream and opposite conditions it loops forever.

trying now

Using two streams and consuming them I get the following warning:

WARN manifold.stream.default - excessive pending puts (> 16384), closing stream

Which makes sense since the generator is infinite

Hmm, where in the source do you indicate that you're implementing that interface though?

class EthylAlcoholFluidForce
  def getFluidForce(x, y, depth)
    area = Math::PI * x * y
    49.4 * area * depth
  end
end

https://github.com/oracle/truffleruby/blob/master/doc/user/jruby-migration.md#implementing-interfaces

It's weird, why can you compile TruffleRuby into a native image but not ye olde classloaders and bytecode JIT? I think Jikes RVM was more flexible than this 20 years ago...

Ah what they do there: on the Java side they create something which implements the interface which then delegates to the Ruby methods

this is what I also do with the reify trick, only the list of interfaces each object implements is too broad

Sort of like gen-class

Hmm. Pure Clojure libraries might not be too bad. Any interfaces that correspond to protocols can be skipped in favor of the extend-protocol mechanism (I assume you already do this) and instance? could be easily replaced. But with Java code some craziness like patching bytecode or Java on Truffle would be required.

@nilern Yes, instance? can be replaced, but not for built-in libraries (without changing them) that are part of the babashka std lib

I meant replacing the Clojure instance? in addition to the sci one

This still won't help for functions that call into clojure.lang.RT

The surface area of some of these interfaces is just too big to start doing this, I think

I think there is no 100% solution. Like IFn does not have overloads for primitives beyond arity 4. In such situations I try to focus on making the tool useful and the limits clear.

I realize there is no perfect solution, but I have to choose between making things work + unexpected bugs, or not support it at all E.g. (fn? (reify ILookup (valAt [this] :foo))) ;;=> true

as long as the reified object has limited usage, this might be fine. I'm not entirely sure about the consequences if you actually call the above object as a function, you will get "not implemented" as a runtime error

I think it is better to have very little support or no support at all than unexpected bugs. But I am probably biased by the static typing literature where undecidability and especially unsoundness are to be avoided like the plague.

Yes, I tend to agree with this

hey everyone, been using windows forever and just got an m1 mac so I'm not tied at home. is everyone using the x64 adoptjdk8 for datomic/shadow-cljs?

Oh just found this: https://www.azul.com/downloads/zulu-community/?version=java-8-lts&os=macos&architecture=arm-64-bit&package=jdk

I use OpenJDK from my package manager. I used to use https://brew.sh/ on Mac.

Is there anything that allows you to write Clojure[Script] code directly in <script type="text/clojure"> HTML tags? Like what Brython does for Python. https://brython.info/

@clojurians-slack100 Possibly with self-hosted #clojurescript, Klipse (= self-hosted CLJS) or #sci

I may have seen a demo years ago. It would require self-hosted, which you usually don't want to ship

Why not? Size?

This site uses #sci https://nextjournal.github.io/clojure-mode/ which is probably the smallest bundle size

However, you can get better perf and bundle size if you just compile a .js file custom for your site

Of course 🙂

Yes and running the compiler on every page load would also be slow (and a bit weird)

Are there any Clojure webassembly projects or plans?

https://hasgeek.com/inclojure/2020/sub/wasm-on-clojure-6CuxGMcGY4otVcGgyJSBF7

related but this is not compiling clj to wasm

I suppose GraalVM is getting us closer to this...

Hi all, I seem to never remember if there is way to get a spec, say

(s/def ::event
  (s/keys :req-un [::id ::payload]))

(s/and ::event ...)

where ... might need to be a predicate that looked inside the map

would that be the preferred route over a multi-spec (I have just recalled I could go that route)

this is your solution implemented

(s/def ::file-id ::model.file/id)
(s/def ::file-deleted-payload
  (s/keys :req-un [::file-id]))

(s/def ::file-deleted-event
  (s/and ::event-bus/event ::file-deleted-event-payload))