Clojurians Log v2

I’m trying to get CI/CD on GitLab, and I keep getting this error at the end of the log:

Syntax error compiling at (/tmp/form-init5582582439791414361.clj:1:73).
688could not find a non empty configuration file to load. looked in the classpath (as a "resource") and on a file system via "conf" system property

I’ve added the test-config.edn file used by luminus apps, and the pipeline script for CI/CD is running the tests with with-profile test so I think this means the tests will be run using the test profile, which uses the .edn file above to configure the database.

What could I be missing?

Have you tried running it locally the same way you thing it is running on the ci server?

Yes, with key word being ‘think’ lol.

I just found the error in the log is from cprop, so I’m going to dig into that library

So I dumped the config when running in the test env, and it’s full of settings…

But this last part makes me wonder:

Syntax error compiling at (/tmp/form-init408014767393427046.clj:1:72).

I can’t seem to track that down, since it’s it’s a CI/CD container

Whats the easiest way to encode/decode an Instant with transit?

I am sending a basically a database record to other nodes

it should "just work", but of course you are free to make your own writer for it if you prefer to translate it

Execution error at com.cognitect.transit.impl.AbstractEmitter/marshal (AbstractEmitter.java:194).
Not supported: class java.time.Instant

if there is a standard-ish extension i would use that

oh, I thought it was there out of the box

maybe i am using an old version

[io.pedestal/pedestal.service "0.5.8"]
     [com.cognitect/transit-clj "0.8.313"]
       [com.cognitect/transit-java "0.8.337"]
         [javax.xml.bind/jaxb-api "2.3.0"]
         [org.msgpack/msgpack "0.6.12"]
           [com.googlecode.json-simple/json-simple "1.1.1" :exclusions [[junit]]]
           [org.javassist/javassist "3.18.1-GA"]

I actually seem to be getting transit via pedestal

which is not ideal

yeah does not seem to be supported even on a newer version

i could always give up and just use serialization

i think that sends me straight the hell, don't collect $200

but...meh

https://github.com/ptaoussanis/nippy

whats the worst that could happen

it's not hard to add config for new datatypes to transit

I'm pulling up an old repo where I do that to put together an example

Execution error (ExceptionInfo) at taoensso.nippy/throw-unfreezable (nippy.clj:1001).
Unfreezable type: class next.jdbc.result_set$navize_row$fn__4512

yeah i actually get an interesting error

#:page_message{:id 7, :page_id_from 1, :page_id_to 1, :contents {:hello world}, :reactions [], :created_at #object[java.time.Instant 0x27c93821 2020-11-09T03:10:02.791910Z], :updated_at #object[java.time.Instant 0xe0fe300 2020-11-09T03:10:02.791910Z]}
clojure.lang.PersistentArrayMap

considering nothing in the data returned has anything but basic datatypes

;; ----------------------------------------------------------------------------
(defn- ->transit [data]
  (let [out (ByteArrayOutputStream. 4096)]
    (transit/write
      (transit/writer out :json
                      {:handlers {Instant
                                  (transit/write-handler
                                    "java.time.Instant"
                                    #(.toEpochMilli %))}})
      data)
    (.toString out)))

;; ----------------------------------------------------------------------------
(defn- <-transit [data]
  (let [in (ByteArrayInputStream. (.getBytes data StandardCharsets/UTF_8))]
    (transit/read
      (transit/reader in :json
                      {:handlers {"java.time.Instant"
                                  (transit/read-handler
                                    #(Instant/ofEpochMilli %))}}))))

but yeah, got it working

ahh, that would be it

The docs say >! "will park if no buffer space is available". What does "park" mean here?

Reason I'm asking is I'm getting Assert failed (< (.size puts) impl/MAX-QUEUE-SIZE) errors when producing a ton of input quickly

Is that the backlog of messages waiting to be put on the channel?

Never mind, found the explanation in go's docstring

Has anyone used https://github.com/mcohen01/amazonica for invoking Lambda functions? I can call invoke, but the payload in the response is in a HeapByteBuffer and I’m not sure how to translate that into something usable

Just a guess, but have you tried slurp?

@asko That’s a good question. I got the following error using slurp > Cannot open <#object[java.nio.HeapByteBuffer 0x2f40a23 “java.nio.HeapByteBuffer[pos=0 lim=281 cap=281]“]> as an InputStream.

Maybe you can use Java’s own java.nio.ByteBuffer

Examples are here: https://stackoverflow.com/questions/30046355/how-to-read-bytes-in-bytebuffer-with-clojure

@asko That’ll help me parse the buffer into a collection of longs. I don’t know what to do with this though. I’m expecting some JSON

Has anyone much experience with print-to-pdf using https://github.com/tatut/clj-chrome-devtools/blob/c294a69084c5e3e655d366bc34a85d232f34219a/src/clj_chrome_devtools/commands/page.clj#L1186-L1188 … it’s tantalizingly close to working for me, but I’m getting PDFs that are showing as blank pages.

I think the data’s there (a 1-pg PDF is coming in 200-400kb, a 2-pg is about 2x that). And I’m sure the headless chrome install is working: I can make pdfs with clojure.java.shell/sh.

For some background on how I’m making files https://gist.github.com/rgm/b3fb7c231ca41aad871eaf6115d7699f

I get different data sizes for different sites, and get 2 pages when I specify a 1-2 page range, so my working hypothesis is that the data is there, but something’s going wrong with a crop someplace.

@asko Here’s the code I ended up using to parse out the payload.

(defn byte-array->string [byte-array]
  (.encodeToString (Base64/getEncoder) byte-array))

(defn decode-byte-buffer [payload]
  (->> payload
       (.decode (Base64/getDecoder))
       (String.)
       (json/parse-string)
       (walk/keywordize-keys)))

(defn parse-payload [invoke-result]
  (->> (:payload invoke-result)
       (.array)
       (byte-array->string)
       (decode-byte-buffer)))

What was the name of the library that offers a novel, concise performant way to do map traversal? It's on the tip of my tongue, but I can't remember/find it.

@marcus.poparcus Do you mean meander maybe?

or specter

@borkdude Ah yes, thank you! It was specter, but I remember that I wanted to check out meander too!

Do you have a preference?

I don't use either actively at the moment, besides experimenting with this: https://github.com/borkdude/grasp#pattern-matching

I think specter has the best performance

Specter and meander are aimed at fairly different use cases and have fairly different performance characteristics. There are definitely cases where I'm sure specter beats meander in performance. But others where they can be more or less the same. We do consider performance a goal. And if you find something that is slower than you would like, always let us know in #meander or on github :) My recommendation is to try out all the things the clojure community has to offer. There are so many great libraries people have made for lots of different purposes. Try them out, learn from them, and see what is right for you. Or make your own and show us all your way of doing things :)

was just about to say, I find Specter and Meander complementary to each other and wouldn’t want to be without either of them!

Just to add something here. One of the things that often makes meander slower is that we care about safety and so do lots of predicate checks to ensure the pattern you have us is really why the data is. We do try to eliminate as much as when can and plan on having a way to opt out of that for performance sake.