Clojurians Log v2

Is there a widely-used, well-documented library for implementing role-based access control in Clojure? I imagine it's often pretty domain-specific and mostly just simple checks on maps/sets etc., but wanted to see if there was something nice out there.

What are you trying to achieve exactly? Find the biggest item in a collection by some definition?

Keeping only the biggest n items in a reducible collection. Imagine it’s a million rows from a db each with a score. Want to keep the largest scored values without requiring the entire result set and sorting to get the largest items

I'm trying to attach some metadata to a defed fn inside of a macro, but nothing quite seems to stick:

(let [doc "docs"
        meta {:arglists '([test])}]
    `(def ^~meta foo ~doc (fn [x] 1)))
;; => error: Metadata must be Symbol,Keyword,String or Map

But when I check, it says it's a map:

(let [meta {:arglists '([test])}]
    `[~(type meta)])
;;=>  [clojure.lang.PersistentArrayMap]

Am I just barking up the wrong tree here?

try: ~(with-meta 'foo meta)

I think it's trying to apply the metadata '~ to the symbol meta in your example

> (defmacro test-fn-meta []
  `(def ~(with-meta 'foo {:a 1}) (fn [])))
> (test-fn-meta)
#'my.ns/foo
> (meta #'foo)
{:a 1,
 :line 929,
 :column 18,
 :file "*cider-repl examples/browser:localhost:49996(clj)*",
 :name foo,
}

ah, that makes perfect sense. I'm adding the meta to the var, I had tried wrapping the fn before but that obviously doesn't do what I'm aiming for

as an improvement, defn allows an optional map between the name and params that will be added to the metadata:

(defmacro test-fn-meta []
  `(defn ~'foo ~{:a 1} [x] 1))

I had tried that, but in my (brief) test I couldn't override the :arglists, it just used the ones from the actual arg vector

actually. it's probably trying to add the metadata ~meta to the symbol foo

let me try with this method, though

ah, that works!

nice

So are you OK witn O(N) performance? If that’s the case, perhaps a simple (filter …) transducer ?

i don't know how i could filter. given that i don't know a priori what some threshold will be

just that i want the largest N items. i don't have any fore knowledge of how to determine those N items until i pass a function over them

i certainly can't beat O(N) in the general case. i think i'll hit O(n log(n)) in the worst case scenario where its essentially just heap sort but i'm discarding values so i have much smaller memory footprint

but that's the point of using a heap here. i don't want to track order or sorting, just a collection of items with constant time access to the min so i can throw that one away if necessary

ah I see, largest N as opposed to a pre-determined threshold, hmm

correct. that's why it needs to be in the reducing function to manipulate the concrete collection rather than in a transducer

Maybe just (assoc …) all the items into a sorted map and (take N …)

(sorted-map-by …)

that realizes the whole collection. if there are a million my server falls over

or continually resorts a collection. you're essentially using it as a heap whereas i'm using the proper heap datastructure here. constant access to smallest item, no sorting of the rest of the collection

Is there a way to get the data sorted out of the database? Or that’s not a good option?

(assuming it’s coming from a database)

no. scoring things in code based on the rows from the db. far too complicated to implement that in sql

Have you seen this? https://github.com/clojure/data.priority-map

(just looking for a more idiomatic solution as opposed to using the Java option)

yeah i looked at that. i forget what it lacked

ah, it was ordering. it maintains the priorities to the items but doesn't expose a way to quickly get the lowest priority item

also, no good way to evict the lowest priority. just add new ones in.

i mean, just abstractly a heap is all that i want with nothing that i don't want

ok, I got everything working the way I wanted in clj, but when I tried using it in cljs I found a weird behavior with arglists:

(defn bar {:b 2 :arglists '([damned lies])}
  [x] 1)
(meta #'bar)
;;=> {:ns user, :name bar, :file "user/main.cljs", :end-column 10, :source "bar", :column 1, :line 1, :b 2, :end-line 1, :arglists ([damned lies]), :doc nil, :test nil}
;; this is just like clj^

(defn foo {:a 1 :arglists '([lies])}
  ([x] 1)
  ([x y] 2))
(meta #'foo)
;;=> {:ns user, :name foo, :file "user/main.cljs", :end-column 10, :top-fn {:variadic? false, :fixed-arity 2, :max-fixed-arity 2, :method-params [[x] [x y]], :arglists ([x] [x y]), :arglists-meta (nil nil)}, :source "foo", :column 1, :line 1, :end-line 1, :arglists ([x] [x y]), :doc "docs", :test nil, :a 1}

It looks like multi-arity functions will override the :arglists supplied in the metadata with the ones it figures out on its own

I'm up way too late, so I'll check in later, but this seems weird to me

b-+

according the docs, cljs has no reified vars, https://www.clojurescript.org/about/differences#_vars_and_the_global_environment the truth is a little bit trickier as there's been a few improvements that address some of the use cases of reified vars. overall, I've found it's worth avoiding vars in cljs if possible. That may change as they keep improving cljs

(require '[net.cgrand.xforms :as x])

(let [data (repeatedly 10000 (fn [] {:score (rand-int 1000)}))]
  (sequence
    (comp
      (x/sort-by :score >)
      (take 3))
    data))

yeah it does. it just keeps all of your items in an array list and on the completing single arity of a transducer it calls (java.util.Collections/sort cmp)

(defn sort
  ([] (sort compare))
  ([cmp]
    (fn [rf]
      (let [buf #?(:clj (java.util.ArrayList.) :cljs #js [])]
        (fn
          ([] (rf))
          ([acc] (rf (core/reduce rf acc (doto buf #?(:clj (java.util.Collections/sort cmp) :cljs (.sort cmp))))))
          ([acc x] (#?(:clj .add :cljs .push) buf x) acc))))))

Yes, I give up 🙂

Depending on the database you’re using, I would still try to get all the items in a sorted order first… That would actually prevent you from having to go through the 1 million items in the first place (even if you’re not keeping them all in memory).

can't. its for searching and there are a few different strategies to rank them

need rows and then run functions over the different columns using different ways to rank them

I see… you need all the items for other purposes.

i agree if we could ask the db engine to rank them and then just take that

well doing things like longest common subsequence between search term and the columns doesn't sound fun in sql

but i appreciate you going through this with me. for sure i'd prefer a native clojure immutable heap or something similar

but those java util collections are rock soid

solid

I am just suggesting asking the DB to do the sorting… I 100% agree about not doing most things in SQL, it would be a mess.

but mutable 😞

the values they sort by are computed dynamically according to search term. no way to ask the db to sort it

I see.

There's no function to quickly check if a collection has duplicates, is there?

frequencies?

(apply distinct? [1 2 3 4 2])
=> false
(apply distinct? [1 2 3 4])
=> true

would prefer this over frequencies since it will short circuit on first repeat

That's the one, thanks! I used borkdude's re-find but I didn't guess to try vararg as opposed to a single collection.

How can I split a string by (char 1)

;;; works, is there a better way?
(vec (.split (str "asdf" (char 1) "asdf") (str (char 1))))

;;; these don't work
(str/split (str "asdf" (char 1) "asdf")
  #"\u00001"
  ;;#"[\\1]"
  ;;#"\\x01"
  )

(let [s "asdf"] (str/split s (re-pattern (str (char 1)))))

if i understand what you're after

I get the same result for these two subexpressions. You do not?

user=> (= (vec (.split s1 (str (char 1)))) (str/split s1 #"\u0001"))

And in fact, for your first and third options that you say didn't work for you:

user=> (require '[clojure.string :as str])
nil
user=> (def s1 (str "asdf" (char 1) "asdf"))
user=> (def a1 (vec (.split s1 (str (char 1)))))
#'user/a1
user=> (def a2 (str/split s1 #"\u0001"))
#'user/a2
user=> (def a3 (str/split s1 #"\x01"))
#'user/a3
user=> (= a1 a2 a3)
true

ohh I had an extra 0 in the \u0001 str, that one works for me

(Note that inside a Clojure string surrounded by "", you need to type \\ to get a single backslash. Inside of #"" you only need to type one backslash to get one backslash in the regex string -- the Clojure reader parses regex strings slightly differently than regular strings, as a convenience to avoid having to type so many double backslashes)

Oh yeah \x01 does work with the single backslash

but that can lead to more confusion if you weren't aware of it (and I just double-checked -- this fact is documented in official docs here if you search for regex: https://clojure.org/reference/reader)

if you have a function that will end up reifying a java.util.Comparator, do you ask the caller to supply a function satisfying the <0,0,>0 contract that a comparator expects and reify it inside the function or do you have the caller just pass you a java.util.Comparator and not worry about it since its clear at the call site that you have that contract?

i guess, do you make everyone go through the ceremony of reify'ing or take a function and do the reify in one spot for conveinece to the caller

functions are Comparators

user=> (instance? java.util.Comparator (fn []))
true
user=>

https://github.com/clojure/clojure/blob/master/src/jvm/clojure/lang/AFunction.java#L50

oh get out of town. awesome

https://clojure.org/guides/comparators for 10x more detail

What Java version do folks recommend or are using for prod? We're looking to upgrade from Java 8 but aren't sure how high is safe to go. Obviously we'll test things out well before we ship, just looking for a recommendation on a target.

My feeling on Java versions is the same for Clojure projects as it is for Java projects: if you want to play with the non-LTS releases, that’s great; if it’s for production code, stick with the LTS Javas: 11, 17, 23… Not everyone feels the same way, of course. If you’re in the Java space, you might want to play with some of the features but in Clojure dev, I think it’s even less compelling as it’s not like there are significant VM changes occurring.

I agree. Even though we are using Adopt OpenJDK versions, rather than Oracle's versions, we feel more comfortable with 11 in production and plan to move to 17 at some point. In dev we've been testing against 11, 14, and I'm currently using 15 -- but I can easily run tests against multiple versions just by using an env var (I have four or five different JDKs installed locally).

Out of curiosity did you run into any issues with 14 or 15?

as in did they not "just work"

I don't recall hitting any problems when I moved up to JDK 14, except for some minor changes in locale-specific number formatting (which I think were bug fixes -- but they broke a couple of our tests).

But locale-specific number formatting has changed a little across several recent JDK versions so that wasn't entirely surprising.

The big jump was from 8 to 11 (because of all the changes in 9). Since then it's been pretty smooth sailing.

Yeah same here basically (on much smaller codebases)

Java 10 or 11 introduced some important changes if you are running in a container (making JVM pay attention to container settings)

so if you're doing that, it's almost definitely worth upgrading from 8 to 11

as of this year, the clojure community has shifted from majority 8 to 11

We still have to maintain JDK 8 compatibility in most of our code because we have a legacy app that we're still slowly rewriting to Clojure and it won't run on JDK 9+ (it's a mix of Clojure and... other stuff... 🙂 ).

officially, we test and recommend Clojure only on LTS Java releases. unofficially, I test it with each release and am not aware of any issues on recent versions

What's the simplest way to get edn/read-string to throw or some other way to validate if the EDN file contains a valid data-structure but atfer it has junk?

config.edn:

{:a 1
 :b 2}
 :c 3}