Clojurians Log v2

laziness is not a promise that as few as possible elements are realized, it's the possibility that fewer than the total number are realized

if your program logic depends on realizing only N items, either use take before map, or don't use laziness - laziness and side effects mix poorly in general

I have a vector of maps, each map has a key :id, i’d like for this vector to actually be a map from the :id to a map of the rest of the data in the map

is there a cool flatten trick here

extract, pick something

or do i create an empty map and loop through and assoc in? doesn’t feel very functional

maybe reduce.

(into {} (map (juxt :id identity)) your-coll)

most codebases end up with some utility function that's u/key-by that does this to some effect. up to you how you resolve if there's the same id in more than one of the maps. first wins, last wins, etc

oooo

thanks!

this is really awesome, juxt combines the functions :id, which is a keyword but why do you need identity?

oh that’s the piece that puts the rest of the map in as the value

((juxt :id identity) {:id 1 :other :stuff}) #_ -> [1 {:id 1, :other :stuff}]

yeah. you make tuples of [k v] and then pour those (into {})

that’s awesome, thanks!

This is one of those things where you often hear people say "what use is a function that always just returns its argument?" and here's a great example.

yeah, I would have asked that question too! but that’s really useful 🙂

When should one use Exception. vs ex-info ?

when consumer of your function expects to see specific Throwable class such as Exception

Thanks!

hi, how can I keep-order when having large elements using group-by? grouping vec (when order matters) is lost on large elements size (result turns to persistent.hashmap rather than persistent.arraymap) i;m interested in the index-ordering to be kept (as in this example)

(->> [{:a 2 :b 2} {:a 1} {:a 1 :b 1}]
     (group-by :a))
     
=> {2 [{:a 2, :b 2}], 1 [{:a 1} {:a 1, :b 1}]}

maps in clojure are not ordered in general. you could coerce result of group-by into sorted-map using (into (sorted-map))

(->> [{:a 2 :b 2} {:a 1} {:a 1 :b 1}]
     (group-by :a)
     (into (sorted-map)))

Not sure what I'm doing wrong exactly but I can't seem to get this clojars library working with a minimal example https://github.com/kahuin/kahdemlia I added [kahdemlia "0.1.0-SNAPSHOT"] to my project.clj and tried to run

(ns ds-test.core
  (:require [kahdemlia.core :refer [make-node!]]))

but i will get

2. Unhandled clojure.lang.Compiler$CompilerException
   Error compiling kahdemlia/core.cljc at (1:1)
   #:clojure.error{:phase :compile-syntax-check,
                   :line 1,
                   :column 1,
                   :source "kahdemlia/core.cljc"}
             Compiler.java: 7648  clojure.lang.Compiler/load
                   RT.java:  381  clojure.lang.RT/loadResourceScript
                   RT.java:  372  clojure.lang.RT/loadResourceScript
                   RT.java:  459  clojure.lang.RT/load
                   RT.java:  424  clojure.lang.RT/load
                  core.clj: 6126  clojure.core/load/fn
                  core.clj: 6125  clojure.core/load
                  core.clj: 6109  clojure.core/load
               RestFn.java:  408  clojure.lang.RestFn/invoke
                  core.clj: 5908  clojure.core/load-one
                  core.clj: 5903  clojure.core/load-one
                  core.clj: 5948  clojure.core/load-lib/fn
                  core.clj: 5947  clojure.core/load-lib
                  core.clj: 5928  clojure.core/load-lib
               RestFn.java:  142  clojure.lang.RestFn/applyTo
                  core.clj:  667  clojure.core/apply
                  core.clj: 5985  clojure.core/load-libs
                  core.clj: 5969  clojure.core/load-libs
               RestFn.java:  137  clojure.lang.RestFn/applyTo
                  core.clj:  667  clojure.core/apply
                  core.clj: 6007  clojure.core/require
                  core.clj: 6007  clojure.core/require
               RestFn.java:  408  clojure.lang.RestFn/invoke
                      REPL:    1  ds-test.core/eval19109/loading--auto--
                      REPL:    1  ds-test.core/eval19109
                      REPL:    1  ds-test.core/eval19109
             Compiler.java: 7177  clojure.lang.Compiler/eval
             Compiler.java: 7166  clojure.lang.Compiler/eval
             Compiler.java: 7132  clojure.lang.Compiler/eval
                  core.clj: 3214  clojure.core/eval
                  core.clj: 3210  clojure.core/eval
    interruptible_eval.clj:   87  nrepl.middleware.interruptible-eval/evaluate/fn/fn
                  AFn.java:  152  clojure.lang.AFn/applyToHelper
                  AFn.java:  144  clojure.lang.AFn/applyTo
                  core.clj:  665  clojure.core/apply
                  core.clj: 1973  clojure.core/with-bindings*
                  core.clj: 1973  clojure.core/with-bindings*
               RestFn.java:  425  clojure.lang.RestFn/invoke
    interruptible_eval.clj:   87  nrepl.middleware.interruptible-eval/evaluate/fn
                  main.clj:  437  clojure.main/repl/read-eval-print/fn
                  main.clj:  437  clojure.main/repl/read-eval-print
                  main.clj:  458  clojure.main/repl/fn
                  main.clj:  458  clojure.main/repl
                  main.clj:  368  clojure.main/repl
               RestFn.java:  137  clojure.lang.RestFn/applyTo
                  core.clj:  665  clojure.core/apply
                  core.clj:  660  clojure.core/apply
                regrow.clj:   20  refactor-nrepl.ns.slam.hound.regrow/wrap-clojure-repl/fn
               RestFn.java: 1523  clojure.lang.RestFn/invoke
    interruptible_eval.clj:   84  nrepl.middleware.interruptible-eval/evaluate
    interruptible_eval.clj:   56  nrepl.middleware.interruptible-eval/evaluate
    interruptible_eval.clj:  152  nrepl.middleware.interruptible-eval/interruptible-eval/fn/fn
                  AFn.java:   22  clojure.lang.AFn/run
               session.clj:  202  nrepl.middleware.session/session-exec/main-loop/fn
               session.clj:  201  nrepl.middleware.session/session-exec/main-loop
                  AFn.java:   22  clojure.lang.AFn/run
               Thread.java:  834  java.lang.Thread/run

1. Caused by java.io.FileNotFoundException
   Could not locate clojure/spec__init.class, clojure/spec.clj or
   clojure/spec.cljc on classpath.

I know, but i'd like the sorted map to keep the index ordering

in my example the order is correct (index oredring), but if i'd increase the element size it'll change

There are implementations of maps in Clojure that guarantee that the keys are sorted in the order that they were inserted into the map, e.g. https://github.com/clj-commons/ordered

Clojure's built-in function group-by does not return such a map, but you can copy and paste group-by's code into your own program, and modify it to use such an insertion-ordered map, if you really want to.

Note that many built-in Clojure functions other than group-by will return maps with no ordering guarantees, so if you use maps that preserve insertion order, and you want to keep using them throughout a portion of your data flow in your program, you will have to check whether all of the operations you are performing on those maps preserve the insertion order, or not, and perhaps create your own custom functions that do preserve it.

It’s failing to compile the kahdemlia.core namespace because it requires clojure.spec , which is a namespace that doesn’t exist in later versions of Clojure. You can try explicitly adding [org.clojure/clojure "1.9.0-alpha15"] to your dependencies, and it should compile. But I wouldn’t suggest using this library for anything serious unless you fork it and update it to work with a newer Clojure version

Looking at the definition of group-by, it uses transients on Clojure's built-in maps, which I suspect the insertion-ordered-map implementation does not support. If you want help creating a modified group-by that returns insertion-order-guaranteed maps, let me know.

I see I see hmmmm, I'll just clone the repo and explore the library directly instead 🙂

That also works for trying it out. The spec library is now under clojure.spec.alpha, it seems like clojure.spec only existed for that alpha release of Clojure. So should be possible to update the library to work with the latest Clojure

generally, when writing Clojure, I almost always use ex-info

i'd like yes 🙂

i'd prefer not to add any dependency to the project (ordered clj-commons)

You can implement your own such variant of maps if you want, independent of that project. There is no such insertion-order-guaranteed map built into Clojure

I understand, what about staying with persistent.arraymap ?

You can also maintain an unordered map, and separately keep a vector or list that contains the keys in the order you want them to be in, and use the list/vector for traversing keys in the desired order.

array-maps have O(n) time lookup for keys. I doubt you would want that for a 1000-entry map.

my element size is finite < 30x

Only finite things fit into a computer's memory 🙂

Do you want an array-map if the lookup time for a key is linear in the number of keys?

yes

So you do not plan to do lookup of keys very often?

I basically do some transformation on the data and send it forward

so there is no lookups anyway

I mean, the performance of your code is up to you -- I'm just warning you of the implications.

I have some structured data with a common key (which I group-by on)

and the order of the elements mean something, so when I group-by I want the result to stay ordered.

The other implication of using an array-map is that if you ever do just about any operation on a large one, it will return an unordered map. e.g. if you do assoc of a new key onto an array-map with at least 9 keys, the return value will not have the same key order guaranteed (and will usually be different)

I see, in my use case I just send the data onwards, so no need

If you use an unordered map returned by the current built-in group-by, you can also do a separate function on your input data to calculate the order of keys you want, and create a vector or list of that order. Then you could use that list/vector to decide what order to send key/value pairs from the map onto some next processing step.

Where are you sending the data onwards to, and why does that place need a particular order of keys?

Depending upon how you send it, and how the receiver reads it, if the receiver is using the Clojure reader to read a sequence of characters as a map, for example, then it will also return an unordered map that has forgotten about the order they appeared in the string.

I group-by, then iterate on the result (map) and the sequence result order matters

so basically my output is not the group-by

I understand that statement. I am asking what other software is receiving your result that cares about the key order. Is it some other software system you are sending this map to?

but the way I iterate over the group-by order matters (as map idx0 result should not change to the original item)

yes

Did you write that receiving software system?

Is it written in Clojure?

nope

react UI which renders the array I send to (in given order)

Are you sending the data as JSON, or something else?

json yep

Sorry if I'm asking a lot of questions, but realize that the situations where key order in a map actually matters are fairly small in the Clojure world, and the order is easily lost at many many different places. I ask because if I tell you how to preserve key order in this one little place, I worry that the very next place you send it to might just forget it.

basically I have DB records with timestamp + common column. I select from DB ordered by creation_date and group-by that column, then do some map iteration then in my result I expect the vector to have same result as the order of the db fetch

Also, now that I look at the implementation of group-by, it frequently looks up keys in the intermediate map it is building up, so if you try to create a group-by-return-array-map that starts with an empty array-map and builds up the result one key/value pair at a time, it will do linear time lookups for every key/value pair in the input.

so O(N^2) time algorithm doing it that way.

hmmms

not good then

https://stackoverflow.com/a/15500064/8162407

If you want to do this faster, then one way is to write a new function that is similar to group-by that keeps an unordered map as an intermediate value inside, and also a vector that remembers the order of keys you want. Return both things.

did some trick here, not sure I got his point

Why do you want to avoid a dependency on the ordering-map library, out of curiosity?

Because I think it is pretty darned small, and can help solve this problem efficiently, i.e. no O(N^2) stuff.

cause I think I can get around the group-by and use something else

I might reduce the ds and get same result as the group-by

If you need an actual map to write as JSON, i.e. it cannot be a list/array/whatever-JSON-calls-its-ordered-sequence-thing, then either you need to have a JSON-writing-library that lets you specify the order of keys separately from a map somehow, or you need a Clojure map-like thing that lets you control the key order completely.

FYI, as far as I have heard, your are also dealing with code that relies on ordering of keys in a JSON map, violating either the letter and/or spirit of the JSON spec.

why not just reduce to an ordered set?

reduce as to do the group-by job

Is the order you want for the keys defined by some simple comparison function, e.g. increasing numerical order?

If so, a sorted-map does the job.

order relies on the index of the elements

If you want some arbitrary custom order of keys, then you could also write a custom comparator function that compares key values in that order.

[m1, m2, m3,....mn] some might have key to group-by on but the result map should be in same order as the array indexes

alright, i'll try something, i'll update here:pray:

thanks!!

e.g. if you have keys "foo" "bar" "baz" and you have some arbitrary order that you can define as a Clojure map to relative numeric values, e.g. {"foo" 1, "bar" 2, "baz" 3}, then you can use that map to create a custom comparator function, and create a sorted-map that uses that comparator function, which will sort the keys in the arbitrary order you want.

thought I'd mention partition-by , I can't exactly tell how you are using the output of the group-by, so if you specifically need the keys from it then partition-by might not help, but my read of the situation is that you want to group up an already ordered collection, each time a value changes.

yes! that's actually all I needed, nice!

it will group-by each time to a seq in iteration order, then i'd iterate it the same way i'd iterate the k/v map of the group-by (that's actually what I needed), great!

glad you found what you were looking for in the first place. Sorry if I distracted you with issues on the thing you didn't quite want 🙂

nah, had the chance to look on some clojure impl code, thanks bud!!

I have a dev/user namespace. Things are working well. I have added a few dev tools like user/dbg. What are some of the approaches for configuring the REPL so I can reference user/dbg as just dbg in other namespaces without having to run require or use in each namespace where I would like to access them?

i’m using an atom to a map, atom of a map? how do you even say that. 🙂 I’m using an atom referencing a map as a database for a tiny app i’m working on. It holds data about baseball players. I have been writing some functions to do data access / updates.

(defn update-player
  [id key value]
  (swap! league
         (fn [current-state]
           (assoc-in current-state [id key] value))))

this seems to work great, but it prints the entire database in the repl everytime

maybe a success or fail would be good or silent success error on fail.. does swap! ever fail? or.. how would silence its output?

Regarding terminology, I think "an atom containing a map" is fairly common.

containing! nice.

"referencing" is clear, too

In a REPL, any time any function can return a value that is very large in being printed, you can do something like (def x1 (some-expression-returning-big value)) , and it will only show #'user/x1 instead of the big value. The big value will be stored in x1 for future reference/manipulation.

Unless the function you pass to swap! can throw an exception, I don't know of any reasons why swap! can fail

Even then, swap! should leave the current value contained in the atom there.

right that makes sense

it seems to be updating the atom AND returning the new value

Atomically swaps the value of atom to be:
(apply f current-value-of-atom args). Note that f may be called
multiple times, and thus should be free of side effects.  Returns
the value that was swapped in.

That is what the last sentence of the doc string printed by (doc swap!) says it should do.

ha ha nice, thanks for the pointer

See also swap-vals! for its different, and sometimes useful, return value

that’s cool, both the old and new

i wrapped the swap! call in a count call to return far less info, is that something people do like, maybe i could wrap it in do and return nil?

I don't know of any way to automatically do that off the top of my head, but if you have an editor or IDE where you have a key sequence of your choice assigned to "evaluate the expression just before the cursor", so you don't have to type the same expression more than once, it can be good to have frequently used expressions like `(require 'user.dbg)` or `(use 'user.dbg)` in a file at the top level of your project, e.g. named something like `scratch.clj`, that you often keep open while working in that project.

^^ @tkjone

A Clojure function body is already implicitly wrapped in a do, without having to type (do ...), so you can make the last expression nil in the body if you always want to return nil

If you aren't worried about large values accidentally being printed at a REPL, in development it can often be nice when functions return a useful value, even if you do not always use it. But yes, I see people sometimes returning a value like nil when they explicitly want that.

oh that’s really useful thanks, i didn’t know about the implicit do

Having your chosen editor/IDE set up so that a very convenient short key sequence can send expressions to your REPL for evaluation is HIGHLY RECOMMENDED

(defn update-player
  [id key value]
  (swap! league
         (fn [current-state]
           (assoc-in current-state [id key] value)))
  (str "updated " id))

this is kinda useful

but yeah who knows maybe having the whole value there is good

anyway, thanks!