Clojurians Log v2

@chromalchemy i think you need to profile what's actually going on

i'm assuming you're calling filter or remove under the hood in those fns

which just walk the sequence and apply their predicate to each item, which should be fast enough

you could maybe get some minor speed benefits doing a single pass with a combined test, e.g. transducers

but not 5-10s worth i wouldn't have thought...

datascript/datomic/datalog is an example of something more generalised for querying the state of your items

maybe checkout https://github.com/alandipert/intension

@chromalchemy if you're rearranging list items and you have internal state for your items then you'll have a much easier time managing that state with a keyed list https://reactjs.org/docs/lists-and-keys.html

even if you carefully wrap all your own state in shared scopes that you manage properly, there's still browser state like focus, values of inputs, cursor positions, selections, etc. (much of which JS cannot even see) that can get mixed up if you can't lock a specific DOM element to a keyed item

I have a UUID for each item. I've been using this to query the list and get/set/test map vals. I was assuming that this was not great, that constantly scanning the list for item id's was inefficient, and that I should be using the implicit index in the vector. But I'm seeing maybe I was on the right track?

I see that the key does not necessarily have to be a UUID, just locally unique.

Am I correct that since my "items" each have an unique :id, then it is a "keyed list"?

@chromalchemy in the keyed-for-tpl you provide a key fn

if you don't provide one then identity is used

which would really only make sense if your items never changed, but you wanted to rearrange them

the tests use keywords as an example of that, so a list like [:a :b :c :d] the key for an item is itself

for your case, :id would be your keyfn

so the work needed is something like

- map keyfn to the list

- walk this list of keys and lookup to see if we already have an el in the cache or we need a new one

- return a new list of els based on the key lookups

- call merge-kids to actually put the els in the DOM because there's no guarantee they line up with the last iteration

building els and calling merge-kids at the end is probably the slowest parts of that

but its all pretty new!

in the future, merge-kids could potentially even become "key aware" and do smarter/faster diffs based on that

Ok thanks, I'll try to walk through, digest that process.

I'm a bit confused about keyfn though. What does it return? Or does it apply the key/val to each list item?

given an item, return its key

or its "identity" in a sense

you're just telling hoplon what about the item makes it unique

if you have a uuid that's easy, just return that

The id value itself right? Not simply the keyword the value is on?

ah well keywords are fns

(keyfn myitem) => 23

so :id is both the key and the function to lookup the key 😛

yeah exactly

Ok, just wanted to make sure, lol

yup, but let's say you needed to combine two things

like um :id and :type or something

you could do (juxt :id :type) for your keyfn

like how a primary key in a db can be a combination of a few fields

So basically the keyfn is the "query" on the list?

uh, it is, but specifically it's the query to tell hoplon how to line things up internally

say i had [{:a 1 :b 2} {:a 3 :b 4}] and i passed that to the loop

the loop needs to build an el for each item

but it also needs to know how to retrieve that same el next time it gets a new version of the list

so if the next iteration of the list looks like

[{:a 1 :b 5}]

what do we do?

if the keyfn is :a then we re-use the first el from before and remove the second el

if the keyfn is :b we need a brand new el and remove all the other els

if the keyfn is (juxt [:a :b]) then we need a new el, and we'll never reuse this new one until we see a new item with both :a 1 and :b 5

if the keyfn is identity (the default) then we'll only reuse the el if we see a new item that is = to the entirety of the current item

ok. And this is all in the context of keyed-for-tpl? And "re-use" means the same dom node, but it can have updated values (outside it's key)?

yeah, same DOM node new values

the important difference between a normal for-tpl is that the mapping between items and DOM nodes is based on the "identity/key" of the item rather than the position in the list

which is crucial for re-ordering, because if you think about it, the whole concept of "reorder" is grounded in the idea that we have some "unique things" that can be reordered in the first place 😛

True, sounds obvious in retrospect. So the keyfn returns an item's unique value. And the keyed-for-tpl tests for equality of that value across updates to better manage (diff) the dom.

yes

but currently it's more about ensuring sensible state downstream than efficiency

although i'm sure we could get improvements on the diffing later too 🙂

With the normal for-tpl, I would filter the list outside of the loop, then feed the filtered results in. But my understanding is that each iteration of any outside filtering, generates a whole new list in memory under the hood. Even if you only add/remove one thing? There's no sharing, and potentially a memory overload if you have too many filterings going on.

depends how you do it

subvec for example uses structural sharing, so it's O(1)

but it's not really a filter per se

calling filter yes, that will make a new list each time

actually tbh i'm not even 100% sure of that, lol

I've been using Specter a lot, so not sure what's going on under it's hood.

part of clojure's immutability is they do stuff "under the hood" to make things faster

even if filter currently makes a new list, it might not in a future version...

i'd be surprised if the making of a new list was causing problems

Like changing [:a :b :c] to [:a :c :b] results in 2 lists in memory?

i dunno, i think "it depends"

i think you have to research structural sharing in clojure to find out

Hmm. Ok I assumed some clojure immutable magic was going on. It only just looks like I'm duplicating all over the place.

yeah exactly

i think there's a lot of magic making it safer than it looks 😛

regardless, you can use transducers if you're worried about this

i highly recommend profiling before you go down this road though, it sounds like a micro optimisation to me

e.g. i'd expect the process of walking the list and applying a predicate to every item to decide what to keep is going to be riskier for CPU than the mechanical process of building the resultant list

even if you only remove one item, filter still needs to check every item

What do you think of the cascading formula cells to filter a list, (per above meta-code), before feeding to for-tpl. If I change the params around I get a lot of different versions of the list. But it seemed like you didn't think this would necessarily lead to a memory explosion in the for-tpl.

i do things like that reasonably regularly

i also use tools that have a query language regularly

inside the for-tpl (at least my proposed versions) i don't think there's much you can "blow up"

the for-tpl only sees the final output of your prior manipulations, and then builds item cells based on the key/position of the items list, and maps them to DOM elements based on the key/position

this is where i got to last night with a simplified for-tpl:

(defn loop-tpl*
 [items tpl]
 (let [els (cell [])
       itemsv (cell= (vec items))
       items-count (cell= (count items))]
  (do-watch items-count
   (fn [_ n]
    (when (< (count @els) n)
     (doseq [i (range (count @els) n)]
      (swap! els assoc i (tpl (cell= (get itemsv i nil))))))))
  (cell= (subvec els 0 items-count))))

there's not really a lot to break there...

and walking a list and applying a pred to filter is O(n) so it's not going to "blow up", just get linearly slower as you add more stuff to the list

"i'd expect the process of walking the list and applying a predicate to every item to decide what to keep is going to be riskier for CPU than the mechanical process of building the resultant list. Even if you only remove one item, filter still needs to check every item" I assume this is what's bogging me down more than anything. How can I avoid this constant walking (event to just pick out a single item with a given UUID)? That's why I was thinking I needed to use the implicit vector index. But the react article seemed to warn against that?

ps. I haven't had a chance to get into Datascript yet 😛

well you can't is the short answer

but looking forward to it

because if you have any downstream local state

you'll just need to do the same calculations there, but probably more than once

using my example from the issue i raised

(let [state (cell {})] ; <-- note the additional state _outside_ the defn
 (defn my-item
  [item]
  (let [k (cell= ...) ; <-- get an id from item
          expand? (j/cell= (get state k) (partial swap! state assoc id))] ; <-- e.g. javelin readme
   ...)))

(for-tpl [x my-list] (my-item x))

we're now just applying keyfn ad-hoc and using it to lookup values from a state cell per DOM element

instead of doing it once up front and letting the DOM elements downstream take a consistent view of an item without additional accounting

essentially there's no such thing as "changing a list" in clojure, as lists are immutable values

you make a new list, and if there's any relationship you want to preserve between list A and list B then you need to make that explicit

which is going to involve a walk somewhere

might as well make sure it's only 1 walk and not 50

Yes, ok. I'm in the 50 walks camp right now, lulled by the idea that by mutating a named cell I have a "consistent" list.

you can use a for-tpl and get some (probably minor) speed benefits if:

- your downstream DOM elements are totally read-only, not even a11y browser interactions are allowed, like focus states

- your downstream DOM elements have no internal state that references the identity/key of an item

I don't understand that last one. Can you give an example?

^^ like the expand? above

like "read more" with a popdown showing more info

you want that to be tied to an item, not a DOM element

if i click "read more" on item 1, then reverse my list, after rearranging i want the last item to be expanded, not the first

i guess i'm just saying the same thing twice 😛 which is that you've got internal state of elements to worry about, which could either be cljs/cell state or browser internal state

I really don't want to worry about browser internal state if can avoid it. So by "item" you mean the item map, presumable contained in a cell. So If I have a "read more" link, I walk that cell for the "more" data, to feed to the view (which might create a new dom element)...?

ah so for the purpose of what we're talking about here there is very little difference between a keyed and non-keyed for-tpl

a (cell [ ... ]) goes in, gets turned into [(cell= ... ) (cell= ... ) (cell= ... )] or {k1 (cell= ... ) k2 (cell= ... ) k3 (cell= ... )}

actually that's not quite right lol

hmmm oversimplified too much 😛

it's actually more like

(cell [ ... ]) goes in, then we check to see if there is an appropriate DOM element for each item, if there is not then we make one like (my-el (j/cell= (get items k nil)))

when you change the items list, we just check to see if new individual item cell/DOM mappings need to be made

if you provide a sensible key to keyed-for-tpl you won't need to worry about browser state getting out of sync with your uuids

my point is just that even though there's some internal stuff going on to make that happen, you can't make it faster by avoiding it because either A. you'll have to do equal or greater work downstream or B. you will have subtle bugs caused by stale/inconsistent state

I think I'm doing what you talked about earlier w/ respect to downstream identity:

(defc state 1)

(defc items [{:id 1} {:id 2}])

(for-tpl [x @items]
   (elem :color (cell= (when (= (:id @x) @state) :red))

yes

(defc items [{:id 1} {:id 2}])
(keyed-for-tpl nil :id [x items]
  (elem :colour (cell= (when (= (:id x) 1) :red)))))

^^ looks like this with a key

ignore the first nil in keyed-for-tpl for now 😛

but what's also important, and is hard to see from this example, is that often these things are in different parts of your codebase

your for-tpl might be in some ns calling elem from another ns, and elem might also be called from multiple different places, so it makes it tedious to track all this state across multiple places and contexts

you really don't want your elem to be "reaching out" to global state in order to get the job done 😕

I think I see. But how do you change the 1 (state value) in your example?

you can make it local to elem safely

it's really

(defn my-item
 [item]
 (let [expand? (cell false)]
  ...))

(keyed-for-tpl nil :id [x my-list] (my-item x))

vs

(let [state (cell {})] ; <-- note the additional state _outside_ the defn
 (defn my-item
  [item]
  (let [k (cell= (:id item)) ; <-- get an id from item
          expand? (j/cell= (get state k) (partial swap! state assoc id))] ; <-- e.g. javelin readme
   ...)))

(for-tpl [x my-list] (my-item x))

the boilerplate in the latter needs to be repeated for every single cell that i want to represent state for a given item

I think I'll have to play about a bit to fully appreciate all this. Right now it's still a bit fuzzy cause

(defc items [{:id 1} {:id 2}])

(for-tpl [x items]
  (elem :colour (cell= (when (= (:id x) 1) :red))))) 

looks almost the same as 

(keyed-for-tpl nil :id [x items]
  (elem :colour (cell= (when (= (:id x) 1) :red)))))

well actually your example works in both for-tpl and keyed-tpl 😛

because your cell attached to :colour is using part of the value of item

that works

expand? is different because you can't read it from item

it is related to item but it isn't part of that data structure

more like this

(defn elem [item]
 (let [toggle? (cell false)]
   (div
    :color (cell= (if toggle? :red :blue))
    :click #(swap! toggle? not))))

(defc items [{:id 1} {:id 2}])
(keyed-for-tpl nil :id [x items]
  (elem x))

when you click the elem it swaps between red and blue

so if i click :id 2 it goes red, if i then reverse the list so that :id 2 is at the start of the list, do i want the first DOM element or the second one to be red?

for-tpl would keep the second element red, so :id 1 would be red now

keyed-for-tpl would make sure the item with :id 2 is red, so the first element would be red now

I think I see now. I think in that situation I would have tried to create a permenant {:id 1 :toggled? true} attribute to get/set/test. But that seemed verbose and redundant and more walking..😬 Using that local state looks a lot more elegant.

yes of course you can keep injecting all this state into item

but you're going to end up with dozens or hundreds of very similar looking keys quite fast >.<

and none of those extra keys really add value to item tbh

toggled? has nothing to do with the fundamental definition of the item, it has to do with the internal state of a particular DOM element that is rendering that item in a certain way

Yes, that's the feeling I was getting. My semantic data was getting littered with view-state complexity! I want to compute the view stuff implicitly, but was searching for an efficient means.

Well thank you for taking the time to explain it in depth. This will help me tremendously. My app is all about filtering and sorting a list, and this clarifies where I need to head, and free's me from some gloom about feeling I was plodding down a dead end path.

I am currently trying to update to Hoplon 7.1 but getting some .spec errors. Do I need 7.1 to use the keyed-for-tpl? or could I plug it into 7.0.3?

actually keyed-for-tpl is not available in anything yet 😛 waiting on @flyboarder to push a snapshot when he's available 🙂

but it would be in 7.2-SNAPSHOT getting ready for the 7.3 release

what errors are you getting from spec?

I got the one about serializing the error message, but I was able to follow your trick of inserting (prn) code into boot.cljs

ah yeah, well that error just hides another error

now I'm getting stuff like

{ :cause 
     "Call to clojure.core/defn did not conform to spec:\nIn: [1 0] val: {:keys [url open-new?], :or {:open-new? false}, :as attr} fails spec: :clojure.core.specs.alpha/arg-list at: [:args :bs :arity-n :bodies :args] predicate: vector?\r\nIn: [1 0] val: ({:keys [url open-new?], :or {:open-new? false}, :as attr} kids) fails spec: :clojure.core.specs.alpha/arg-list at: [:args :bs :arity-1 :args] predicate: (cat :args (* :clojure.core.specs.alpha/binding-form) :varargs (? (cat :amp #{(quote &)} :form :clojure.core.specs.alpha/binding-form))),  Extra input\r\n"
  :data 
     {:clojure.spec.alpha/problems , 
        ( 
          {:path [:args :bs :arity-1 :args], :reason "Extra input", :pred (clojure.spec.alpha/cat :args (clojure.spec.alpha/* :clojure.core.specs.alpha/binding-form) :varargs (clojure.spec.alpha/? (clojure.spec.alpha/cat :amp #{(quote &)} :form :clojure.core.specs.alpha/binding-form))), :val ({:keys [url open-new?], :or {:open-new? false}, :as attr} kids), :via [:clojure.core.specs.alpha/defn-args :clojure.core.specs.alpha/args+body :clojure.core.specs.alpha/arg-list :clojure.core.specs.alpha/arg-list], :in [1 0]}
          
          {:path [:args :bs :arity-n :bodies :args], :pred clojure.core/vector?, :val {:keys [url open-new?], :or {:open-new? false}, :as attr}, :via [:clojure.core.specs.alpha/defn-args :clojure.core.specs.alpha/args+body :clojure.core.specs.alpha/args+body :clojure.core.specs.alpha/args+body :clojure.core.specs.alpha/arg-list :clojure.core.specs.alpha/arg-list], :in [1 0]})
      :clojure.spec.alpha/spec #object[clojure.spec.alpha$regex_spec_impl$reify__2436 0x13d54144 "clojure.spec.alpha$regex_spec_impl$reify__2436@13d54144"], :clojure.spec.alpha/value , 
        (link [{:keys [url open-new?], :or {:open-new? false}, :as attr} kids] (elem :td :underline :tc :blue :m :pointer :click (fn* [] (.open js/window url (if (not open-new?) "_self"))) (dissoc attr :url :open-new?) kids))
      :clojure.spec.alpha/args 
        (link [{:keys [url open-new?], :or {:open-new? false}, :as attr} kids] (elem :td :underline :tc :blue :m :pointer :click (fn* [] (.open js/window url (if (not open-new?) "_self"))) (dissoc attr :url :open-new?) kids))}
  :via
    [{:type clojure.lang.ExceptionInfo
      :message 
            "src\\elements\\misc.cljs [line 90, col 1] Call to clojure.core/defn did not conform to spec:\nIn: [1 0] val: {:keys [url open-new?], :or {:open-new? false}, :as attr} fails spec: :clojure.core.specs.alpha/arg-list at: [:args :bs :arity-n :bodies :args] predicate: vector?\r\nIn: [1 0] val: ({:keys [url open-new?], :or {:open-new? false}, :as attr} kids) fails spec: :clojure.core.specs.alpha/arg-list at: [:args :bs :arity-1 :args] predicate: (cat :args (* :clojure.core.specs.alpha/binding-form) :varargs (? (cat :amp #{(quote &)} :form :clojure.core.specs.alpha/binding-form))),  Extra input\r\n"}]}

{:keys [url open-new?], :or {:open-new? false}, :as attr}

Call to clojure.core/defn did not conform to spec

yeah looks like that's clojure itself complaining about some of your syntax

you probably just got bumped up a few versions of cljs to support 7.1

Yeah, I think so. Hopefully I wont hit a wall with it 😅

My current codebase is in Hoplon.UI, which hasn't been updated in a while. I'm not sure If i can update to 7.2 if there are breaking changes, at least until a major refactoring. Can I cut and paste the keyed-for-tple stuff on top of a 7.1 base. Or does it depend on 7.2 internals?

well if you get stuck let us know

yeah you can paste it in

the only thing is that it is so new that it will probably evolve a bit

so don't get too attached to the version you paste

in theory there aren't breaking changes in 7.2, it was a big refactor internally but the idea was to keep the external APIs the same or extended with new things only

Sure. I think anything might help me out right now. Especially just learning how to orient to that keyed internal state you so eloquently demonstrated.

Ok I'll give it a try

What about using a 7.0.3 base, just in case?

ah actually hmmm

i lied

it won't totally work without the new version of merge-kids

😕

i forgot about that

since for-tpl never re-arranges anything, hoplon doesn't actually support re-arranging >.<

i had to add that support while i was working on keyed-for-tpl

dang

which also sucks for me because it means i can't use it either until the snapshot goes up, lol

on that note i think i'm going to go get a ☕

don't stress too much about 7.1 -> 7.2 though, i don't see that being a major hurdle at all

Ha. Lifestyle of the bleeding edge! I'll probably refactor and catch up to 7.2/7.3 asap either way. Thank you for the help, and all your recent work!!!

np, glad to help

and also it's great if other ppl start using what i'm writing so i can get feedback and make sure its rock solid

Sure. I'll test and give some feedback as soon as I'm able. Later

:thumbsup: catch

@thedavidmeister I’ll get it in snapshot soon I promise

Nice explanation! This is a great enhancement!

@flyboarder hah, all good, i've got plenty of other things to keep me busy 😉

i think this also fits well with the "easy interop with 3rd party libs" principle, as they will often have their own internal state that hoplon shouldn't damage or need to know about - imagine a WYSIWYG editor attached to an item, we don't want to be thrashing a setData() method on every reorder, that could be very expensive as it would internally trigger a parse/render cycle on the editor for every item

@chromalchemy https://www.youtube.com/watch?v=6mTbuzafcII <-- good watching if you're concerned about lots of list rebuilding

you're basically doing what he describes at around 6:00

Hi all. I am new to hoplon as well as clojure(script) so please excuse my naive question

I am trying to get the value of a plain text input field and create dynamically a number of text input fields based on the number I will get. I am not sure how to get this number.

Fixed 🙂

(let [c (cell 0)]
 (input :value c :input #(reset! c @%))
 (for-tpl [i (cell= (range (int c)))]
  (input)))

@corelon ^^ something like this