Clojurians Log v2

is there a way to get the entire list (or tree) of namespaces that a given namespace depends on, given just the name (symbol) of the ns? e.g.

(ns-deps 'my.project.namespace)
;; ('my.other.namespace 'some.other.dep ...)

given that you can use a symbol by fully qualified name, that's not really possible - of course we can never use fully qualified names as a reasonable convention

there are ns-aliases and ns-refers

user=> (into #{} (map (comp namespace symbol val)) (ns-refers *ns*))
#{"clojure.pprint" "clojure.java.javadoc" "clojure.core" "clojure.repl"}

(ins)user=> (ns-aliases *ns*)
{}
(ins)user=> (require '[clojure.string :as string])
nil
(cmd)user=> (ns-aliases *ns*)
{string #object[clojure.lang.Namespace 0x73044cdf "clojure.string"]}

so by "never use fully qualified names" you mean always use :as? I think that can work for my use-case, thanks for the suggestion!

I am not sure if this is what you are after, but you can see a log of all namespaces that end up being require'd when you do a require with the :verbose option, e.g. like this (require 'my.project.namespace :verbose)

The output will be longer if none of the namespaces it requires have been loaded already. If any have, the things they require will not be shown in that log

I think it nukes itself on large sequences

yeah anything xform or transducer goes over my head still, so i'm gonna pass on those versions

I mean that you need either use or refer (which shows up in ns-refers ) or as (which shows up in ns-aliases, and avoid the situation where in one ns you use require [clojure.string :as string] and in another you use (clojure.string/join ...) - that's the situation where the relationship is invisible / unqueriable

what about combining :reload-all and :verbose ?

Good idea. That should be more complete than leaving out :reload-all

An adventure in OO (a short story) There's an API which creates something called a Registry. Registry is an object, currently it only contains an id. There's another API that deletes a Registry, it takes the same Registry object. Things changed, and when creating a Registry, people wanted it to also take a name, an owner, and some options. So they changed the Registry class with those additional fields. Now, when you want to delete a Registry, it turns out the delete API only really needs the id from it, but the Registry object now takes a bunch of stuff. The first issue is, people thought they'd be smart, and make that class so you're forced to provide all the mandatory parameters for creating a Registry, but that means when you delete you have to pass in data that's not needed and you might not have. So those validations were removed so everything except id is optional on this object now. But even then, the users of the API for deleting are really confused, why is there all these fields on the Registry? Which one need to be provided? Now, subtyping is the solution to everything (most of the time) in OO. So someone said, ok, we need to change the hierarchy of Registry. It turns out, we have a Registry that only has an id. And we also have a CreationRegistry that extend Registry with the additional fields like name, owner, and some options. Genius yes? Well, sure, except it is a breaking change, because you already have a bunch of clients calling that API and their code does: new Registry(). So if you make that change, you'd break them. The same thing happens if you try to make a DeleteRegistry object. So there's just no real solution to this, and everyone just deals with a stupid Registry object that depending where you use it, some fields are mandatory and in other place they are optional 😛 The End

Could that have been avoided if the method that deletes a Registry took only the ID instead of the entire Registry object?

Yup

I mean, that would also have been a breaking change, but since you have static types, it’s not really a problem, right? 😜

Doing it now would be breaking, if people had had the foresight of this scenario, they could have made the delete take only the id

Exactly, yeah.

But I've seen this play out a lot in OO, its because creating new class is cumbersome, people are lazy, so classes get shared as much as possible.

And people think of things as the class

So an API called: deleteRegistry(...) ? what would it take as input if not a Registry ? 😛

In fact, people would probably do: registry.delete()

Also, for APIs, its kind of common I've seen for people to think... we might need to accept more things in the future, so if you have it take an object, it lets you add optional fields to it in the future. Where if you change the signature to have more argument, its breaking. Which is another reason people avoid doing deleteRegistry(id). But then, they don't think much further, and honestly, its normal, how can you anticipate everything, so I'm pretty sure people thought they were smart using Registry as input 😛

Also, there's an integ test, and the test does:

registry = new RegistryBuilder().withId(123).withName("wtv").withEtc....;

client.createRegistry(registry);
// validate it was created
client.deleteRegistry(registry);
//validate it was deleted

Oh, absolutely.

@seancorfield: Yeah there’s definitely not enough reasons to migrate a large existing system from component to integrant; However for greenfield stuff I’d see no reason to pick component over integrant, integrant’s just a lot more flexible, and bakes data/configuration in (and all the benefits that brings) as a default. In terms of high level architecture though, they’ll lead to an almost identical decomposition of a system; it’s just expressed with different primitives, protocols vs multimethods; closures vs data, object references vs namespaced keywords/composite-keys etc.

@jayzawrotny: If you’re looking to do a webstack in integrant, you definitely want to look at duct. We use it for most greenfield web apps. It’s pretty good. You get a pretty standard but configurable ring stack out of the box. I’d definitely recommend it, but it’s also definitely not perfect — but it could be. Duct is also a pretty good starting point that lets you grow beyond duct’s limitations. For example our main app is pretty far from a standard duct app; but it was very easy to tweak ducts primitives for our needs. For example duct assumes an app is for a single customer; yet we were able to easily tweak the project layout etc to support multi-tenant (i.e. each customer of ours has their own build/configuration as a profile layered over a base profile of the app managed in the same source tree). We also use tools.deps not lein, and leverage most of duct’s core ring middleware stack, we do isomorphic SSR’d clj(s) with reagent, shadow-cljs and have full stack integration testing with etaoin, and recently added devcards as a harness and test bed for stand alone front end components.

Anyway #duct is quite a good channel for support around duct etc… there are quite a few folk there with large duct apps.

Why not add deleteRegistry(id) as an overload? Shouldn't break existing calls but allows people to use the more convenient method going forward

is there a j.u.concurrent mechanism (or clojure lib) that gives you a "grant" to use something (out of a pool of N items) for M seconds?

of course clj being clj this is pretty easy to implement from scratch, I'm just curious. I think I had seen this mechanism somewhere

Java has a Semaphore for granting but it doesn’t have the time aspect

The time aspect seems nontrivial to me

Okay, creating uberjars using deps.edn and depstar... is that broken for anyone else? I get:

Execution error (FileAlreadyExistsException) at jdk.nio.zipfs.ZipFileSystem/createDirectory (ZipFileSystem.java:695).
/META-INF/versions/9

Full report at:
/tmp/clojure-13855814055111068112.edn

Ah I guess uberdeps is the hot new thing now

@zilti depstar is very actively maintained and I take bug reports very seriously. I've not seen that error before but I'm happy to help debug it (and fix it, if it's a bug in depstar).

Oh, okay! Yes, I will file a bug report soon then. I guess it is fair to say it is in depstar, because uberdeps builds it fine

there is a #depstar room, maybe you should post your deps.edn in there

I don't think uberdeps handles MR jars, which is what /META-INF/versions/9 indicates you are using

Well, it may be that uberdeps simply ignores the problem and the resulting JAR just happens to work 🙂

MR == multi-release

Yeah, what he said 🙂

Yea, could also be

Oh, I wasn't aware about there being a room 🙂

So far, I've not hit problems with MR JARs -- they are supported by depstar -- but you may have encountered an edge case. The #depstar room is new so we don't flood #tools-deps with chatter about it 🙂

not sure if it is helpful but @zilti take a look on how I use depstar https://github.com/kwladyka/form-validator-cljs/blob/master/.github/workflows/clojars.yaml + prepare https://github.com/kwladyka/form-validator-cljs/blob/master/pom.xml

I'll save it for now and will have a look

Thanks! Maybe a "time-expiring hashmap" could kill a big chunk of the problem. The clj/java alternatives seemed to be bit of a heavy dependency though

What’s the idiomatic way to iterate over a sequence and execute a side effect functions for each element of the sequence given that we also want the returned values from the functions? For instance, iterating over do-stuff! that returns a meaninful value. One option is:

(mapv do-stuff! my-seq)

Another option is: (doall (map do-stuff! my-seq))

Or, I could write a loop/recur

run!

run! doesn't return the results though

ah, I missed the last part of the question :)

mapv is good for this use case

mapv doesn’t convey the fact that we are interested in the side effects (At least that’s how I perceive it)

It seems that clojure.core is missing a function for this use case. Could it be that this use case is not interesting?

can these fail? do you care about order? should you short circuit if they start failing? lots of different things change what this would look like in practice. can't imagine a single function in core really covering everything. do you need a thread pool to run these things? blocking, etc

reduce/transduce

Not sure if this is intentional:

user=> #_s/foo [1 2 3] ;; ok, although there is no alias s
[1 2 3]
user=> #_::s/foo [1 2 3] ;; not ok, reader fails!
Syntax error reading source at (REPL:5:10).
Invalid token: ::s/foo
[1 2 3]

we filed a similar bug around discarded tagged literals recently. Probably shouldn't try to autoresolve an alias for something that is discarded

:thumbsup:

should I make a JIRA about it?

sure, that would be nice. https://clojure.atlassian.net/jira/software/c/projects/CLJ/issues/CLJ-2577 is the related issue

done: https://clojure.atlassian.net/browse/CLJ-2581

I dunno, that seems pretty clear cut as the correct behavior for "read a valid form and discard it"

s/foo in the example reads correctly, even is s is not an alias in the current namespace

::s/foo doesn't

so ::s/foo is not a valid form if the alias s doesn't exist in *ns*

yeah, maybe you're right

it's definitely debatable

I just used #_ to comment out a form I just copy pasted from another namespace which then broke my CLJS compilation

I was taking valid form meaning not being able to discard 12ab34 because that's not readable at all

which is kind of what #_ is for right, commenting out forms temporarily

and it does accept s/foo if the alias doesn't exist, which gives kind of a precedent

but that's because 's/foo is valid even if s doesn't exist

while '::s/foo isn't

you'll get the same behavior from (comment ...) as well

that's only because comment is a macro. this is a reader implementation decision

user=> (comment s/foo)
nil
user=> (comment ::s/foo)
Syntax error reading source at (REPL:2:17).
Invalid token: ::s/foo
Syntax error reading source at (REPL:2:18).
Unmatched delimiter: )
user=>

it is because s/foo can be read and not evaled, while ::s/foo cannot be read

resolution of s/foo doesn't happen at read time

comment is different though

if we make #_::s/foo work then why not #_#{1 1} aswell

I'm considering it debatable because of the treatment of unregistered tags in CLJ-2577

(though edn has specific language around that, I think it could apply to alias resolution, too)

(it's a parallel problem)

one of the concrete ways this harms application development is if you have a require and an aliased keyword in a reader conditional. I ran into this last week, actually

alias resolution doesn't even exist in the edn spec

right, but tag handling and alias resolution are similar mechanisms (table lookup)

e.g. if you have some namespace:

(ns my-app.feature
   (:require
     #?(:clj [foo.bar :as fb])))

#?(:clj ::fb/baz)

which it should

:/

I disagree; while it may match some definition of consistency, it harms CLJS apps. I don’t want to include foo.bar in my app, or literally can’t

it can work in the other direction too. I guess I should say it “harms cross-platform code bases”

I personally do not see this as an error - the #_ is for reading, then skipping. ::s/foo where s is undefined is not readable.

it still seems inconsistent with #_s/foo to me, all technical details aside

is s/foo readable?

I don't care, it's just an implementation decision

it's not

it's being consistent

the difference is in the semantics, not an implementation detail

(read-string "s/foo") doesn't resolve s

#_ means read, then skip

(read-string "::s/foo") does

so why are symbol namespaces not resolved in the reader again?

one could argue that one inconsistency is that

`s/foo

@borkdude because how would you read (quote s/foo) if s was resolved by the reader?

(grabs :popcorn )

the difference here is in :: which is inherently contextual

it'd be resolved-s/foo by the time quote sees it

::pop/corn

right 🙂

calling a reader for tagged literal is similarly contextual, but the EDN spec says those should not be called when discarding

(I know clojure code != edn)

Any conversation with you, alexmiller, ghadi, and bronsa all submitting examples and points of view is guaranteed to exhibit something I've not thought about before.

Be very very careful with the #=(popcorn) though

Meanwhile I'm littering my emacs buffer with ;; 🍿

yes, that is a clear violation of the intent of the spec

I was going to jump in with ; as my go-to solution, but that was already known to all participants, of course.

And a nice recipe for messing up balanced parens

What’s the difference between my use case and the use case covered by run! and doseq ?

i have no idea what your use case is

in XML there was a distinction between "well-formed" and "valid". I would like to be able to discard invalid things (tags that don't exist, aliases that don't exist) while maintaining the behavior of throwing when something isn't well-formed

I have very practiced muscle memory of keeping parens balanced without using slurp/barf kinds of Emacs keybindings. Call me a luddite.

but run! will block and execute things in sequence. and no way to short circuit, all work done on current thread,

if that fits your use case then bob's your uncle and your request for a function in core to handle your use case is satisfied

skip the following s-expression

actually emacs clojure-mode does quite well with it, now that I try to mess it up

unless it contains unbalanced s-expressions

#¡ is the obvious choice for the reader tag 🙂

this discussion hits on a couple of pain points that I have with using namespace aliases and keywords. I have two concrete use cases: 1. I want to require a namespace in a reader conditional and use that as an alias in a keyword for a reader conditional 2. I want to be able to create and use an alias without having a corresponding concrete namespace I feel like the solution to either of those two problems could be related.

specifically focusing on (1) for now, I think there should be some way to inform the reader, “do not progress further” and I would really appreciate if that was baked into reader conditionals

This kind of thing is why I wanted "reader-conditionals" to be a macro and not baked into the reader at all

I think that is open for discussion and may even already have a ticket

so it was very clear that whatever the code was for any branch it must be valid and readable on any other branch

and 2 is something we definitely have a ticket for and are aware of as an issue that may see some attention in 1.11

I doubt it exists anymore but there used to be a confluence page with a fair bit if discussion about reader conditionals with lots of comments, etc

I think that whether or not conditional branches were a macro or a reader form doesn’t really matter to me. I still have use cases that necessitate including code conditionally and using aliased keywords

and you insist on using ::

Waht do you mean by “bob is my uncle”?

the very simple work around is just don't use ::

which is mostly my position on :: regardless of comments and conditionals

its an idiom meaning "you're good to go"

ah ok

no idea where it comes from 🙂

My use case is that I need something like run! with the slight difference that it returns a sequence of the values returned by the inner function

Hey guys I know reading other people's code can feel worse than waking up after a night of drinking some really cheap vodka. But is anyone willing to do a little code review. Its about 130 loc.

the tying of namespaces for keywords to the namespaces for code is gross, and insisting that the language grow a feature to make generating empty code namespaces so that it is easier to use namespaced keywords is also gross

there's a #code-reviews channel. and also smaller snippets are always fair game in #clojure and #beginners and other topic related channels

Ill head there

wasn't this inspired by spec initially?

no

what happened is the example docs for spec, for the sake of brevity use '::'

and people just copied that willy nilly

nothing requires the use of ::

> the very simple work around is just don’t use `::` yes that is a simple workaround. I disagree that it is a solution; otherwise, why have ::? I ran into this specifically when using a library that used namespaced keywords with long namespaces. :org.wscode.pathom.connect/input over and over is onerous to write and difficult to validate as a human reader (I have intentionally misspelled something here).

autoresolved keywords have been part of Clojure since 1.0

(def long-key :org.wscode.pathom.connect/input)

I also agree that generating empty code namespaces is gross. I wouldn’t suggest that as a solution

I agree it is gross, I haven't had a problem with long keys

that is what you are pushing for

that is what all this easy alias stuff is

I have not suggested any solutions. I have only outlined my motivation for a solution to the use cases I have for my projects

we are not expecting that to be the solution in Clojure

I spend 70% of my time in ClojureScript. generating an empty code namespace is not a workable solution at all in that context IMO

I seem to recall watching a Clojure talk sometime ago that discussed the functional programming approach with Clojure. The basic steps were: 1. Define the problem 2. Define a data structure which solves the problem 3. Define functions which operate on that data structure I’m fuzzy on the exact details which is why I’m trying to find the original talk again. Does anyone happen to know which talk this might have been? It’s neither this https://www.youtube.com/watch?v=Tb823aqgX_0 nor this https://www.youtube.com/watch?v=vK1DazRK_a0 (although both address similar topics).

there were a couple talks like this at very early conj's

https://www.youtube.com/watch?v=qqNevHwH47w

https://www.youtube.com/watch?v=3yvrs9S0RIw

and probably others as well :)

is there any issue with using async/onto-chan with large collections? 500k or so items? is this abusing the notion of channels or can they handle large work queues?

not that I know of, but make sure you use onto-chan! or onto-chan!!

is that newer? not seeing it in our copy with (apropos "onto-chan")

yeah

in the former case it will spin a go block to push to the channel, in the latter a thread

these were added recently

thanks for the pointers. i'll bump and check it out

1.2.603 or later

yeesh. goodbye "0.3.456" 🙂

https://clojure.github.io/core.async/ - docs

the difference there is not as large as you might presume :)

yeah i figured. i'm bumping and expecting no work to do but use the new goodies

actually, that is pretty old, nvm :)

but shouldn't be anything breaking afaik

So I have an ordered sequence of items . I need to map through this sequence with a function that accepts the previous element as well as all elements preceding it in my ordered sequence. Is there a good idiomatic way to do this in Clojure? Should I just map over the range of indices, take the current index + 1, and use butlast? Right now I have this:

(->> (range (count items))
  (map (fn [idx]
         (let [entries (take (inc idx) items)
               prev-items (butlast entries)
               item (last entries)]
              ;;do something interesting here
             ))))

butlast is O(N) complexity, so avoid that

clojure
Clojure 1.10.2-alpha1
user=> (doc partition)
-------------------------
clojure.core/partition
([n coll] [n step coll] [n step pad coll])
  Returns a lazy sequence of lists of n items each, at offsets step
  apart. If step is not supplied, defaults to n, i.e. the partitions
  do not overlap. If a pad collection is supplied, use its elements as
  necessary to complete last partition upto n items. In case there are
  not enough padding elements, return a partition with less than n items.

use (partition 2 1 coll) for overlapping pairs

I don’t think partition will work because the partitions need to be of varying / increasing size

oh, I misread you

Perhaps a loop ?

reduce

a loop if you're producing N values at the end, where N is the length of the list

otherwise a reduce is sufficient

(reduce (fn [[previous acc] x]
          [(conj previous x) (apply + x previous)])
        [[] 0]
        (range 5))

Yeah, I suppose either loop or reduce will work. I do need to produce N values at the end so I’ll probably go with loop

you could also do some mapping over (reductions conj [] coll) but might run into memory issues?

Oh, I like that even more, that should work for my use case.

We use aliasing to shorten long keywords in natural language all the time. In the north of England "tea" means an evening meal, and "dinner" means lunch. A southern colleague was asked to deploy some code at dinnertime, and she stayed late! We could tell everyone to say ":northern/dinner" instead, and it might have helped her out, but they won't because most of the time in our context, there's no need. But you do have to clarify when you're crossing contexts 🙂 Similarly I can tell (and have told) developers their JSON keys in their APIs are too short and ambiguous, sufficing only for the narrow context they had at the time... And I long for namespaced clarity in the overloaded terms that are used multifariously in different contexts at my work. But I'm starting to appreciate that there's always shorthand language for local conversations, and more precise language for general audiences, and so aliasing is inevitable and to be supported.

k

Just an FYI on what happened: I took your advice and asked internally. We do have several interested in Eric’s REPL course. I’m not sure whether we can get funding for subscriptions now, but I’m going to keep pushing on management until we do. Half a year max IMO. Good thinking @seancorfield, though now I owe you too many thanks to count. Do you have a monthly github funds/patreon style thing setup for donations?

Thank you, but I do not. Somehow it always makes me feel a bit odd when folks try to pay me for just being a decent netizen 🙂

Maybe one day we'll meet IRL at a conference and you can buy me a beer? 🍺

Now that said — I can understand not feeling the need to take donations either! But I’m sincere in my offer to “put my money where my mouth is”, so perhaps that’s worth something by itself 🙂