Clojurians Log v2

You’re typing directly into the REPL?

(if you’re working from an editor, the REPL integration takes care of changing the REPL to the namespace you’re working in so code being evaluated has access to whatever is in that namespace — when you’re working with exploratory code in a Rich Comment Form (comment ..) for example)

using your prior suggestion of vscode, that only occurs once i have performed a form eval from the target namespace. otherwise, the nrepl is sitting in clj.user

if i eval a form in the editor, the namespace switches to the form's current ns

In the (very rare) cases that I’m typing directly into a REPL, I tend to stay in the user ns and require things in with an alias — following the same pattern as I would in code.

@franco.gasperino Yeah, most integrations assume you will eval the file before you start editing (some variant on “load file”, “evaluate file”, or whatever)

thats a reasonable default, but it can be what i'll call a tooling snag for newcomers

in-ns is reasonable for switching between namespaces that you have previously require'd

i iterate quite a lot at the repl with form evals, and use those sessions as a basis for deftest later

If I’m writing a new file from scratch, the first thing I write is the ns form and it’s so habitual for me to hit the key to “eval top-level form” as I’m editing that the namespace is known to the REPL before I write any other code. For an existing file, I tend to just hit the key to load the file into the REPL as soon as I open it, if I think it hasn’t already been required. Depends on where I start editing.

But my REPLs run for days so they tend to accumulate my entire codebase fairly quickly.

The REPL I’ve been using to work on HoneySQL v2:

user=> (dev/up-since)
#inst "2021-01-31T00:34:03.572-00:00"

if the ns is re-bound in the repl, a load-file / top-level eval will discard to the on-file state?

I almost never type into a REPL — even exploratory eval I do inside a (comment ..) form in a source file. And then the code is already in my editor if I want to turn it into a test.

@franco.gasperino Not sure what you’re asking?

if in-file i have:

(ns my-ns)
(def myvar 1)

and at the repl, after eval'ing those 2 forms, i perform the following:

clj:my-ns:> (def myvar 2)

reloading / re-eval the in-file forms will rebind myvar to it's original state, yes?

e.g. discarding my in-repl binds

Namespaces are mutable. Only one instance of each namespace exists.

got it

And your editor “REPL” and your console REPL are both clients into the same JVM and the same Clojure process.

An example from work: we run Socket REPLs in many of our production processes. If I VPN in and set up an ssh tunnel and then telnet into a production Socket REPL, I get a regular user=> prompt. If I do stuff in that session and then exit telnet and shut down the tunnel and VPN, then come back days later and connect back into that production process, everything I did in the past REPL session is still there in the user namespace.

following that rabbit hole, can the prod jvm process link live to a repl, and you as the remote user to the repl, and then fiddle (view, update, ...) an atom ref which the process is using?

e.g. dump current application state (atom ref) to disk?

Yes, you have full access in the REPL to any global state in the running application.

impressive

(and depending on how you built & deployed the app, you may even be able to redefine functions on the fly and have them have effect immediately — we’ve done that quite a few times with one of our internal apps to fix problems without needing downtime)

When I’m developing locally, I have a REPL running, I connect my editor to it, and I build the app via the REPL — by eval’ing code in the editor — while the app is running.

I did presentations to both London Clojurians and Clojure Provo in the last few months showing how I build a web app directly via the REPL, modifying it while it is running. Even adding new libraries without restarting the REPL.

https://corfield.org/blog/2020/11/24/talks-clojures-superpower/ has links to the recordings.

you know, i watched you give that on youtube a few days ago

with the cottage backdrop

My mum’s house in England 🙂

beautiful lot

This shows a window where I started a REPL at the top. Then I connected to the same REPL in another window (below) and listed the available public vars. In the original REPL, I defined a var, then in the other REPL I referenced it. Just to show that even with multiple REPL clients, they “share” user:

OK, time to go feed the cats. I’ll be back online later!

thanks for the tips

DEPRECATED: Libs must be qualified, change compojure => compojure/compojure (deps.edn)

hey there, just added compojure to my deps.edn

confused what this means? thought it seems to work

like i know i can change it to compojure/compojure but how would i have known that?

The deprecated message you quoted is the way I found out about it 🙂

ha, ok awesome

is it just double the name?

Leiningen and Clojure CLI tools have been permissive in allowing just an artifact name, without a group name, for many years.

e.g. Clojure's full name is org.clojure/clojure, where I believe org.clojure is the group name, and clojure is the artifact name within the group org.clojure.

Thisi is a Maven naming thing.

ahh

The Clojure CLI tools have recently deprecated the earlier common practice of leaving out the group name.

This earlier common practice was that if you didn't want to come up with a group name, then if your project's name wasn't already a group name on http://Clojars.org, then your project name also became the group name of that project.

(Caveat: This is my understanding, which could be off in several important details)

that’s cool thanks

it looks like if you go search clojars they will tell you the right string to put in

deps.edn

which is nice

🙂

so i’ve been hacking away at a single source file, core.clj which has a main function I don’t use, i’ve just been evaluating stuff in the REPL, i want to move some of this code to a differnet file and use this core to start a web server and then call into that other code

do i make a new file that has its own namespace?

is there a good thing i should read before i ask every individual question about this process?

I'm not sure of a good resource that covers this question, but would not be surprised if there was one.

You are on the well supported and easier to understand path if you put every Clojure namespace in a separate file, each beginning with an ns form with the name of that namespace, and the namespace name corresponds one-to-one with the file name it is stored in.

Whenever you do require or use in Clojure, it searches all directories in your class path for one of a few file names that correspond with the namespace name, e.g. require of a namespace foo.bar.baz , and if your project's classpath contains the directory src (a common choice, but not mandated by Clojure), then it will look for a file named src/foo/bar/baz.clj

I would actually recommend against using a namespace ending with core, but it is a fairly minor reason -- if you ever get a stack trace because of some exception thrown in your code, it contains not the full path names of source files, but only the last part, e.g. bar.clj, not src/foo/bar/baz.clj in the stack trace lines, so if you have a namespace foo.core, any lines in a stack trace for that file will show up as core.clj, which is the same file name for many functions built into Clojure that likely also show up in your stack trace.

ah this is all very helpful, thank you!

is this java stuff or does clojure do its own file loading ?

Warning: you are certainly allowed to have dashes - in elements of your namespace names in Clojure. You must replace those dashes with underscores when you create the corresponding file or directory names in your source code, because that is what require and use look for.

ha, i bet that has cost some people some time

require is Clojure-specific, and the turning of dashes into underscores is Clojure-specific, because Clojure explicitly allows dashes in names, but Java does not allow dashes in class names. Part of the bottom of Clojure's require implementation uses a Java method getResource that searches the classpath, if I recall correctly: https://docs.oracle.com/javase/7/docs/api/java/lang/ClassLoader.html#getResource(java.lang.String)

oh awesome

I appreciate the explanation

https://github.com/abedra/shouter

I’m copying the bones of this

The naming of a file core.clj is common in Clojure projects, and very well might trace back to the use of core.clj in Clojure's implementation, and/or the creation of a file core.clj when Leiningen creates a template project for you in its default template.

I'd recommend feeling free to rename that file and namespace in any such template.

Through habit or whatever, I’m tied to debugging/developing using println. I like being able to see the changes to data over time which I don’t get when only seeing output from evaluating forms in a comment block

Regarding the lib name thing (`compojure` => compojure/compojure) there is another change which folks need to be aware of if they plan to publish a library (to Clojars; Maven is already stricter). Clojars historically allowed "any" group ID so a lot of early libraries just picked a single name, like ring or compojure or hiccup, and that became the group ID as well as the artifact ID (these are Maven terms really). For security reasons, Clojars is implementing a policy where group IDs must be reverse domain name format -- which is already common in the Java world -- and you must be able to verify that the reverse domain name belongs to you. By default, they offer net.clojars.<username> which everyone can use (because your username is "verified" by virtue of you logging into Clojars). In addition, if you use your GitHub ID to login to Clojars, you can use com.github.<username> for your group (verified by virtue of you logging into GitHub to authenticate).

There's a good thread about this on ClojureVerse if you're interested in more about it: https://clojureverse.org/t/clojars-verified-group-names/7297

When I started publishing libraries, I chose seancorfield as my group ID but I am slowing migrating my libraries to com.github.seancorfield. If you use the CLI and deps.edn, and you create projects with clj-new, the latest version tries hard to create library projects that will conform to the new security policy at Clojars. If you ask clj-new to create you a new project with the name <username>/<projectname> then it will assume you mean com.github.<username> for the group ID and will set you up to publish your library that way.

that’s really interesting

where does clojars end and maven begin?

Maven has much tighter control over what gets published and how. Clojure itself is published to Maven, along with all the "Contrib libraries" (everything under the org.clojure group). Clojars is maintained by the Clojure community (funded by Clojurists Together and various companies) and allows "everyone" to publish libraries there. Leiningen, Boot, and the Clojure CLI all know to look on Maven for a library and if it isn't there they look on Clojars for it.

(by default)

ah that’s cool

you don’t really have to know where the thing you want is

i’ve been using vscode to start a REPL and jack-in

but i’d like to learn how it actually works so i’m following

https://www.clojure.org/guides/deps_and_cli

clj -X schepball.main/-main

is this right?

doesn't look like it. -X needs to call a function that takes a single map. -main is usually a function that accepts a sequence of string args. the two aren't really comptible

ahh

-main is generally invoked via "main opts" which means -M and passes zero or more strings to the function (which has [& args] for arguments); executing functions via -X which pass a single hash map to the function.

@seancorfield to pull out a part of the earlier conversation, how do you handle branches when developing in long-running repl sessions?

ah that is interesting

i’m finding myself wishing for a clj run that new how to run my “main” file

I wrote a small utility with a similar idea, https://github.com/phronmophobic/clirun Args are parsed as edn, but you can specify the main function. For example:

# Write to a file using clojure.core/split
$ clj -M:run clojure.core/spit '"foo.txt"' '[1 2 3]'
$ cat foo.txt
[1 2 3]

clojure -M -m my.namespace isn't much work though...

ah i didn’t know about that flag

as usual i should read more before asking questions 🙂

the -m flag makes sense to me I think

but the -M i’m not understanding

it uses clojure.main?

It's a good question! Maybe it's our working style at World Singles Networks but I don't find it's a problem. My REPL state just followed the code that I'm working on, so when I branch for a feature/bugfix, I just keep working in the REPL, until I'm done and push the branch for a PR review. If I need to work on something that builds on that, I'll cut the new branch off the previous branch rather than the trunk, and keep working. If it's orthogonal work, the changed functions in the REPL won't affect it anyway, even I start over from the trunk.

and clojure -X core/one-of-my-tasks works just as well. can only be a single main in a file, whereas -X can start with any function taking a single map as an arg

(doc clojure.main/main)
-------------------------
clojure.main/main
([& args])
  Usage: java -cp clojure.jar clojure.main [init-opt*] [main-opt] [arg*]

  With no options or args, runs an interactive Read-Eval-Print Loop

  init options:
    -i, --init path     Load a file or resource
    -e, --eval string   Evaluate expressions in string; print non-nil values
    --report target     Report uncaught exception to "file" (default), "stderr",
                        or "none", overrides System property clojure.main.report

  main options:
    -m, --main ns-name  Call the -main function from a namespace with args

Most of our branches are fairly short-lived. We merge everything to the trunk as soon as it is reviewed and approved. And we deploy to staging for business review almost every day (sometimes several times a day), and from there almost all apps can go to production completely automatically as many times a day as business want (the business team can deploy from staging to production by checking a box and clicking a button, even for database migrations).

If I have to go back and forth between two branches that actually do "interfere" a "reload all" generally re-syncs the state (literally just (require 'main.ns :reload-all) -- even while the app is running).

so with -M you're just creating arguments for clojure.main/main. And one of those arguments is -m which expects a ns-name, and it will call the -main function of that ns with any command line args

-M can have other arguments: just a script name, for example:

seanc@DESKTOP-30ICA76:~/clojure$ cat > script.clj
(println "Hello, World!")
seanc@DESKTOP-30ICA76:~/clojure$ clojure -M script.clj
Hello, World!

seanc@DESKTOP-30ICA76:~/clojure$ clojure -M -e '(println "Hello, Command-Line
!")'
Hello, Command-Line!

Take a look at tap> which lets you debug by sending values to specific "listeners" such as Portal or Reveal (or Cognitect's REBL).

Or you can just (add-tap println) and it'll behave like print debugging.

But the nice thing is that you can leave tap> in your code and add and remove listeners any time you need to debug.

oh interesting

looks like you can skip the -M but you get a warning

WARNING: When invoking clojure.main, use -M

😄

Yes. In the future, you'll have to use -M for clojure.main.

easing us into it

Right now, -A will also run clojure.main and :main-opts (in deps.edn aliases) but that will change at some point and it will only start a REPL.

(and if you use -A for main stuff, you'll get a warning that you should use -M instead 🙂 )

so you should use -M for aliases?

-X, -M, and -A all accept aliases and combine them and then a) exec a function, b) run clojure.main, c) start a REPL with those aliases.

I'm just learning about Crux, so I just started a REPL with this command:

seanc@DESKTOP-30ICA76:~/clojure$ clojure -Sdeps '{:deps {juxt/crux-core {:mvn/version "RELEASE"}}}' -M:rebel:reveal:add-libs:dev/repl

is dev is the file and repl is the function in it?

That starts Rebel Readline as my interactive REPL, starts Reveal for tap>'ing data into to visualize it, the add-libs alias brings in a branch of tools.deps.alpha that lets me add new dependencies without restarting my REPL, and :dev/repl runs my dev.clj script.

No, :dev/repl is just a keyword, an alias.

ah ok

See https://github.com/seancorfield/dot-clojure/blob/develop/deps.edn#L105-L112

cool that is really helpful

oh so you have aliases that you can use on any project?

Yes, in ~/.clojure/deps.edn

(on an XDG setup it's in ~/.config/clojure/deps.edn I believe)

that’s cool, lots to learn!

My :dev/repl alias -- via the dev.clj script -- also starts a Socket REPL so I can connect VS Code to it (using Clover).

i’ve seen you mention that a few times, what is the difference between a socket repl and nrepl?

Socket REPL is built into Clojure and has no dependencies at all. So we run them in several production processes.

You just specify a JVM option when starting up a Clojure program.

But most editor tooling expects nREPL. Hence I use Clover which supports Socket REPL instead.

But you can also connect via telnet directly to a Socket REPL 🙂

ha, awesome

tap> is pretty slick

would be great if tap> returned its argument

would compose a bit better

There's probably a tap-> coming in 1.11.

You can use (doto tap>) today for that.

(-> 42 (doto tap>) inc (doto tap>)) will tap 42, increment it, and then tap 43.

another discovery. only had seen (do)

doto applies functions to its argument and then returns the argument itself. Mostly used for mutable (Java) objects, but I've often dropped (doto println) into a -> expression to print an intermediate value and also returns it (`println` returns nil).

dev=> (doto 42 (println 1) (println 2))
42 1
42 2
42

yes, handy

i wanted to (tap>) in a thread-after series

->> col 
(work)
(tap>)
(work)
(tap>)

You can always start with -> and use ->> inside it as needed...

dev=> (-> [1 2 3 4]
 #_=>     (doto tap>)
 #_=>     (->> (map inc))
 #_=>     (doto tap>)
 #_=>     (->> (filter even?))
 #_=>     (doto tap>))
(2 4)

i immediately ran into the thread after

defn node-valid? [schema event]
  (->> schema
    (map (fn [[k v]] (leaf-valid? k v event)))
    (filter false?)
    (first)
    (nil?)))

attempting a decending map validator

or do this:

dev=> (->> [1 2 3 4]
 #_=>      (#(doto % tap>))
 #_=>      (map inc)
 #_=>      (#(doto % tap>))
 #_=>      (filter even?)
 #_=>      (#(doto % tap>)))
(2 4)

i have a map that's the allowed schema, and an event which i'm validating against the schema

yea thats useful

now looking at tail recursion and how do decend to N child nodes

*descend

oh the (comment) tip is great

next i want to get the repl into debug mode so i can use line breaks

i was pleasantly surprised when i read that i could run a socket repl on a container. That will allow me to design a cljs UI, allow input from the user as edn, have the cljs ship that to a socket repl to load-file or eval for correctness

crossing hosted runtime

can then validate the user-supplied edn, display some feedback on the ui, and optionally save the edn to a loadable file for later

code = data, data = code really is powerful

Hi Everyone, may I ask what is the most common way to transmit data from Clojurescript to Clojure, vice-versa, via HTTP server? From what I understand is that EDN is converted to JSON and back? And another way is things like transit ?

“next i want to get the repl into debug mode so i can use line breaks” — not sure what you mean by this?

I think both EDN and Transit are fairly common. EDN is a bit easier, Transit is generally a bit faster (esp. if you have larger amounts of data?).

Can I just use ring to give a "application/edn" response?

Do you know if there are any resources I can look at for this? 🙂

Okay i think this might be good https://swannodette.github.io/2014/07/26/transit-clojurescript/

vscode claims it can support line breaks during form evaluation. not sure how much its needed now with tap>

Why would I use something like a record without a body?

(defrecord MyRecord [a b c])

You might want to extend a protocol to support structured record

I was referring this file https://medium.com/@dashora.rajnish/how-to-create-apis-in-clojure-supporting-file-upload-and-data-transformation-using-ring-and-ad40fc3ca2d0 and I found request created as below format `

{...
 :params
  {"file" {:filename     "sample.csv"
           :content-type "text/csv"
           :tempfile     #object[java.io.File ...]
           :size         51}}
...}

how can i test post request while uploading file in clojure

Can someone tell me what I'm doing wrong here:

(defn build-map [acc [{:keys [hb-id status group]}]]
  (if (some? (acc hb-id))
    (update-in acc [hb-id :groups] conj group)
    (assoc acc hb-id {:status status :groups #{group}})))

(let [data [{:hb-id 1 :status "ERR" :group 1}
            {:hb-id 1 :status "ERR" :group 2}
            {:hb-id 2 :status "OK" :group 1}
            {:hb-id 3 :status "INFO" :group 1}
            {:hb-id 4 :status "WRN" :group 2}
            {:hb-id 5 :status "OK" :group 2}]]
  (->> (group-by :hb-id data)
       (reduce build-map {})))

{1 {:status "ERR" :groups #{1 2}}
 2 {:status "OK" :groups #{1}}
 3 {:status "INFO" :groups #{1}}
 4 {:status "WRN" :groups #{2}}
 5 {:status "OK" :groups #{2}}}

I think its my update-in I'm not understanding

But I tried this, and this works:

(let [hb    2
      group 3
      data  {1 {:status "ERR" :groups #{1 2}}
             2 {:status "OK" :groups #{1}}
             3 {:status "INFO" :groups #{1}}
             4 {:status "WRN" :groups #{2}}
             5 {:status "OK" :groups #{2}}}]
  (update-in data [hb :groups] conj group))
=> 
{1 {:status "ERR", :groups #{1 2}},
 2 {:status "OK", :groups #{1 3}},
 3 {:status "INFO", :groups #{1}},
 4 {:status "WRN", :groups #{2}},
 5 {:status "OK", :groups #{2}}}

group-by returns a map of {grouping [items]}. But in your reduce you're only expecting [items].

ie, calling first on your group by yields [1 [{:hb-id 1, :status "ERR", :group 1} {:hb-id 1, :status "ERR", :group 2}]] which is a different shape than expected by your reducing function

ah!! Yes, I see! Thanks. I'll have a rethink of how to do this, maybe group-by isn't the way to go

yes i think you could more easily just reduce over your input data

yeah, i think so!

well maybe not. you're almost there if you build a function that turns [{:hb-id 1, :status "ERR", :group 1} {:hb-id 1, :status "ERR", :group 2}] into {:status "ERR" :groups #{1 2}} i guess. although i haven't looked at the consistency of your data

This seems to work, I had the syntax wrong for desttructuring as well, had an uncessary [] around the {:keys}

(defn build-map [acc {:keys [hb-id status group]}]
  (if (some? (acc hb-id))
    (update-in acc [hb-id :groups] conj group)
    (assoc acc hb-id {:status status :groups #{group}})))

(let [data [{:hb-id 1 :status "ERR" :group 1}
            {:hb-id 1 :status "ERR" :group 2}
            {:hb-id 2 :status "OK" :group 1}
            {:hb-id 3 :status "INFO" :group 1}
            {:hb-id 4 :status "WRN" :group 2}
            {:hb-id 5 :status "OK" :group 2}]]
  (reduce build-map {} data))

that looks even more wrong to me

ah nevermind. the group by is gone

why did it let me do this?

(defn foo [acc [{:keys [bar quax]}]]
  
  )

(reduce foo {} {:bar :quax})
=> nil

(defn foo [acc {:keys [bar quax]}]
  
  )

destructuring is as recursive as the structure. (let [[[[[foo]]]] [[[[3]]]]] foo) returns 3

the binding forms to some extent mimc the shape of what you are destructuring

I guess the question is about why it is not barfing

It is a legal destructuring form, if you want to destructure a map as a first element of a vector given as a parameter

ok! Yeah, i was expecting it to either not compile or throw

And not only a vector, but any sequential thing. Perhaps it is calling seq on the map given as a parameter? Not sure.

i think what happened is that key-value pairs are associative and index, so that allowed []. and the hb-id was not destructured against the :keys thing. There seems to be some safety added in

(let [{:keys [a]} 1]) throws an error about type hinting a primitive. but the same thing inside vectors (let [[{:keys [a]}] [1]] a) returns nil

Is it common to convert seqs to vectors regularly? is the performance cost high? I find myself doing it frequently to be able to randomly grab elements from a seq. Limiting myself to small subset of contructs from clojure.core to be able to keep vectors as vectors doesn't feel right

I think random access to a seq is the weird part

it’s got to be at least O(n) to convert, with a bunch of frees and allocs by my guess

use the datastructure that provides efficient operations that you need

they get converted to seqs 😉

can you give an example? we can give some options that might help out

well, I know how to keep vectors as vectors but most functions work on seqs, so you have have a limited API to keep vectors as vectors, why kind of example do you have in mind?

Isn't it kind of obvious?

not particularly. can you give an example in your actual usage where you found yourself converting a seq to a vector?

I suspect i know the gist of what you're talking about but want a concrete case to delve into it

the answer is basically "don't write programs that mix mapping and filtering with random indexing", so @dpsutton's question is at a high level "can you show us your program that mixes mapping and filtering with random indexing so we can help you transform it into one that doesn't do that"

ideally your operations on seqs form a kind of pipeline

so that if you must convert back and forth, you have that on each end of a pipeline of a lot of operations

I’m curious about the problem space where you are “frequently .. grab[bing] elements from a seq”, i.e., where you need random access, in a context of processing seqs (mapping and filtering) @jjaws?

@hiredman yes, that's what I'm doing right now, doing lots of data massaging using sequence functions then converting to vectors at the end of each pipeline for random access

i was going to bring up transducers as well where you can control things a bit more

than it sort of depends on the pipeline

yes. which is why i wanted a concrete example

@seancorfield working with lots of CSV stuff, transforming it, then random access on fields

Can you be a bit more specific? I’d expect CSV data to end up as a sequence of hash maps, and then each row can have O(1) access to specific fields by name.

@dpsutton yep, transducers help with the 'no multiple passes over the data'

not really, transducers and a lazy seq pipeline will have the same number of passes

hiredman has a gist for this purpose here.

you want to be able to query this as a database essentially

transducers will remove some object allocation overhead for seqs, and will turn your pipeline into a very lean vector -> vector transformation using something like into

i'm reading through tim baldridge's build yourself a logic engine that creates a db like this as well.

@hiredman doesn't it avoid intermediate sequences?

it does

but intermediate sequences are traversed all at once

unless you are traversing intermediate results for some reason

@seancorfield don't see the difference, both are associative, maps will still be converted to seqs

@jjaws I suspect @seancorfield is thinking about (map f all-rows-in-csv) and you are thinking about (map f a-row-in-a-csv)

oh ok, yes, was thinking of a-row-in-a-csv vectors vs maps

and this is kind of why a more concrete example can help

I'll try come to up with something small but not so contrived

if you need a query language over your csv, you could look into datascript or your own hand rolled triple store or set/index can be useful as well

regarding clojure.spec, it would reason that the more common use cases would be for library design as well as external input validation. am i correct with this assumption?

https://corfield.org/blog/2019/09/13/using-spec/ — we’ve been very heavy users of Spec in production as well as dev/test since it first appeared.

learning via lurking... 🙂

Hi all ! newbie here! anyone using neovim 5.0 and clojure-lsp ? some pretty basic config in lua ? thanks in advance.

welcome 🙂