Clojurians Log v2

depending on how simple your tool is, you may not even bother with aliases or a deps.edn at all

for example https://git.sr.ht/~hiredman/lions doesn't have a deps.edn at all, and the README just lists a bunch of different clj -X ... invocations

Can anyone recommend a tool for clojure/cljs code scanning for SAST (Static Application Security Testing)?

Hi there, I am trying to run clj -X:some-alias to use a tool. But all I get is a -X:some-alias (File or directory not found). So apparently -X is not recognized as a parameter. Any Ideas what might cause this?

What version of clj do you use? It's on the first line of clj -h.

Its not shown for me

clj -h
Usage: clojure [dep-opt*] [init-opt*] [main-opt] [arg*]
       clj     [dep-opt*] [init-opt*] [main-opt] [arg*]

The clojure script is a runner for Clojure. clj is a wrapper

I installed the 1.10.2 rc1 from the AUR on Arch. Unless something went wrong it should be the newest available.

I have no clue, sorry. I've installed the latest release by following the instructions at https://clojure.org/guides/getting_started This is how the very first section of the output looks like:

$ clj -h                                                                
Version: 1.10.1.763

You use the Clojure tools ('clj' or 'clojure') to run Clojure programs
on the JVM, e.g. to start a REPL or invoke a specific function with data.
The Clojure tools will configure the JVM process by defining a classpath
(of desired libraries), an execution environment (JVM options) and
specifying a main class and args. 

Perhaps the AUR has something else as clj, something that doesn't come from the original sh installation script.

Thanks for looking into it. It must be an error on my OS end. I will investigate futher!

You could try just clojure?

clj is packaged with the clojure-tools distribution

I use Clojure on arch, and just recently (i.e., 30 mins ago) started to use the updated depstar which uses -X now

all works well

@avivak Do you need to have some off the shelve thing that looks for certain security things or do you want a tool that you can scan code with and look for certain things yourself?

off the shelve tool

Hey folks! Bit of a security question here. Playing around with read-string :

user> (defn foo [] :bar) => #'user/foo
user> (foo) => :bar
user> (read-string "foo") => foo
user> ((read-string "foo"))
Execution error (ArityException) at user/eval244751 (form-init5501808611653817338.clj:455).
Wrong number of args (0) passed to: clojure.lang.Symbol
user> ((resolve (read-string "foo"))) => :bar

@david.russell What do you mean with inject function calls ?

Btw, we just had a conversation about read-string in #clojure-europe which led to this script for finding read-string in libs: https://gist.github.com/borkdude/57984ca1df6c3cf8f302196cb37b0f43 For example I found this namespace that has a couple of them: https://github.com/cognitect-labs/aws-api/blob/master/src/cognitect/aws/shape.clj

Maybe continue in a thread. @david.russell

@borkdude I guess what I mean is that there appear to be certain ways by which clojure forms become malicious. eval is surely one, but in the example above (unless I'm misunderstanding something 🙂) someone could make functions calls in your namespace if your business logic happens to call resolve (and then executes) a symbol parsed by read-string.

So I'm trying to get a sense of the ways that injected clojure data could be abused (other than being data itself, which should be checked for in user-facing codepaths anyways)

yes, calling functions resolved by dynamic user input can be dangerous

there are ways around this, e.g. using a sandboxed eval like https://github.com/borkdude/sci

Ah that's cool. Between sci and edn/read-string I guess most use-cases are handled, at least for code you have control over

@david.russell use clojure.edn/read-string. that is safe without all the eval concerns.

@david.russell See the long-ish description on this page about bad things that can happen if you use clojure.core/read, all of which apply to clojure.core/read-string, too: https://clojuredocs.org/clojure.core/read

@borkdude that use of read-string in the aws-api lib is qualified with a condition that requires the string to contain only ASCII decimal digits, so looks pretty safe to me. Requires some code examination to determine it is safe, but that one is pretty local to the read-string call, thankfully.

Or rather, one of the calls to read-string is as I described. There are others...

thanks for pointing that out @borkdude. the aws-api shape stuff needs an overhaul

@andy.fingerhut thanks! That section on unexpected effects with *read-eval* disabled, is exactly what I was looking for. Super interesting. I'd be curious to look into what other unexpected read-string behavior still exists in Clojure 1.10...

The examples there I believe do not work in Clojure 1.5 or 1.6 and later, but they should scare readers enough to make them want to avoid clojure.core/read and read-string except for highly trusted data sources. If you come across scary examples that still exist in Clojure 1.10 when *read-eval* is false, I would be happy to add them to that page. Such examples are not necessarily quick or easy to find, though.

(FYI, anyone can add more examples to that page with a free http://ClojureDocs.org account, not just me)

I’m using compojure and the recommended way to apply middlewares is to use wrap-routes to avoid middleware from routes that didn’t match from being applied… the issue is that it seems to apply middlewares in reverse… outermost wrap-routes middleware gets entered last instead of first… any way around that?

Team, If I run below function in repl it is giving me 2 value RX, why this is giving 2 values?

(re-find #"(\S+)" " RX packets:1871074138 errors:5 dropped:48 overruns:9")

["RX" "RX"]

I'm unable to use println in a go block

(defn -main
  "I don't do a whole lot ... yet."
  [& args]
  (go
    (println "here")))

How you are calling function?

it's my main function that's invoked when I do lein run

because it uses re-groups to return groups

(defn -main
  "I don't do a whole lot ... yet."
  [& args]
  (println "foo")
  (go
    (println "here")))

is there any other function go in your file?

no, i'm just playing around with core.async

the thread which is executing -main function finish before go block

ahhhh

Thanks!!!

Is there any way where we can return 1 value?

When you have NO "capture expressions" in your regex, like the (\S+) part is, then re-find returns only one matching string for the entire regex, or nil if no match was found.

If you have capture expressions and a match is found, then in addition to returning the entire string that matched the complete regex, you also get additional elements in the returned vector that matched each of those capture expressions.

you could change regex like #"\S+" in that case no groups will be returned

If you want that, great. If you want to parenthesize something in a regex but do not want it to be a capture expression, you can put a special character sequence just after the left paren, which if I recall correctly is ?:, i.e. #"(?:\S+)" for your regex.

capture expressions can help you write a single longer regex, but pull out sub-pieces of the match, all at once.

(def text "My string ${hello_world}") => #'user/text (def nvar-rex #"\$\{(\S*?)\}") => #'user/nvar-rex (def matchr (re-matcher nvar-rex text)) => #'user/matchr (re-find matchr) => ["${hello_world}" "hello_world"]

how it gave one with $ and one without dollar symbol?

The parenthesized part of your regex does not include the \$

you parenthesize the parts of the regex you want, to capture the parts you want (or disable capturing if you use (?: ...)

Oh I understood 🙂 Thanks 🙂

you shouldn't print from a go block or do any IO because it can/will starve the executor pool

(it's fine while exploring but I have seen many deadlocks in the wild caused by IO in the go block)

I am trying to use jSerialComm (https://fazecast.github.io/jSerialComm/javadoc/com/fazecast/jSerialComm/SerialPort.html) from clojure, and I have stumble to the following issue: i am trying to use setComPortTimeouts method (https://fazecast.github.io/jSerialComm/javadoc/com/fazecast/jSerialComm/SerialPort.html#setComPortTimeouts-int-int-int-) with both TIMEOUT_READ_BLOCKING and TIMEOUT_WRITE_BLOCKING and the docs indicate that I should OR them in the first argument which i am not sure how to achive it via java interop. As an example i found the following snipet:

public SerialPort openSerialPort(final SerialPort serialPort,
                                 final long timeout) {
    serialPort.setComPortParameters(getSerialPortBaudRate(),
            getSerialPortDataBits(), getSerialPortStopBits(),
            getSerialPortParity());
    serialPort.setComPortTimeouts(SerialPort.TIMEOUT_READ_BLOCKING
                    | SerialPort.TIMEOUT_WRITE_BLOCKING, (int) timeout,
            (int) timeout);

    if (!serialPort.openPort()) {
        throw new IllegalStateException("failed to open serial port: "
                + serialPort.getSystemPortName());
    }


    return serialPort;
}

user=> (run! prn (sort (apropos "bit")))
clojure.core/bit-and
clojure.core/bit-and-not
clojure.core/bit-clear
clojure.core/bit-flip
clojure.core/bit-not
clojure.core/bit-or
clojure.core/bit-set
clojure.core/bit-shift-left
clojure.core/bit-shift-right
clojure.core/bit-test
clojure.core/bit-xor
clojure.core/unsigned-bit-shift-right

I’m intrigued by the connecting-things-together setup in Clojure Applied using Component, where all components get supplied core.async channels to take in and give out data. This seems to rely on the (component/system-map ,,,) being called within a system-map generating function, so that there’s one obvious to actually make those channels: a let before calling system-map. Does anyone have any analogous examples for integrant or clip with aero, where the component map is more usually an edn file?

As an example of pulling out the pieces of interest, if you were parsing lines like this "RX 1234 packets TX 4321 packets", you can do the following:

user=> (re-find #"RX (\d+) packets TX (\d+) packets" "RX 1234 packets TX 4321 packets")
["RX 1234 packets TX 4321 packets" "1234" "4321"]

For that particular case, you need bit-or - it's the equivalent of Java's |.

thanks i am going to try them

Hi! What does the dash/minus symbol in front of protocol method mean?

usually - nothing, just part of symbol

I think it may be sometimes used as a convention to mean "internal implementation function, not intended for casual/public use" ?

But given that when one use :gen-class in Clojure and then defines a function named -main, I suspect there may be more to it than merely convention.

that is true for top level public functions, but not for protocols

Yes, my question is more about protocols.

for example https://github.com/seancorfield/next-jdbc/blob/develop/src/next/jdbc/protocols.clj

cc @seancorfield

I think you can ask author here

Maybe I just copied what I saw and rationalized it later, but at least in my code, the - is because there's typically an API function with the same name but without the dash. That one then calls the one with the dash, which gets dispatched.

The protocol functions use a leading - when they are "just" implementations for wrapper functions. Protocols can't be variadic so it's common to provide multi-arity wrappers.

In next.jdbc, the execute and transaction functions all have multi-arity wrappers. prepare and the two get-* protocol functions only have the specified arity so I didn't bother with wrappers.

(it would probably be more consistent to have wrappers for all of them but it isn't necessary and I have tried to avoid unnecessary overhead in that library)

Thank you for explanation, that makes sense.

Pretty minor nitpick but > Protocols can't be variadic this isn't true is it?

(defprotocol X
  (foo [this] [this n]))

(foo
  (reify X (foo [_] 3)))
;;;3 
(foo
  (reify X (foo [_ n] n))
  6)
;;; 6

Variadic: & args -- they can have multiple arities.

Ohhhh, gotcha

But in next.jdbc the multi-arity wrappers exist to provide default argument values. Mostly (foo x y) means (foo x y {}). But there's no point in declaring multiple arities of protocols for that since all implementations would need to be provided when reifying it.

If the implementations were likely to be different for different arities, then yes, you'd definitely want explicit multiple arities in the protocol itself.

There are other reasons for wrapping protocols: you can't instrument a protocol function so you need wrappers for those, as I recall.

@seancorfield I wonder why not:

user=> (defprotocol Foo (bar [_]))
Foo
user=> (var? #'bar)
true

Hi Team, I am new to clojure , I was debugging some production code (which has reduce inside reduce.).. Can some one guide me how can I debug clojure code in a best way?

https://clojure.atlassian.net/browse/CLJ-2109 does not explain why... 😞

That's a broad question; try https://github.com/gfredericks/debug-repl

there’s a debugger in cider and one in cursive (maybe others but those are the ones I’ve used). But honestly I hardly ever use them and don’t miss them. Typically the best approach is to break up complicated code in smaller functions, and use the repl to check the individual parts. I also very frequently use timbre/spy to print out intermediate results in threaded parts of code.

If that codebase is running on Clojure 1.10, you could add tap> calls in various places and then use add-tap to capture/log the "tap"'d values.

The nice thing about tap> is that you can leave it in production code and it does "nothing" unless you have an active add-tap listener in place (since you can remove-tap when you're done).

But, yeah, mostly the recommended way to work is to debug these things locally via the REPL (connected to your editor) and focus on small functions.

Does reduce inside reduce is same as for loop inside for loop?

There was an article going at length about using "-foo" in protocols and providing fn wrappers "foo" but I can't find it now 😞. Read it ~5 years back. CLJS does this extensively:

(defprotocol IIterable
  (-iterator [coll]))

Thanks team for response

I was looking into the implementation of source because I have my own implementation in sci/babashka, but this relies on location information which I might want to get rid off some day. So I took a closer look at clojure.repl/source and figured it's a bit too straightforward in reading the source. E.g. it can't deal with aliases used in functions without throwing:

(ns foo.core
  (:require [clojure.string :as s]))

(defn foo []
  ::s/foo)

user=> (source foo.core/foo)
Execution error at user/eval277 (REPL:1).
Invalid token: ::s/foo

Part of me wishes you could just parse any Clojure form without any namespace / alias context, but for the source function that doesn't even matter: it's only interested in the string it successfully parsed, not the form itself.

This could potentially be fixed with (binding [*reader-resolver* resolver] ..), I'll try that

All I found is this thread: https://groups.google.com/g/clojure/c/YwhOGCWquIM/m/bLC1YYGxe8MJ

Gist: "Recently the received wisdom has been: protocols are a low-level implementation detail. Actual APIs should be built with normal functions that call the protocol methods."

That works. Let me know if an issue / patch for this is welcome... (cc @alexmiller)

This fixed worked:

(if (= :unknown *read-eval*)
              (throw (IllegalStateException. "Unable to read source while *read-eval* is :unknown."))
              (binding [clojure.core/*reader-resolver*
                        (reify clojure.lang.LispReader$Resolver
                          (currentNS [_] 'user)
                          (resolveClass [_ _] 'class)
                          (resolveAlias [_ sym] 'alias)
                          (resolveVar [_ _] 'var))]
                (read read-opts (java.io.PushbackReader. pbr))))

@popeyepwr For basic questions, you'll find #beginners a much more welcoming space. Folks there have opted in to spend plenty of time with folks who are new to Clojure.

thx @seancorfield: i remember reading about the tap> mechanism in the release notes back then, but completely forgot about it 🙂

tap> is awesome 🙂

My 6th sense says taps were added to deal with dynamic bindings being hijacked; like trying to debug a macro in cljs prints into the compiler output instead stdout because *out* is hijacked for that purpose.

Popeye I would recommend you read Programming Clojure, or Brave Clojure. I found it is more practical to go through a comprehensive introduction. The question is answered by the concepts of recursion, seq functions, function composition. These things require a bit of a mental shift if you’re used to imperative programming, so it’s very beneficial to open a REPL and follow through increasingly more complex examples, while doing some explorations yourself. The REPL is the best teacher I found.