Clojurians Log v2

Anybody has a trick for discarding an argument when using #() ?

Say I'm doing: (mapv #(rand-int 10) (range 100))

I guess I can do: (mapv #(do % (rand-int 10)) (range 100)) hum... :thinking_face:. But that's the same as using fn[_]

Perhaps not what you’re looking for but:

(vec (repeatedly 100 #(rand-int 10)))

Good answer 😛, but ya I was just using this as an example. I kind of often stumble on scenarios where I don't care about the input, and would like a short way to wrap my thing in a side-effect. Like when I use agents for example, sometimes I don't actually care what the current agent value is.

Right… yea I don’t know of a way to ignore it in those cases; accidentally, on ClojureScript your example works, but that’s very much by accident because of the different way JS works 🙂

@didibus (mapv #(#_% rand-int 10) (range 100))

You can use it to ignore multiple anonymous arguments too:

user=> (mapv #(#_%1 #_%2 rand-int 100) (range 10) (range 10))
[47 57 65 16 15 28 56 72 10 82]

(I know this is a bit of a repeat, but I wanted the channel to see it, in a single coherent response 🙂 )

Wow, neat, haha, that's the exact kind of trick I was looking for.

Ya, I'm a bit worried about this maybe being too depended on accidental implementation details of the Clojure reader though

Like it seems the reader first process the forms with #(), and then with #_

But is this order guaranteed?

Nice, it works in different orders too: (mapv #(rand-int 10 #_%) (range 100)) which I find a bit more readable

Wouldn't simple fn be even more readable?

that's super neat and crazy and phenomenal to know, but if this is for real code and not playing around, that is absolutely not the trick you are looking for 🙂

Well, yes and no. Yes cause its ugly to put #_% at the end, and most people might be thrown off by it. But no, for the same reason Rich Hickey added `#()` in the first place :stuck_out_tongue:

Re: different orders -- yeah, it's not going to make any difference where the ignored arg form is. I originally stuck it in the middle: #(rand-int #_% 10)

Well, I don't fully know why he did, but for me, there's something visually nice about the parenthesis not repeating.

Looks like it works in ClojureScript as well

But not in Babashka 😞

File a GitHub issue! @borkdude will be thrilled by this weirdness 🙂

Haha, he does like that kind of stuff

Yes, I've still not decided if I'm that kind of madman or not haha

its 100% madman. and anyone reading it will be quite confused

But I will def use it when messing around at the REPL

I just looked over the source of the Clojure reader and I think you can rely on this behavior: the ArgReader (which processes % and %<n>) is what tracks the highest arg count in an expression, and the DiscardReader (for #_) has to read the next form in order to discard it. The ArgReader is a reader macro so it will be triggered just by reading. So the arguments are always going to be read (and tracked), even if the resulting form containing them is subsequently discarded.

@dpsutton Have you seen my "trick" with discard forms in deps.edn so you can embed code forms that can be eval'd from an editor?

no. is it in your deps.edn repo?

No, I showed it in my RDD talk to Clojure Provo. I'll show it in my London talk too.

i couldn't make the provo one. if london is convenient for my time zone i'm gonna definitely be there

Because I use add-libs from t.d.a to add deps to a running REPL, and I don't want deps.edn to get out of sync, I add an ns with a :require of t.d.a's repl namespace and then put a repl/add-libs call between :deps and the hash map. Then with just a minor edit, you can run add-libs with all your deps, and then with a minor edit, turn it back into valid EDN.

@dpsutton January 12th, 10:30 am Pacific time.

that's not so terrible. 8:30 here in central. i'll be there with some coffee

Surely it's 12:30 Central?

Its not possible that the DiscardReader runs first, modifies the form, and then the ArgReader would run, no longer seeing the discarded code?

Or to reader macros like all run on the same original forms?

https://github.com/babashka/babashka/issues/702

Nice, it also works in Clojerl

Just in case: - https://stuartsierra.com/2019/09/15/clojure-donts-numbered-parameters - https://stuartsierra.com/2019/09/11/clojure-donts-short-fn-syntax

DiscardReader invokes the reader on the following form: The form to be discarded has to be read. The ArgReader is how the % forms are read. Could the code be changed so that any reader side-effects could also be erased by the DiscardReader? I guess it could but it would be a fair bit of effort to do it correctly without breaking anything I suspect. That said, I'm sure Alex and Rich would say, absolutely don't do this 🙂

@p-himik Yeah, if ever I find myself reaching for %2 I take a step back and think about readability!

I guess it goes to show how little excitement I have in my life, but seeing #_% within a # function riles me up just as much as reading political news does. And honestly, I'm a bit surprised by that myself. :)

Ya, but:

(let [agents (repeatedly 5 #(agent []))]
  (run! #(call-api client) agents))

(let [agents (repeatedly 5 #(agent []))]
  (run! #(call-api client #_%) agents))

(let [agents (repeatedly 5 #(agent []))]
  (run! (fn [_] (call-api client)) agents))

Maybe I'm just waking up, but how does having agents affect anything in your code? It seems like it can be just

(dotimes [_ 5]
  (call-api client))

Oh sorry

(and I absolutely without a shadow of a doubt prefer the fn version)

Ya, bad example, well. I can't remember what the situation was, but, when you do lots of doing side effects inside loops there's a few times it comes in handy

Oh, now I remember, it was:

(send-off agnt #(call-api client))

"Handy" does not mean "worth it". :) So many more things come in handy in other languages - and all of those are almost exclusively the reason why I stopped using them. Or rather, over-reliance on such things by colleagues and overall language community. Oh, I just figured out why I feel so strongly about #_% - it brings back the memories of having to write "clever" C++ code that heavily relied on macros, templates, and quirks of a particular version of MSVC.

I don't disagree, but some things are handy and worth it. Now this particular one, I don't think is worth it, cause it does still seem like an accident that it works.

Hahaha... Ah, yes, that brings back memories of being on the ANSI C++ Committee for eight years and having several discussions with the MS rep about VC++

(send-off agnt (fn [_] (call-api client))) ?

That said, in clojure, not all scenarios have the same level of needing to be worthy. Sometimes I code Clojure on my phone for example, and on such a device, edits are really hard lol, so this is a nice trick to know. Same thing, sometimes I do things on a command line REPL with terrible read-line support, so you can't move the cursor back, you have to delete everything, etc. So I can see scenarios where this is useful

But... honestly this syntax is growing on me, I feel somethings like this as well are a matter of idiom, people could pretty easily get used to it. #(rand-int 1 #_%) If I read it as: "call rand-int with arg 1 and discard passed in argument" its not that bad actually. A bit like how using _ is an idiom when you discard an arg in fn

I won't send a PR with it though I swear 😋

You are totally correct. Had that backwards. Thanks

This feels like clojure behavior that needs to be silently smothered in 1.11 before anyone realizes it exists

I have the following code that I’m trying to access a webpage with on the route about/something, but I’m getting 405 error:

(ns humboiserver.routes.home
  (:require
   [humboiserver.layout :as layout]
   [<http://clojure.java.io|clojure.java.io> :as io]
   [humboiserver.middleware :as middleware]
   [ring.util.response]
   [ring.util.http-response :as response]))

(defn home-page [request]
  (layout/render request "home.html" {:docs (-&gt; "docs/docs.md" io/resource slurp)}))

(defn about-page [request]
  (layout/render request "about.html"))

(defn home-routes []
  [""
   {:middleware [middleware/wrap-csrf
                 middleware/wrap-formats]}
   ["/" home-page]
   ["/about"
    ["/something"
     (ring.util.response/response {:something "something else"})]]])

@ps pardon my ignorance, but I can't tell from that snippet what your router is - you have a function that returns a data structure that is clearly meant to describe routes, but no indication of what program is using that structure

reitit

405 indicates that the request method is wrong, but nothing in your route description indicates what methods are valid

@ps the reitit examples I see don't use data structure nesting for child routes, they use it for enumerating request methods

they would have ["about/something" ...] and [about/something-else] as separate entries

but it works if I have a layout/render with that route

OK - I'll let someone that knows reitit help

I was thinking that it had something to do with incorrectly using ring.util.response

I would be very surprised if that caused a 405, a 405 has a precise meaning, and to me that points to giving something else where reitit thinks it's getting data describing request methods

what should I do to diagnose and fix this?

i think that the response map has to be wrapped in something, but I don’t know what specifically

Just a guess - but your home-page and about-page need to return {:status 200 :body <html>}

well, the other route is taking as an argument a function that takes a request and renders a response

your broken route is rendering a response inline with the data, before seeing a request

@lukaszkorecki that's what ring.util.response/response is doing - it doesn't do much else actually

I made it an anonymous function

@ps a hunch - it doesnt' error since a response map is a callable, but it just fubars when it gets passed a request

but that gives wrong number of arguments

@ps well it should take a request

@noisesmith Right, but the root route ("/") is using home-page function directly, in the snippet, ring.util is used in only one place. That said, I'm just guessing here - not sure what that router is

it's reitit

ring is very smart about coercing results, the error here is happening on the layer of route dispatch

it works when wrapped in (fn [req] …)

that's what I'd expect, cheers :D

Hello. My task is process an indefinitely long reader My first approach was a simple loop/recur But then I had the idea of implement as a lazy-seq I have some questions 1. read-all use non-tail call recursion. it may run into "stackoverflow" problem? 2. read-all will cause some GC issue? Just clean nodes at end or something like that? 3. There is any advantage of loop/recur approach?

(letfn [(read-all
          [rdr]
          ;; [clojure.data.json :as json]
          (let [v (json/read rdr
                             :eof-error? false
                             :eof-value rdr)]
            (when-not (identical? rdr v)
              (cons v (lazy-seq
                        (read-all rdr))))))
        (proc-all-loop [rdr]
          (loop []
            (let [v (json/read rdr
                               :eof-error? false
                               :eof-value rdr)]
              (when-not (identical? rdr v)
                (my-proc v)
                (recur)))))
        (proc-all-lazy [rdr]
          (run! my-proc (read-all rdr)))]
  ;; which is "better"
  (proc-all-loop *in*)
  (proc-all-lazy *in*))

@souenzzo the classic problem with laziness is resource usage, here you can't really know when to close the reader / the stream the reader is built on

(in the lazy version that is)

if your intent is to eagerly consume, and you throw away the produced values (via run!)I don't know why you are using laziness

proc-all-loop

semantics are clear, no laziness other than waiting on the reader

lazy-seqs don't cause stack overflow unless you nest large numbers of unrealized lazy transforms, which is caused by mixing lazy and eager code sloppily

(usually - I mean you could have (->> coll (map a) (map b) (map c) ...) until the stack blows up but your code would be a huge mess before that happend...)

If you want to be able to handle the whole thing in sequence, you can make an IReduceInit from the reader that will be invalid when the reader is closed

Since your semantics really aren't the same as a lazy-seq - 32 elements at a time will block and maybe deadlock your program

or, sorry

right, but using lazy-seq directly won't impose that chunking

just an iterator

oh it wont?

nvm ignore me

other ops that take multiple collections could take that lazy-seq and return a chunking one, but that's more convoluted

@emccue and the root point is a good one - lazy-seqs are bad for situations where realizing an element blocks or changes the state of some resource

(defn reducible-json-rdr [rdr]
  (reify IReduceInit
    (reduce [f start]
      (let [v (json/read rdr :eof-error? false :eof-value rdr)]
        (if (identical? rdr v)
          start
          (recur f (f start v))))))

^ just because it always feels neat to write out

IReduceInit needs an init value

public interface IReduceInit{
Object reduce(IFn f, Object start) ;
}

(reduce [_ f start] ...)

oh yeah this

(defn reducible-json-rdr [rdr]
  (reify IReduceInit
    (reduce [_ f start]
      (loop [value start]
        (let [v (json/read rdr :eof-error? false :eof-value rdr)]
          (if (identical? rdr v)
            value
            (recur (f value v)))))))

it's the simple loop/recur wrapped on reify reduceinit wrapped on a function

Don’t forget IReduceInit implementations need reduced? handling

Hi all. I need some sort of bounded queue that automaticaly drops elements older than X (seconds, minutes, hours, whatever). In general terms, every time I add a new element to the queue, I want it to remove elements that don’t satisfy a certain predicate. I was trying to accomplish that using sorted-set and disj but I’m not sure if this is optimal, something roughly like this:

(let [queue (sorted-set 5 3 4 1 2 9 6 8 7 0)]
  (println queue)
  (println (apply disj queue (take-while #(< % 3) queue))))

#{0 1 2 3 4 5 6 7 8 9}
#{3 4 5 6 7 8 9}

There are many many options for this kind of thing, but you may need to work on your requirements to figure out what you actually want

e.g. doing stuff on a time limit is much easier then doing stuff for an arbitrary function

do you need immutable data structures? is this building some kind of cache? etc

@hiredman Imagine an in-memory collection of maps, each containing a :timestamp field whose value is a ZonedDateTime. Every time a new element (ie: a new map) is added to the collection, I need to remove all elements older than, say, 24 hours.

that is a bounded cache with ttl eviction

Correct (Thanks, I was also looking for the terminology 🙂)

strange to see the ttl requirement enforced solely on addition and not retrieval

@dpsutton In my case the eviction on retrieval would be a nice bonus.

seems a necessity

For my purposes, not strictly necessary.

if you don't add anything for 7 days, everything is evicted, but if that's only enforced on addition you'll get bad data

the reason arbitrary function vs. time matters is you can build an index based on a known field, but not on an arbitrary function

@dpsutton Not a problem for my application.

then i think you can use clojure.core.cache. the caches let you get a seq or iterator of the underlying hashmap that keeps the values. and when getting the iterator or seq, the cache invalidation is not respected (ie could have expired things in there)

https://github.com/clojure/data.priority-map

Thanks @dpsutton I think that’s what I need.

i think they are composable. ie, you can wrap a ttl around a bounded queue one.

that’s awesome

Does anyone have any recommended best practices for doing remote REPL work in a sensitive data environment (HR/accounting etc)? Permissions, policies, technologies, ACL? I think auditability, monitoring, and permissions are the primary concerns here.

a good baseline is ssh access, with the same user as the app runs under - and don't provide access to anyone you wouldn't provide a root shell on that machine to

I think going finer grained would just be a mess - there's too many ways to get permissions in a jvm, and no way to truly hide data once you have vm access

How about something like auditability or monitoring of REPL sessions?

Ever worked with anything like that?

(by ssh access, I mean tunneling on an ssh connection, and the standard logging of ssh access)

would ssh access log remote REPL stuff?

beyond the layer logging of when connection happens, I think there's too many ways to undermine it

true 😕

of course you could take clojure.main and make a logged version, then make a policy of "always use the logged repl"

interesting, I didn't know that was a possibility. Makes total sense!

but that's not very easy to enforce - it's so easy to get a repl once you have a connection

true :thinking-face:

@goomba yeah, at the root REPL is just a loop, you could say "only use this specific repl", and then check the logs, but there's still some layer of honor system there surely

do you suppose most folks just use the honor system? I mean, surely someone out there uses the REPL on sensitive systems

if its nrepl you could have a middleware that logs all messages back and forth

sure - I guess I'm no expert, I'm just reasoning first principles on what one can do once you have a repl for the most part

@dpsutton right, sure - that's easy to attach to any repl, the hard part is actually enforcing that that repl is used and not modified

clojure.main is not a lot of code, a logged version is an afternoon project at most

remote repl sounds like something else manages the process. you have whatever repls it exposes. seems like a logging nrepl server exposed on that would be the easiest

I guess there's always "some things are logged, if it looks like you are doing something shady you better have a good explanation", but that's nearly implicit on remote hardware

if its socket repl it might be even easier to have the functions spit their ins and outs to a file. dunno. just thinking of ease of use tooling wise. nrepl is pretty standard to work with

@dpsutton that kind of sandboxing is fragile and illusory

with clojure that is

"something else manages the process" - until you run a single line of code that starts a new unlogged repl

for example, someone on #clojure IRC found a one liner that turned the number 3 into the number 5

no way! 😮

it's like that Jimi Hendrix song

of course that didn't affect cached unboxed values

but otherwise, the cached Long instance of 3, was changed to now contain 5

it was remarkable that some things didn't break lol

Why expose a remote repl to being with? Can it be avoided? If not, what about exposing a sci session instead?

sci session?

remote repls are great

Small Clojure Interpreter

That way you can't modify the running program

but you have to trust whoever has access to the repl

that's a big assertion to be making

They are, but with great power and all. They're a security nightmare

people can modify running C programs lol

I think "trust but verify" would be an acceptable solution (in my particular situation... not dealing with nuclear missiles or anything)

Well, sci is a better sandbox than just giving someone repl access. You can control which functions are exposed, for example

@ben.sless I'd take sci over nothing! As long as there's a jdbc connector

I would be so annoyed with sci

There is everything you would expose to the sandboxed environment

the crazy stuff I've down with a repl over the years just would not be possible

(that sounds like it should be its own channel... #hiredmanstories 😄 )

It's a compromise. For every crazy creative programmer you have a stumbling newbie who can break production

So does everyone either just roll the dice or not provide a production REPL...?

We don't expose production repls

Even for data analysis?

"oh, this doesn't have as much instrumentation as I would like, but I want to monitor it when it gets first deployed" -> write a program to connect the remote repl, execute code that reflective walks some objects and pulls out numbers and sends it back so I can stick it in a locally running graphite

would love to read that blog post

oh, we lost a bunch of customer data and need an error recovery process -> write it up, stick it a (company controlled) pastebin, use pssh to load it into the repl of every server via slurp

A lot can be achieved with running locally with production data. Another compromise is running in a staging environment which mirrors or reads production data but can't change anything

but yeah, repl access is the keys to the kingdom, so if you can't trust people then don't give it to them

Yeah, fair enough. I guess that's the bottom line

I think this is all dev-trust complete. The most sensitive thing (liability wise in particular) is the customer data. Either a dev can be trusted to access it responsibly or not, the rest introduces a lot of work and frustration with little established benefit.

So I guess I need to figure out a way to find a small, read-only, sand castle kingdom 😛

@goomba there's a lot you can do with configurable loggers plus an environment where a repl can process those logs

then security obfuscation / monitoring can be introduced as a middleware - you have a lot more control

puts me in mind of https://twitter.com/QuinnyPig/status/1346339906902130689

@noisesmith so you're saying something like, use a REPL to consume/transform obfuscated logs (as opposed to directly consuming data?)

some of production servers don't always run a repl server now, you have to restart them with a special flag to turn on a repl, and even that bums me out

@goomba right, logs as data (or even db entries used as if they were logs, ordered by timestamp), plus a separate process (not the production app) to consume and manipulate that data

for example all sensitive customer info (everything identifiable) can be UUIDs / numeric ids, whatever pointing to a separate table you shouldn't need for dev

you can debug app logic / data flow using the id to id correlation without exposing anything especially important (at least not in an easy to extract way...)

Yeah makes sense. Hopefully someone listens!

apropos of repls, for some reason a while ago I wanted the feature I think a number of clojure ide kind of enviroments have, where you can can get a repl running the context of some other running code, I didn't have that feature so I wrote this code to "throw" a repl over tap https://gist.github.com/a2630ea6153d06840a2723d5b2c9698c

it is kind of neat

hiredman has said most of what I was going to say -- yes, we run socket REPLs in several of our production processes; yes, we ssh tunnel into production and connect a client to production (I connect VS Code on my desktop to production sometimes 🙂 ); since we AOT-compile our uberjars with direct-linking enabled, there's a limit to what we can redefine dynamically -- except in a couple of legacy processes that load Clojure from source at runtime (for reasons) and those can be live-patched all day long. Perhaps one of the most important considerations here is that any process that runs Clojure can be told to start a REPL via JVM properties at startup -- no code is needed inside the Clojure codebase, so anyone who has access to how a Clojure-based process is (re)started can enable a socket REPL in it, and then you have unlimited access, assuming you can get network access to that socket!

if I understand what that does, that's cool

you might be able to use http://clojure.github.io/clojure/clojure.core-api.html#clojure.core/PrintWriter-on too, not sure

you put a call to start-repl somewhere in your code, then run your code, then in a repl connected to the same process (if your code runs in another thread you can use the same repl that you used to run your code) you call wait-for-repl, and once execution hits start-repl the repl where wait-for-repl is running is taken over and inputs and outputs are forwarded to and from a repl running where the call to start-repl is

yeah, PrintWriter-on looks handy, I need to remember it next time I write one of these

Reminds me a bit of https://github.com/technomancy/limit-break (although it's different)

well it was born out of doing this kind of stuff from prepl :)

I'm no ssh expert, but I'd be surprised if there was no way to log what is transferred through the ssh connection

At the very least, ssh should have access logs.

The thing is, the REPL won't give you more power than the SSH itself. Once I'm SSHed in, I can simply replace the service with another one, change the class files or source files, I can read the computer memory, steal the credential files, etc.

Well root ssh

So if that's allowed, the REPL through SSH isn't any riskier

You could argue the data to steal is made more obscure without a REPL, but :man-shrugging:

@didibus sure, but once I am in a jvm with a clojure process I can open up any method I find convenient to communicate - I'm not limited to the repl I first connected to

access logs are a great start, and maybe even logging what comes across the wire in that first connection - just don't pretend it's especially limiting

Maybe I explained myself wrong. I meant, once ssh with sudo is compromised, you're f***ed REPL or no REPL.

So if your company allows ssh with sudo, and they deem they have secured that to allow it, the REPL doesn't add to the threat vector

right - I don't think sudo / root access is a given (we can and should drop app privleges when running)

but you have at least the privs of the process running the jvm, if you can repl in that jvm

and if there are operating systems in production without local privilege escalations, they aren't used often

I thought most places gave dev ssh sudo access to prod hosts

So what I mean is, if you are already granted that permission, they trust you with a lot, the REPL doesn't let you do more things than ssh + sudo

So I don't see why they'd be against it

Now, if you only get ssh with some restricted user permissions, and those permissions are less then the user of the JVM, that's different

But as long as your ssh user has the same or more permissions as the user of the JVM, the REPL does not expose more things to you, its just a nicer UX

For example, your DB credentials are going to be stored in some file which the user of the JVM has permission to read, so I can easily ssh, read the file, get the creds, ssh tunnel my SQL Workbench and connect to your DB

If you have ssh access and no permissions, you can still tunnel to the server and connect to a socket REPL.

The socket connection on the loopback isn't restricted to just certain user accounts.

We tunnel in via a low-privilege user and the JVM runs under a separate user to which that tunneling account has pretty much no access, yet it can still connect to the REPL's port.

So a socket REPL is more access that just what ssh allows, in that respect.

Yes, when your ssh user has less permissions. I'm saying, if your InfoSec department lets you have SSH access with equal or more permissions than your JVM user, than the REPL isn't doing anything worse.

I don't know if that's the case for OP, but they should check. If they are already allowed to SSH with a user of similar or more permissions to the user running their app, then they shouldn't need to do anything more to "secure" the use of the REPL, since all data that can be accessed by the REPL, and all commands the REPL can execute on the machine can also be accessed and executed through other means.

Otherwise, and something I've done in the past is that our app does not run with a REPL open. Instead, you ssh into the host, and you start a second instance of your app with a REPL in it, that second app instance is thus launched with your ssh user, and is restricted to those permissions, then you can REPL into that.

We also do this to protect ourselves from accidentally reloading some buggy or broken code and causing prod issues