Clojurians Log v2

Morning

Good Morning

👋

Morning 🌅

Mornin' all

Morning everyone!

morning

morning folks

Morning

Good morning!

I came across <https://github.com/bguthrie/shrubbery> this morning, and didn’t realise that it would be possible to conduct side effectful testing (i.e., attempting to call a db, or a website API) using protocols to encapsulate the external call. Anyone else use shrubbery?

(or the approach of using protocols to wrap external calls)

Morning

mogge

i've find using protocols to encapsulate external interfaces very useful for testing and shrubbery looks like it would make life easier. shame it's not supporting cljs too though

interesting. I don't use cljs, so I must have a play and see if it will work for me 😉

(in my testing on the backend code that I maintain)

I also find protocols for external interfaces massively helpful when testing, and it forces more structure on the codebase too.

Morning!

I’ve used shrubbery briefly when I was playing with Duct. As Duct uses protocols for external dependencies by default.

But that was in ‘toy’ projects as none of the clients I’ve worked at use it

I keep forgetting to say hi to folk on here until it's almost the end of the work day for me. But I start a new job in Feb and back to doing Clojure, so time to get back into the community and extricate myself from all the Python Data Engineering 😄

re: protocols to encapsulate external calls, isn’t it what the good old https://en.wikipedia.org/wiki/Hexagonal_architecture_(software) advocated?

@carr0t where are you going to be working?

congrats on the new gig

Remote working Software Engineer/DevOps/SRE for Griffin, a bank startup with Clojure and Kafka core in their stack. I believe from what others have said before that they might have posted in here (or another channel in the workspace) as part of recruitment

Ta 😄

ahhh yes

had not heard of Hexagonal architecture – TIL! I’m surprised the idea is even that new (2005)

+1. It's always great news when somebody switches to a Clojure role 😁

arguably it’s an even older idea or practice with new-ish name. But I found this way of thinking useful from time to time, especially asking myself “Is this piece of code part of my Application or is it something that should be wrapped in one of those ports or adapters?“. I think it helps prevent your architecture being driven by frameworks or libs you’re using.

+1

Morning.

morning o/

How do you onboard new Clojure devs? Before, we have been putting them to running teams and asked colleagues to help them. But now, we got to a scale, where we are basically getting to the point, where we will have to run “Clojure kindergarten” (sorry for not having better name for that) as we’re going to onboard 10-15 new Clojure devs in following 60 days.

wow 10+ devs

even leaving out Clojure, onboarding that many devs is always a challenge

Yes, we have some 8 Clojure teams, going to 10 and making some of existing ones stronger. A lot of maps, filters & things to teach and hand over.

how many existing devs have you got?

1-1 buddying is your best bet if they're new to clj probably

In Clj, it is around 40 I think.

make sure everybody has somebody they're comfortable dropping a line to, to ask them questions for starters make sure the people doing the mentoring have the time to spend with the new people suggest that people put in an hour a week to catch up with the person they're paired with?

idk this stuff is hard hard

very 'you can lead a horse to water' kinda territory

esp as everybody learns very differently

Yes, this is what we did until now. Have a newcomer in the team and all extroverted devs taking care of them. But on this scale, we are thinking of really a period where there is a few weeks spent just by learning, not by being in any particular team.

Interesting question. You could perhaps draw some inspiration from other companies that have that problem, twitter and guardian run a “Scala school” for new joiners and I’ve heard of something similar for Ocaml at Jane Street. Link to twitter materials https://twitter.github.io/scala_school/index.html .

Thank you! This is exactly what I am looking for, just for Clojure.

But I agree with @alex.lynham, no syllabus will replace working with someone experienced. Maybe let them build some internal tools (or just toy projects), but on real infrastructure and under supervision ?

We plan to have experienced devs to be with them.

one of the north american clojure shops used to do a clojure bootcamp

iirc

they def had a blog post & open sourced the materials

Would they typically have functional experience from other languages?

@dev964 all of them have at least pet projects in Clojure, but from my experience Clj devs recruit from other functional languages, Ruby or JVM-land (and most often Java).

Nice to see clojure training being offered, I was just thrown in the deep end to float or sink.

similarish, i had a project an a deadline and then just had to crawl to rick for help when i hit a brick wall

although from memory i was learning emacs at the same time and often my issues were tooling (repl/versions/emacs config)

coming from ruby/js the JVM was a big overhead of stuff to know to get the project and repl building smoothly and keeping everything ticking along

and this was... 20...14? so some things had sharper edges from memory. maybe i just don't feel that pain in the same way cos i know the gotchas, idk

I get you. I was switching from Erlang and LiveScript without preexisting experience with Lisps, JVM, Vim user, somehow around 2012 I remember and there were lots of rough edges.

You'll get mixed answers about whether this is idiomatic clojure or not in my experience :)

I wound up having to teach someone who was hired before me the Clojure ropes at the last place I worked, and I was basically left to onboard myself, though I guess I have the opposite experience to most being more familiar with Lisp than with how the industry operates. I'd market myself as a trainer but I don't have the experience to make that stick 😅

Been trying to keep myself busy and train up that particular muscle by teaching non-professional interested friends and reflecting on the pedagogy.

i think that particularly with clojure there's a not insignificant element of it needing to fit with your mindset

it seems to click more naturally for some than others

like, if you show haskell or whatever the hangup is usually the complexity of category theory, not the essentials of FP, as let's face it, most decent JS devs are comfy with FP these days

there's something specific about lisp that is a hurdle possibly

but that Lisp self selection has already occurred if they all have clojure pet projects. They just need experience with bigger codebases and a particular tech stack, don’t they ?

and that’s the thing, we like to focus on programming languages, but often learning tools, libraries, runtime, infrastructure and operational processes at the company takes a lot more time imho ( I might be biased, after all I like learning new languages..). So making them build something real enough that it will go through all stages of dev process on a real codebase, ideally up to some silent prod (parallel)? release will probably go miles towards getting them to the stage where they’re productive

I feel like there's a certain minimalism coupled with a desire for aesthetic elegance that seems to be kernel of the "Lisp mindset". At least I've found that folks who really to take to Clojure seem to lean that way. The sort of Daoist approach to cognitive flow is also something I've found to be a hurdle in that the language is very much informed by it, but the subtlety is easy to miss if you're not looking for it. Can't appreciate the forest for the trees, as it were.

Interesting. Dominic, what’s the nature of the argument against using protocols for external calls? Choosing the right abstraction sometimes requires a bit of thought, and for a very small service that has maybe one dependency it might be overkill I guess.

The arguments are usually along the lines of: * by the time you've written all your mocks, what are you even testing? * why is your code under test not just a function that takes the stateful result and processes it (which doesn't require a protocol to test)

By code under test do you mean the application code that interfaces with the protocols, or the protocols themselves? If I understand correctly the former should be functions all the way down. I agree that testing anything the other side of the protocol is built on assumptions about how that dependency works, its failure cases etc.; that’s why I usually advocate for them being as thin as possible.e

I mean that instead of:

(defn foo [slack]
  (let [users (get-users slack)]
    (map inc users)))

(defn foo [slack-users]
  (map inc slack-users))

(defn some-composing-fn
  [slack]
  (foo (get-users slack)))

where composition isn't tested

@jiriknesl: I’m no expert; but I’ve taught clojure to a bunch of people over the past 12/13 years or so. Obviously do all the stuff you’re already doing and the stuff @alex.lynham suggested too. However if you have 10+ new folk who you need to bring up to speed quickly I’d suggest you also need to give the them the time to properly learn the basics. i.e. don’t just throw them in at the deep end and expect them to swim. So I’d suggest the easy option might be to find them a proper training course. iirc juxt used to provide clojure training courses in the UK, though they appear to be a bit more product focussed these days; but might still be worth asking. Even a little help in educating people early on will take off a lot of pressure allowing the experienced folk on the team to still ship stuff, and bring them up to speed on the aspects that are bespoke to your product/business. Also Alex says I threw him in at the deep end; and I certainly did that 🙊, but I’d suggest that’s only effective if the student is willing, and you’re willing to also teach/help on the side explain the fundamentals. It was a long time ago now, but I’m pretty sure I sat Alex down at a REPL a few dozen times to explain fundamental clojure… because I think the fundamentals of clojure are the interesting subtle bits that people overlook when learning it… they think “ok that’s how you do if ✅ ” ; rather than think about how if is actually a special-form and how special forms relate to macros, evaluation etc. Spending sufficient time on the basics won’t necessarily feel like the quickest way to get people to build a webapp; but it will pay off many times over, and help people understand why error messages are the way they are, for example. I’ve met many folk using clojure, who ignore useful clues in stacktraces for example, because they don’t appreciate the details of clojure’s construction and evaluation model etc. So that’s my main tip; is spend time teaching people the language itself and working at the repl. Foster a culture of being able to debug anything by finding a small expression that recreates the problem in the repl etc… Some of that work you can probably outsource; and if you don’t that’s fine; but I’d suggest make sure you teach that in parallel to the on the job stuff.

from memory we took a morning and made a simple compojure app in the repl and then you basically mapped the system-in-repl onto ruby/rails contexts cos that was easier to explain in terms of what the running system/program was & how to interact w/it and the repl flow :thinking_face: it was a while ago tho!

i def remember having to ask a lot of qs about the JVM and emacs. lein felt like a monster compared to the ruby/js runtimes

Ok that actually rings a bell now… I’m pretty sure whenever you hit a problem though, I’d probably have backed right up and explained some fundamentals… either to reduce the problem, or to explain what you were seeing and why.

i definitely remember something about how everything is a macro

cos i remember you hitting macroexpand on a load of forms and being a bit like 'wut, that's not a lang primitive' on lots of operations

ok that was probably some api we were using… maybe mount :face_vomiting: or compojure or the elasticsearch library :man-shrugging:

yeah must have been mount

cos moving to integrant was at the tail end of my swirrl days iirc

Appsflyer do the same thing for Clojure 👍

but the point stands that going over the fundamentals and the repl workflow is worth doing 😂

Either way, get-users could be a function or a protocol in this example. But this touches on another important point: what the order of invocation should be to best facilitate testing. For that the latter wins.

Yep. There's often no true scotsman arguments in this space, especially around integration tests and mocks. Personally, I flip flop on this. Right now I'm not as keen because I'm finding it's quite hard to mock out our ever-growing use of the slack API, and I wish I'd never started testing it that way.

i like our routing which is all data -> fn -> effect-data and is very easy to test without any mocks... but our push-notification service has logic around retry and other behaviour when errors are returned from the external service, and that seems to test nicely with mocks :man-shrugging:

I'm one of those who is against wrapping all the external stuff in protocols. It's looks fine in small examples where the protocol only has one or two "methods" but if you've got, say, 100 database tables and your code has to do a variety of CRUD-like operations on each one, now you have maybe 100 protocols, with 200-400 "methods" in total, with separate declarations and implementations, and the supposed benefit is to be able to reimplement those 200-400 functions with mock behavior so you can test code without needing the external systems around. That seems like a huge amount of boilerplate and duplication to me.

Stuart Sierra made an argument in favor of protocols and separate test/production implementations around his Component library and, again, the examples he showed were all simple things: a few protocols, each with only one or two functions, and it looks fine at that scale... but even though that was part of his "Clojure in the Large" talk about structure and (code/complexity) scale, he didn't show what a "large" system actually looked like with that approach and I've never seen that approach used in "large" codebases.

Our plan is that in the beginning they’ll work on some open source (as we are way behind in open source for an org of this size) on a stack that is similar to the stack they will use.

I think a better approach for testing code that ultimately does CRUD against a database is to have a separate test DB, preferably in memory if you can find a common SQL dialect between that and your production DB. We use MySQL (Percona) at work so we test against a local scratch database with SQL migrations to build it up from nothing (if needed). If we were using PostgreSQL, I'd probably have our test DB in memory via Embedded PostgreSQL (which I use when testing next.jdbc).

oh, i definitely agree with you there too @seancorfield - all that boilerplate would be awful

for our db CI we run up a fresh db server in a container... i never managed to make an in-memory cassandra instance work very well

I can relate somewhat to what Sean and Dominic have experienced. The codebase for one of our mature services has maybe 4 or 5 protocols each containing several methods and that can be a chore to mock. We have multiple protocols defined for the same store but grouped by entity (e.g. UserStore, AnswerStore), which goes some way in avoiding behemoth protocols. I have felt the pain of the boilerplate. It means the tests have less chance of being written, or taking longer to write, time which could be spent writing more tests.e

Yeah, we run docker-compose to spin up ElasticSearch, Percona, and Redis locally for testing. That way no one has to maintain any infrastructure and the Docker yaml file is part of the repo, along with scripts to fully populate the system from a cold start.

@lsnape I just checked and we have over 300 DB tables at this point... the number of protocols and methods for mocking that lot... 👀

One other thing I like about protocols is that they are distinct from plain-old pure functions. I really liked the caveats in DDIA that Kleppmann makes about RPCs, that they behave like a function in the happy path but can fail it all sorts of ways that a pure function direct call can’t. I think the same point applies to external dependencies.

Although I appreciate protocols are almost indistinguishable from functions at the call site 🙂 (you might be able to infer from the args)

The real solution here is a pure Clojure implementation of MySQL

Another potential downside to protocols: you can't instrument them so you need to write wrapper functions if you go down that path.

(and they can't be variadic, right? so you'd need wrapper functions for any operation that you wanted to be variadic, even if you didn't want to instrument them)