clojure-uk

A place for people in the UK, near the UK, visiting the UK, planning to visit the UK or just vaguely interested to randomly chat about things (often vi and emacs, occasionally clojure). More general the #ldnclj
jiriknesl 2020-11-23T07:33:16.035600Z

Morning

dharrigan 2020-11-23T08:02:34.035800Z

Good Morning

djm 2020-11-23T08:08:00.036Z

šŸ‘‹

codeasone 2020-11-23T08:21:03.036200Z

Morning šŸŒ…

Russ Anderson 2020-11-23T08:36:13.036400Z

Mornin' all

2020-11-23T08:39:58.036600Z

Morning everyone!

alexlynham 2020-11-23T08:48:26.036700Z

morning

Alex J Henderson 2020-11-23T09:18:26.036900Z

morning folks

gcaban 2020-11-23T09:42:19.037100Z

Morning

2020-11-23T09:44:52.037300Z

Good morning!

dharrigan 2020-11-23T09:58:28.039300Z

I came across <https://github.com/bguthrie/shrubbery> this morning, and didnā€™t realise that it would be possible to conduct side effectful testing (i.e., attempting to call a db, or a website API) using protocols to encapsulate the external call. Anyone else use shrubbery?

dharrigan 2020-11-23T09:58:41.039600Z

(or the approach of using protocols to wrap external calls)

2020-11-23T10:10:08.039800Z

Morning

thomas 2020-11-23T10:15:37.040Z

mogge

mccraigmccraig 2020-11-23T10:28:35.040100Z

i've find using protocols to encapsulate external interfaces very useful for testing and shrubbery looks like it would make life easier. shame it's not supporting cljs too though

dharrigan 2020-11-23T10:33:11.040300Z

interesting. I don't use cljs, so I must have a play and see if it will work for me šŸ˜‰

dharrigan 2020-11-23T10:33:25.040500Z

(in my testing on the backend code that I maintain)

2020-11-23T10:48:12.040700Z

I also find protocols for external interfaces massively helpful when testing, and it forces more structure on the codebase too.

danm 2020-11-23T10:51:17.041Z

Morning!

agile_geek 2020-11-23T10:51:29.041100Z

Iā€™ve used shrubbery briefly when I was playing with Duct. As Duct uses protocols for external dependencies by default.

agile_geek 2020-11-23T10:51:55.041700Z

But that was in ā€˜toyā€™ projects as none of the clients Iā€™ve worked at use it

danm 2020-11-23T10:52:31.042600Z

I keep forgetting to say hi to folk on here until it's almost the end of the work day for me. But I start a new job in Feb and back to doing Clojure, so time to get back into the community and extricate myself from all the Python Data Engineering šŸ˜„

1
šŸ¦œ 1
šŸŽ‰ 1
gcaban 2020-11-23T11:08:09.042700Z

re: protocols to encapsulate external calls, isnā€™t it what the good old https://en.wikipedia.org/wiki/Hexagonal_architecture_(software) advocated?

alexlynham 2020-11-23T11:25:07.043Z

@carr0t where are you going to be working?

alexlynham 2020-11-23T11:26:00.043100Z

congrats on the new gig

āž• 1
danm 2020-11-23T11:26:51.043200Z

Remote working Software Engineer/DevOps/SRE for Griffin, a bank startup with Clojure and Kafka core in their stack. I believe from what others have said before that they might have posted in here (or another channel in the workspace) as part of recruitment

danm 2020-11-23T11:26:58.043400Z

Ta šŸ˜„

alexlynham 2020-11-23T11:29:49.043600Z

ahhh yes

2020-11-23T11:36:13.043800Z

had not heard of Hexagonal architecture ā€“ TIL! Iā€™m surprised the idea is even that new (2005)

Mario Giampietri 2020-11-23T11:55:03.044400Z

+1. It's always great news when somebody switches to a Clojure role šŸ˜

gcaban 2020-11-23T12:16:32.044600Z

arguably itā€™s an even older idea or practice with new-ish name. But I found this way of thinking useful from time to time, especially asking myself ā€œIs this piece of code part of my Application or is it something that should be wrapped in one of those ports or adapters?ā€œ. I think it helps prevent your architecture being driven by frameworks or libs youā€™re using.

šŸ’Æ 3
gcaban 2020-11-23T12:27:24.044900Z

+1

2020-11-23T12:56:56.045600Z

Morning.

yogidevbear 2020-11-23T13:00:22.045900Z

morning o/

jiriknesl 2020-11-23T13:42:39.049Z

How do you onboard new Clojure devs? Before, we have been putting them to running teams and asked colleagues to help them. But now, we got to a scale, where we are basically getting to the point, where we will have to run ā€œClojure kindergartenā€ (sorry for not having better name for that) as weā€™re going to onboard 10-15 new Clojure devs in following 60 days.

dharrigan 2020-11-23T13:44:26.049300Z

wow 10+ devs

dharrigan 2020-11-23T13:44:43.049800Z

even leaving out Clojure, onboarding that many devs is always a challenge

jiriknesl 2020-11-23T13:56:54.050900Z

Yes, we have some 8 Clojure teams, going to 10 and making some of existing ones stronger. A lot of maps, filters & things to teach and hand over.

alexlynham 2020-11-23T13:59:01.052100Z

how many existing devs have you got?

alexlynham 2020-11-23T13:59:17.052200Z

1-1 buddying is your best bet if they're new to clj probably

jiriknesl 2020-11-23T13:59:35.052600Z

In Clj, it is around 40 I think.

alexlynham 2020-11-23T14:00:57.052800Z

make sure everybody has somebody they're comfortable dropping a line to, to ask them questions for starters make sure the people doing the mentoring have the time to spend with the new people suggest that people put in an hour a week to catch up with the person they're paired with?

alexlynham 2020-11-23T14:01:01.052900Z

idk this stuff is hard hard

alexlynham 2020-11-23T14:01:14.053Z

very 'you can lead a horse to water' kinda territory

alexlynham 2020-11-23T14:01:26.053300Z

esp as everybody learns very differently

jiriknesl 2020-11-23T14:02:16.054300Z

Yes, this is what we did until now. Have a newcomer in the team and all extroverted devs taking care of them. But on this scale, we are thinking of really a period where there is a few weeks spent just by learning, not by being in any particular team.

gcaban 2020-11-23T14:08:13.058100Z

Interesting question. You could perhaps draw some inspiration from other companies that have that problem, twitter and guardian run a ā€œScala schoolā€ for new joiners and Iā€™ve heard of something similar for Ocaml at Jane Street. Link to twitter materials https://twitter.github.io/scala_school/index.html .

jiriknesl 2020-11-23T14:09:01.059Z

Thank you! This is exactly what I am looking for, just for Clojure.

gcaban 2020-11-23T14:09:01.059100Z

But I agree with @alex.lynham, no syllabus will replace working with someone experienced. Maybe let them build some internal tools (or just toy projects), but on real infrastructure and under supervision ?

jiriknesl 2020-11-23T14:09:18.059500Z

We plan to have experienced devs to be with them.

alexlynham 2020-11-23T14:10:11.059600Z

one of the north american clojure shops used to do a clojure bootcamp

alexlynham 2020-11-23T14:10:14.059700Z

iirc

alexlynham 2020-11-23T14:10:24.059800Z

they def had a blog post & open sourced the materials

2020-11-23T14:11:08.060300Z

Would they typically have functional experience from other languages?

jiriknesl 2020-11-23T14:14:21.061800Z

@dev964 all of them have at least pet projects in Clojure, but from my experience Clj devs recruit from other functional languages, Ruby or JVM-land (and most often Java).

2020-11-23T14:14:48.062300Z

Nice to see clojure training being offered, I was just thrown in the deep end to float or sink.

alexlynham 2020-11-23T14:22:07.064Z

similarish, i had a project an a deadline and then just had to crawl to rick for help when i hit a brick wall

1
alexlynham 2020-11-23T14:22:35.064100Z

although from memory i was learning emacs at the same time and often my issues were tooling (repl/versions/emacs config)

alexlynham 2020-11-23T14:23:03.064200Z

coming from ruby/js the JVM was a big overhead of stuff to know to get the project and repl building smoothly and keeping everything ticking along

alexlynham 2020-11-23T14:23:41.064300Z

and this was... 20...14? so some things had sharper edges from memory. maybe i just don't feel that pain in the same way cos i know the gotchas, idk

jiriknesl 2020-11-23T14:30:15.065500Z

I get you. I was switching from Erlang and LiveScript without preexisting experience with Lisps, JVM, Vim user, somehow around 2012 I remember and there were lots of rough edges.

dominicm 2020-11-23T14:49:19.067Z

You'll get mixed answers about whether this is idiomatic clojure or not in my experience :)

Edward Hughes 2020-11-23T15:04:36.073700Z

I wound up having to teach someone who was hired before me the Clojure ropes at the last place I worked, and I was basically left to onboard myself, though I guess I have the opposite experience to most being more familiar with Lisp than with how the industry operates. I'd market myself as a trainer but I don't have the experience to make that stick šŸ˜…

Edward Hughes 2020-11-23T15:11:19.075500Z

Been trying to keep myself busy and train up that particular muscle by teaching non-professional interested friends and reflecting on the pedagogy.

alexlynham 2020-11-23T15:15:09.075700Z

i think that particularly with clojure there's a not insignificant element of it needing to fit with your mindset

alexlynham 2020-11-23T15:15:20.075800Z

it seems to click more naturally for some than others

alexlynham 2020-11-23T15:16:23.075900Z

like, if you show haskell or whatever the hangup is usually the complexity of category theory, not the essentials of FP, as let's face it, most decent JS devs are comfy with FP these days

alexlynham 2020-11-23T15:16:36.076Z

there's something specific about lisp that is a hurdle possibly

gcaban 2020-11-23T15:20:34.078100Z

but that Lisp self selection has already occurred if they all have clojure pet projects. They just need experience with bigger codebases and a particular tech stack, donā€™t they ?

gcaban 2020-11-23T15:21:36.078800Z

and thatā€™s the thing, we like to focus on programming languages, but often learning tools, libraries, runtime, infrastructure and operational processes at the company takes a lot more time imho ( I might be biased, after all I like learning new languages..). So making them build something real enough that it will go through all stages of dev process on a real codebase, ideally up to some silent prod (parallel)? release will probably go miles towards getting them to the stage where theyā€™re productive

Edward Hughes 2020-11-23T15:41:58.079Z

I feel like there's a certain minimalism coupled with a desire for aesthetic elegance that seems to be kernel of the "Lisp mindset". At least I've found that folks who really to take to Clojure seem to lean that way. The sort of Daoist approach to cognitive flow is also something I've found to be a hurdle in that the language is very much informed by it, but the subtlety is easy to miss if you're not looking for it. Can't appreciate the forest for the trees, as it were.

2020-11-23T16:00:49.079200Z

Interesting. Dominic, whatā€™s the nature of the argument against using protocols for external calls? Choosing the right abstraction sometimes requires a bit of thought, and for a very small service that has maybe one dependency it might be overkill I guess.

dominicm 2020-11-23T16:08:58.079500Z

The arguments are usually along the lines of: * by the time you've written all your mocks, what are you even testing? * why is your code under test not just a function that takes the stateful result and processes it (which doesn't require a protocol to test)

2020-11-23T16:18:08.084500Z

By code under test do you mean the application code that interfaces with the protocols, or the protocols themselves? If I understand correctly the former should be functions all the way down. I agree that testing anything the other side of the protocol is built on assumptions about how that dependency works, its failure cases etc.; thatā€™s why I usually advocate for them being as thin as possible.e

dominicm 2020-11-23T16:19:45.085900Z

I mean that instead of:

(defn foo [slack]
  (let [users (get-users slack)]
    (map inc users)))
Why is it not written
(defn foo [slack-users]
  (map inc slack-users))

(defn some-composing-fn
  [slack]
  (foo (get-users slack)))

dominicm 2020-11-23T16:19:52.086100Z

where composition isn't tested

šŸ‘Œ 1
2020-11-23T16:31:13.096700Z

@jiriknesl: Iā€™m no expert; but Iā€™ve taught clojure to a bunch of people over the past 12/13 years or so. Obviously do all the stuff youā€™re already doing and the stuff @alex.lynham suggested too. However if you have 10+ new folk who you need to bring up to speed quickly Iā€™d suggest you also need to give the them the time to properly learn the basics. i.e. donā€™t just throw them in at the deep end and expect them to swim. So Iā€™d suggest the easy option might be to find them a proper training course. iirc juxt used to provide clojure training courses in the UK, though they appear to be a bit more product focussed these days; but might still be worth asking. Even a little help in educating people early on will take off a lot of pressure allowing the experienced folk on the team to still ship stuff, and bring them up to speed on the aspects that are bespoke to your product/business. Also Alex says I threw him in at the deep end; and I certainly did that šŸ™Š, but Iā€™d suggest thatā€™s only effective if the student is willing, and youā€™re willing to also teach/help on the side explain the fundamentals. It was a long time ago now, but Iā€™m pretty sure I sat Alex down at a REPL a few dozen times to explain fundamental clojureā€¦ because I think the fundamentals of clojure are the interesting subtle bits that people overlook when learning itā€¦ they think ā€œok thatā€™s how you do if āœ… ā€ ; rather than think about how if is actually a special-form and how special forms relate to macros, evaluation etc. Spending sufficient time on the basics wonā€™t necessarily feel like the quickest way to get people to build a webapp; but it will pay off many times over, and help people understand why error messages are the way they are, for example. Iā€™ve met many folk using clojure, who ignore useful clues in stacktraces for example, because they donā€™t appreciate the details of clojureā€™s construction and evaluation model etc. So thatā€™s my main tip; is spend time teaching people the language itself and working at the repl. Foster a culture of being able to debug anything by finding a small expression that recreates the problem in the repl etcā€¦ Some of that work you can probably outsource; and if you donā€™t thatā€™s fine; but Iā€™d suggest make sure you teach that in parallel to the on the job stuff.

šŸ‘ 2
alexlynham 2020-11-23T16:34:31.097700Z

from memory we took a morning and made a simple compojure app in the repl and then you basically mapped the system-in-repl onto ruby/rails contexts cos that was easier to explain in terms of what the running system/program was & how to interact w/it and the repl flow :thinking_face: it was a while ago tho!

alexlynham 2020-11-23T16:35:18.098100Z

i def remember having to ask a lot of qs about the JVM and emacs. lein felt like a monster compared to the ruby/js runtimes

2020-11-23T16:37:30.099800Z

Ok that actually rings a bell nowā€¦ Iā€™m pretty sure whenever you hit a problem though, Iā€™d probably have backed right up and explained some fundamentalsā€¦ either to reduce the problem, or to explain what you were seeing and why.

alexlynham 2020-11-23T16:38:09.100500Z

i definitely remember something about how everything is a macro

alexlynham 2020-11-23T16:38:40.100700Z

cos i remember you hitting macroexpand on a load of forms and being a bit like 'wut, that's not a lang primitive' on lots of operations

2020-11-23T16:40:41.102Z

ok that was probably some api we were usingā€¦ maybe mount :face_vomiting: or compojure or the elasticsearch library :man-shrugging:

alexlynham 2020-11-23T16:41:39.102100Z

yeah must have been mount

alexlynham 2020-11-23T16:42:14.102200Z

cos moving to integrant was at the tail end of my swirrl days iirc

yogidevbear 2020-11-23T16:43:45.102300Z

Appsflyer do the same thing for Clojure šŸ‘

alexlynham 2020-11-23T16:44:10.102500Z

but the point stands that going over the fundamentals and the repl workflow is worth doing šŸ˜‚

šŸ‘ 1
2020-11-23T17:05:49.107900Z

Either way, get-users could be a function or a protocol in this example. But this touches on another important point: what the order of invocation should be to best facilitate testing. For that the latter wins.

dominicm 2020-11-23T17:23:43.108100Z

Yep. There's often no true scotsman arguments in this space, especially around integration tests and mocks. Personally, I flip flop on this. Right now I'm not as keen because I'm finding it's quite hard to mock out our ever-growing use of the slack API, and I wish I'd never started testing it that way.

mccraigmccraig 2020-11-23T17:30:27.108300Z

i like our routing which is all data -&gt; fn -&gt; effect-data and is very easy to test without any mocks... but our push-notification service has logic around retry and other behaviour when errors are returned from the external service, and that seems to test nicely with mocks :man-shrugging:

seancorfield 2020-11-23T17:39:23.108500Z

I'm one of those who is against wrapping all the external stuff in protocols. It's looks fine in small examples where the protocol only has one or two "methods" but if you've got, say, 100 database tables and your code has to do a variety of CRUD-like operations on each one, now you have maybe 100 protocols, with 200-400 "methods" in total, with separate declarations and implementations, and the supposed benefit is to be able to reimplement those 200-400 functions with mock behavior so you can test code without needing the external systems around. That seems like a huge amount of boilerplate and duplication to me.

seancorfield 2020-11-23T17:42:31.109200Z

Stuart Sierra made an argument in favor of protocols and separate test/production implementations around his Component library and, again, the examples he showed were all simple things: a few protocols, each with only one or two functions, and it looks fine at that scale... but even though that was part of his "Clojure in the Large" talk about structure and (code/complexity) scale, he didn't show what a "large" system actually looked like with that approach and I've never seen that approach used in "large" codebases.

jiriknesl 2020-11-23T17:43:09.109700Z

Our plan is that in the beginning theyā€™ll work on some open source (as we are way behind in open source for an org of this size) on a stack that is similar to the stack they will use.

seancorfield 2020-11-23T17:45:21.109800Z

I think a better approach for testing code that ultimately does CRUD against a database is to have a separate test DB, preferably in memory if you can find a common SQL dialect between that and your production DB. We use MySQL (Percona) at work so we test against a local scratch database with SQL migrations to build it up from nothing (if needed). If we were using PostgreSQL, I'd probably have our test DB in memory via Embedded PostgreSQL (which I use when testing next.jdbc).

mccraigmccraig 2020-11-23T17:45:31.110Z

oh, i definitely agree with you there too @seancorfield - all that boilerplate would be awful

mccraigmccraig 2020-11-23T17:47:27.110300Z

for our db CI we run up a fresh db server in a container... i never managed to make an in-memory cassandra instance work very well

2020-11-23T17:50:02.110500Z

I can relate somewhat to what Sean and Dominic have experienced. The codebase for one of our mature services has maybe 4 or 5 protocols each containing several methods and that can be a chore to mock. We have multiple protocols defined for the same store but grouped by entity (e.g. UserStore, AnswerStore), which goes some way in avoiding behemoth protocols. I have felt the pain of the boilerplate. It means the tests have less chance of being written, or taking longer to write, time which could be spent writing more tests.e

seancorfield 2020-11-23T17:50:32.110800Z

Yeah, we run docker-compose to spin up ElasticSearch, Percona, and Redis locally for testing. That way no one has to maintain any infrastructure and the Docker yaml file is part of the repo, along with scripts to fully populate the system from a cold start.

seancorfield 2020-11-23T17:53:36.111100Z

@lsnape I just checked and we have over 300 DB tables at this point... the number of protocols and methods for mocking that lot... šŸ‘€

šŸ˜± 1
2020-11-23T17:59:45.111400Z

One other thing I like about protocols is that they are distinct from plain-old pure functions. I really liked the caveats in DDIA that Kleppmann makes about RPCs, that they behave like a function in the happy path but can fail it all sorts of ways that a pure function direct call canā€™t. I think the same point applies to external dependencies.

2020-11-23T18:00:37.111600Z

Although I appreciate protocols are almost indistinguishable from functions at the call site šŸ™‚ (you might be able to infer from the args)

dominicm 2020-11-23T18:01:27.111900Z

The real solution here is a pure Clojure implementation of MySQL

seancorfield 2020-11-23T18:04:42.112100Z

Another potential downside to protocols: you can't instrument them so you need to write wrapper functions if you go down that path.

seancorfield 2020-11-23T18:06:13.112300Z

(and they can't be variadic, right? so you'd need wrapper functions for any operation that you wanted to be variadic, even if you didn't want to instrument them)