Clojurians Log v2

yes, for your solvathon … it’s an intriguing proposal

🙂 Well, I'll give you 2 very slightly related ideas/concepts:

1. zero downtime deployments: I know there are solutions, but I don't like most of them. For now, for example, the best way I found to have zero downtime during deployment, is starting your app 2 times - supposing you're having a (modular) monolith - listening each to another port. Restarting your nginx (or whatever you use), with a different config and when nginx is restarted, proxying the new version, kill the old one. (nginx keeps on running the old config until the new one is up and running). This is ok, but it's fiddling with ports and configs. I don't like it. There are other solutions as well. However, on linux system, there's something like SO_REUSEPORT/epoll and I know that netty somehow supports this (https://github.com/netty/netty/issues/2376). I just wonder if there a possibility to use this in your app, so that, if you start your app twice, the OS takes care of balance loading and redirecting calls to any of the apps listening to the same port! I tried it once, rather quickly, but it didn't work out fast enough for me, so I stopped trying 🙂. The question here would be: how to do zero time deployments with as less as config fiddling as possible, or something like that.

2. distributed modular apps: I call this slightly related, since, if you have a really modular app (as I describe below), you don't need apply a new api pretty often, and you can just deploy new versions of 'modules' wherever you like, whenever you like. Real modular apps, well, I know it's all possible, and there are many possibilities here (many of them involving message queues). It's about: how to deploy a new module, or a new version of an existing module without bringing the app down? In many cases, I have an app in which I want to (dynamically) add new behavior. Most of the time, now, I use core.async channels, having a tuple [:topic message]. Than I have a bunch of namespaces, each of them listening to a topic. However, I have to 'register' all the namespaces. Adding a new topic, leads to a change in the existing code, where I have to add a 'registration' of that namespaces (otherwise, it's never picked up). I actually would prefer not to have such a 'registration'. That's something vert.x is doing really well (vert.x is rather nice, especially version 3, but there's no clojure support in version 3). I consider a few levels here: - in-app, you can probably use core.async channels, but I'd like to get rid of the 'registration', and also, deploying a new version of 1 module, well, for now, it's mainly deploying a whole new app (there may be possibilities with mount and component here) - in the same JVM: how to deploy different modules in the same JVM and let them talk to each other (again, I like the way vert.x handles this: it is possible to unload and load new verticles, and they're all connected to the same eventbus). - between machines/JVM's: what if we want to deploy modules on different machines? How to do autodiscovery? (vert.x uses Hazelcast, and it seems to be a pluggable system, so as far as I remember, they were busy with zookeeper a few months ago) How to have a kind of 'pluggable core.async channels', turning it into a real distributed message bus (or just wrap vert.x around it, for example)? How to limit the memory overhead of the JVM? What's the minimal Java-image (using JDK9 Jigsaw) you need to run clojure? ...? vert.x has some solutions to this - the vertx event bus with the autodiscovery is a very powerful concept - and also the guys from paralleluniverse have some solutions here. This doesn't mean, however, that clojure couldn't have an own solution ('distributed core.async': just send and don't care about the rest, possibly adding a reply channel? Or register all components somewhere in memory to dispatch to the right machine?). It might also just be fun to find out about performance, resilience, ... . This information is usually very scattered.

- or even, why not, use datomic to store the functions and distribute them in some way (although Datomic doesn't have a really nice/cheap license - free version includes only 1 year updates and support).

- what else is: why not having a system that can pull immediately from a repo (maven, or even just a source code repo) when there's a new version? For example: a new version of a module is pushed to the repo, the repo pushes a message to the app, the app downloads the repo, compiles if necessary, installs it, and removes the old version of the module (if necessary).

interesting topics

we are only aiming for the 'different machines' approach and are using uberjar + docker java image as the resulting artifact of a build

and using kubernetes to do the service discovery (it uses etcd behind the scenes) and scheduling of containers

that way you're not only limited to JVM solutions but can use any technology stack

zero downtime deployments are a matter of properly configuring rolling updates of new versions of the containers

I'd like to add: complex build pipelines. when your dependencies change, how will you make sure your entire deployment makes use of the right dependencies?

@kurt-yagram I think these are quite difficult challenges … setting out constraints and seeing what groups of people come back with could be quite interesting. Not sure if it’s great for a meet-up though to be honest

having a back and forth and sharing solutions could be good

but getting people to solve these complex problems is a big ask

or organize some small discussion groups?

we can also do something like this https://www.factual.com/blog/clojure-office-hours once in a while

yes - like it - that seems like an interesting approach where people bring code to discuss

yeah, like it as well. I like people actually coding. It's easy to find discussions and blogs, but it's hard to find the actual real stuff.

@stijn: > zero downtime deployments are a matter of properly configuring rolling updates of new versions of the containers Right, things are always 'just a matter of', however, prefer minimizing 'just a matter of'. If one can make SO_REUSEPORT work, well, it's not 'just a matter of' anymore, since part disappears 🙂. Same with docker, kubernetes, and stuff: I have done it (not extensively, though), and it certainly brings value, but not for smaller projects (spanning over 3 or 4 machines). You just have extra stuff to manage (the docker containers). We're already discussing 😛. > complex build pipelines. when your dependencies change, how will you make sure your entire deployment makes use of the right dependencies? Well, if you have small containers (in the sense of deployables), you actually avoid this complex stuff: each one can have it's own small set of dependencies, right? That's why a very modular design appeals to me a lot. Using vert.x in the past, I had verticles of less than 100 lines of code. The thing is: I could use an existing JVM to start them, so I could avoid the memory overhead for such a small module. I'd really differ between different levels of modularity, each needing a different kind of tech (higher level will probably lead to slower but more generic solutions).

A topic that I would be very much interested in is how to implement CQRS and event-sourcing in Clojure. Not exactly the why but more the how in Clojure. I do have some ideas about that, but I would love to hear about a real-world implementation. Any thoughts about this topic?

You have a use case? It might be fun to discuss that one, I think.

(Well, using Datomic leads rather automatically to something 'event sourcing'-like)

Well, there are a lot of examples in “classic” OO languages like Java and C#. But in the functional realm I only know of a couple decent examples in F# and this one in Clojure: https://github.com/rill-event-sourcing/rill

I was just wondering whether someone in the Belgian Clojure community has any experiences to share regarding ES and CQRS in particular