clojure

New to Clojure? Try the #beginners channel. Official docs: https://clojure.org/ Searchable message archives: https://clojurians-log.clojureverse.org/
zhuxun2 2020-11-17T00:48:41.130900Z

After I made a clojure.spec.alpha/fdef, there doesn't have to be any run-time validity checks. Do I have to put the same specs for args and ret again into :pre and :post?

zhuxun2 2020-11-17T00:49:06.131500Z

Is there a way to use the fdef directly as a run-time validity check?

2020-11-17T00:51:04.131600Z

You need to call instrument from clojure.spec.test.alpha

2020-11-17T00:51:35.131800Z

(require '[clojure.spec.test.alpha :as stest])
(stest/instrument)

šŸ‘ 1
zhuxun2 2020-11-17T00:51:49.132Z

Got it. Thanks.

2020-11-17T00:51:58.132300Z

That will instrument all functions that have an fdef so when called, they auto-validate.

2020-11-17T00:53:38.132500Z

Its not recommended to have them instrumented in prod (because it slows down performance to have it everywhere). So in prod, it is better to selectively use calls to s/valid? in your code (could be in a :pre or just inside your function as you please). Where you'd only validate user/client/DB input and data that you output to a user or DB or some other system like a client.

šŸ‘ 1
2020-11-17T00:54:03.132700Z

But at the REPL and in tests, do call instrument.

2020-11-17T00:55:54.133Z

Also, small caveat, this won't instrument return values, just input args. Clojure assumes you will assert valid output for a given valid input using generative testing. But because that can be inconvenient to setup for all functions sometimes, there's a lib that has a drop in replacement for instrument that will instrument return as well: https://github.com/jeaye/orchestra

šŸ‘ 2
zhuxun2 2020-11-17T01:00:44.133500Z

Thanks for the tips. These are very helpful! @didibus

dgb23 2020-11-17T01:04:05.133800Z

I also use orchestra for repl sessions. It is very simple and convenient.

alexmiller 2020-11-17T01:55:00.135Z

due to the perf costs, no. fdef specs are for test-time checking.

defa 2020-11-17T14:13:27.155900Z

Is there a shorter form for testing if v is a list/vector or LazySeq?

(or (vector? v) (list? v) (instance? clojure.lang.LazySeq v))

alexmiller 2020-11-17T14:13:59.156400Z

sequential?

defa 2020-11-17T14:15:18.156700Z

šŸ™ˆ of course... thank you! I got confused with seq? which is of course something completely different.

2020-11-17T14:18:52.157900Z

hey folks, is anyone aware of if it is possible to dynamically add routes to a running server without restarting the application?

2020-11-17T14:19:26.158900Z

what I'm trying to achieve is the following

2020-11-17T14:20:24.160100Z

create a minimal service running one post endpoint that accepts a enpoint path and a json file to be returned to dynamically create fake endpoints

2020-11-17T14:21:44.161600Z

thought of using mount to restart api component after endpoint recieved post with valid data, but maybe there is a more elegant way

2020-11-17T14:26:32.165400Z

or you could have two endpoints: one POST as you described, another (with lower precedence) to catch all other requests. Something like /:path with stateful handler which will hold an atom (as an example) with all routes posted via first POST. This is relatively easy to achieve with reitit. Iā€™m saying that with high confidence because I have a project that does that šŸ˜‰

borkdude 2020-11-17T14:28:35.166200Z

@dev-hartmann This is pretty simple using no routing library at all and just ring. Just look at the :uri and :method keys of the request and act accordingly.

2020-11-17T14:29:42.166900Z

@delaguardo thought about that too, good to know someone else already validated that šŸ˜„

2020-11-17T14:30:27.167800Z

@borkdude will look into that, ultimately that would remove two dependencies, which for a minimal dev tool would be awesome!

2020-11-17T14:30:36.168100Z

thanks for the replies folks

borkdude 2020-11-17T14:30:43.168400Z

@dev-hartmann you can even do this with babashka only

2020-11-17T14:31:00.168900Z

that's what I was ultimately eyeing at

borkdude 2020-11-17T14:31:01.169Z

just one small script

2020-11-17T14:31:15.169200Z

thx

Clojavu 2020-11-17T16:56:37.170700Z

Hi, I am migrating my application from lein to clojure cli. I have two questions 1. Would deps.edn support profiles? I have a few in my project.clj. Is there a reference for all the available options for deps.edn? 2. How can I call clojure functions from my deps.edn file. I used to do this with a ~ in my project.clj Thanks

borkdude 2020-11-17T16:58:01.171200Z

@dejavuakku 1) deps.edn aliases would be the closest equivalent to lein profiles. 2) you can't.

Clojavu 2020-11-17T16:58:50.171600Z

Okay. Thanks @borkdude

šŸ‘ 1
hipster coder 2020-11-17T18:12:15.172600Z

can someone tell me if they have tested Clojure and any concurrency models against Golang CSP, Communicating Sequential Processes

hipster coder 2020-11-17T18:12:42.173300Z

I am going into a consultationā€¦ and need to debate if we use another concurrency modelā€¦ and use the JVM with possibly Clojure

hipster coder 2020-11-17T18:12:55.173700Z

The CSP model is not fault tolerant. for 99% uptime, for web dev

hipster coder 2020-11-17T18:13:13.174100Z

when channels fail, the receivers fail, and thereā€™s no que of backlogged messages

hipster coder 2020-11-17T18:13:53.174500Z

Netflix hit this problem with Goā€¦ and had to use Hystrix

2020-11-17T18:16:26.175700Z

@nathantech2005 sounds like what you want is something that spans multiple processes / services, where CSP is usually done inside the scope of one process (we have core.async which is also CSP)

hipster coder 2020-11-17T18:17:02.176300Z

Yes. And I favor the actor model, for web devā€¦ itā€™s like Elixirā€¦ Very fault tolerant.

hipster coder 2020-11-17T18:17:20.176800Z

CSP only uses 1 system process?

2020-11-17T18:17:40.177500Z

clojure doesn't offer any inter-process / inter-service model out of the box, that's outside the language design scope, but various libraries can be used

hipster coder 2020-11-17T18:18:06.178300Z

Iā€™d be willing to use a library for Clojure for actor model

hipster coder 2020-11-17T18:18:13.178700Z

or mix Clojure with other JVM langs

2020-11-17T18:18:18.179Z

@smith.adriane right, CSP isn't fault tolerant, it's about guaranteeing that certain concurrency errors within one process won't occur

2020-11-17T18:18:24.179200Z

it's a different problem

hipster coder 2020-11-17T18:18:38.179600Z

@smith.adriane Elixir is based on Actor Modelā€¦ almost as similar to CSP

2020-11-17T18:18:54.180100Z

actor model and CSP are very different, they address completely different problems

hipster coder 2020-11-17T18:18:59.180400Z

so CSP is better at controlling 1 process

2020-11-17T18:19:01.180600Z

and the models are totally different

hipster coder 2020-11-17T18:19:15.180900Z

being able to monitor the 1 system process, for problem

hipster coder 2020-11-17T18:19:31.181400Z

opposed to sending off messages to actors on other machinesā€¦ and not knowing what happened

hipster coder 2020-11-17T18:20:07.182300Z

I canā€™t think of a domain problemā€¦ CSP is great forā€¦ off the top of my head? any ideas?

phronmophobic 2020-11-17T18:20:40.183200Z

I thought erlang was "communicating sequential processes?"

hipster coder 2020-11-17T18:20:44.183500Z

maybe like in a wharehouseā€¦ controlling the flows of goods, machnes

2020-11-17T18:20:56.183900Z

CSP solves async coordination problems (deadlock, livelock)

hipster coder 2020-11-17T18:20:58.184100Z

Golang can monitor 1 processā€¦ over many stepsā€¦ in a wharehouse

2020-11-17T18:21:02.184200Z

@smith.adriane no, erlang is async

2020-11-17T18:21:08.184400Z

the CSP model is sync

hipster coder 2020-11-17T18:21:18.184700Z

ahhhh

2020-11-17T18:21:19.184800Z

(in a specific way...)

hipster coder 2020-11-17T18:21:33.185200Z

CSP is syncā€¦ thatā€™s news to me tooā€¦ did not realize that

2020-11-17T18:21:45.185500Z

"sequential" processes

hipster coder 2020-11-17T18:21:50.185800Z

in order

hipster coder 2020-11-17T18:21:56.186100Z

got it

hipster coder 2020-11-17T18:22:04.186400Z

so you can control the order, steps

2020-11-17T18:22:13.186700Z

it has an official order that the events were processed, unlike erlang which can have different orders for different observers

hipster coder 2020-11-17T18:22:29.187100Z

ok, thanks, yes, that is totally different problem sets

2020-11-17T18:22:58.187500Z

the CSP paper isn't super difficult and it clears up a lot: https://www.cs.cmu.edu/~crary/819-f09/Hoare78.pdf

hipster coder 2020-11-17T18:23:12.187900Z

yep, I printed thatā€¦ was reading that one

2020-11-17T18:23:18.188100Z

oops, I said "concurrent" above, and it's "communicating"

hipster coder 2020-11-17T18:23:40.188500Z

hahahaā€¦ you clarified alot

hipster coder 2020-11-17T18:23:49.188800Z

when you explained, ordered vs non ordered

hipster coder 2020-11-17T18:24:01.189200Z

because I was under assumption, CSP was nearly like Actors

phronmophobic 2020-11-17T18:24:21.189700Z

> A consequence of ignoring time is that we refuse to answer or even to askwhether one event occurs simultaneously with another. When simultaneity of apair of events is important (e.g. in synchronisation) we represent it as a single-event occurrence; and when it is not, we allow two potentially simultaneousevent occurrences to be recorded in either order.

hipster coder 2020-11-17T18:24:27.189900Z

Actors work great for Whatsapp. chatā€¦

hipster coder 2020-11-17T18:24:34.190200Z

Go works great for robotics, wharehouse control

2020-11-17T18:24:58.190300Z

what's the source ther?

phronmophobic 2020-11-17T18:25:09.190500Z

http://usingcsp.com/cspbook.pdf

phronmophobic 2020-11-17T18:25:14.190700Z

chapter 1

2020-11-17T18:25:42.190900Z

perhaps I'm misunderstanding what "sequential" means then...

hipster coder 2020-11-17T18:25:46.191100Z

are you sayingā€¦ you think CSP ignores order?

hipster coder 2020-11-17T18:25:59.191300Z

because everyone I talked to, using GO, told me, they can control the order

phronmophobic 2020-11-17T18:27:20.191500Z

sequential refers to the "events" within a single process.

phronmophobic 2020-11-17T18:27:55.191700Z

if you're focusing on fault tolerance, I highly recommend the erlang paper

phronmophobic 2020-11-17T18:28:26.191900Z

one of the major ideas in the erlang paper is that you can't have true fault tolerance on a single machine since machines can fail

hipster coder 2020-11-17T18:28:30.192100Z

yesā€¦ because itā€™s 2 different domains 1. IoT devices, that can crash 2. 99% web uptime

hipster coder 2020-11-17T18:28:56.192300Z

yes, I like the erlang model, how it can mirror processes onto different machines

phronmophobic 2020-11-17T18:29:01.192500Z

if you have multiple machines, than you can't really gaurantee the latency of message between them

hipster coder 2020-11-17T18:29:03.192700Z

so if one crashes, another machine just steps in

hipster coder 2020-11-17T18:29:15.192900Z

ya, so you canā€™t control the order

hipster coder 2020-11-17T18:29:22.193100Z

which is ok, for sayā€¦ a chat system

hipster coder 2020-11-17T18:29:26.193300Z

telecom

hipster coder 2020-11-17T18:29:48.193500Z

I readā€¦ Whatsapp has 13 main engineers, managing billions of messages

hipster coder 2020-11-17T18:29:55.193700Z

thatā€™s how well the actor model worked for them

hipster coder 2020-11-17T18:30:46.193900Z

hereā€™s my question thoughā€¦ how do I scale beyond 1 process, on Goā€¦. haha

hipster coder 2020-11-17T18:30:51.194100Z

I will research that next

phronmophobic 2020-11-17T18:31:15.194300Z

if you're interested in how to build system, I also highly recommend https://www.youtube.com/watch?v=ROor6_NGIWU

2020-11-17T18:31:25.194500Z

distributed systems introduce a whole set of failure modes that CSP doesn't handle, including the one you mentioned where a backlog of unhandled messages are dropped

phronmophobic 2020-11-17T18:31:31.194700Z

but definitely also check out http://erlang.org/download/armstrong_thesis_2003.pdf

hipster coder 2020-11-17T18:32:20.194900Z

k, watching nowā€¦ Rich Hickey gives incredible talks

phronmophobic 2020-11-17T18:32:24.195100Z

I'm not sure what you mean by doesn't handle. you can certainly model a system that drops a backlog of handled messages using CSP, but you have explicitly model that part of your system

hipster coder 2020-11-17T18:32:34.195300Z

honestly, I always go to Clojure community, for discussing these types of problems

hipster coder 2020-11-17T18:32:56.195500Z

he means, CSP does not handle dropped channels, by default

hipster coder 2020-11-17T18:33:12.195700Z

if a channel fails, all the receivers fail

hipster coder 2020-11-17T18:33:27.195900Z

so the data is lost, the messages are lost

phronmophobic 2020-11-17T18:33:42.196100Z

that's how CSP is implemented, but I'm referring to the CSP model from Hoare's paper

phronmophobic 2020-11-17T18:33:47.196300Z

which is more abstract

hipster coder 2020-11-17T18:33:53.196500Z

ah ok

souenzzo 2020-11-17T18:44:16.196800Z

seqable? also a thing

2020-11-17T18:46:27.197Z

@smith.adriane what I mean is that CSP isn't designed for distributed failure tolerance, that's not what it's even for

šŸ‘ 2
2020-11-17T18:48:00.197200Z

not that a distributed CSP system couldn't be made (maybe it could), but it would rely on something else for failure states, retries, monitoring, time outs, etc. etc. and you'd need a layer that removed those things conceptually

2020-11-17T19:08:43.197700Z

also, going back and reading the context, and reading the quote more carefully: > A consequence of ignoring time is that we refuse to answer or even to askwhether one event occurs simultaneously with another. When simultaneity of apair of events is important (e.g. in synchronisation) we represent it as a single-event occurrence; and when it is not, we allow two potentially simultaneousevent occurrences to be recorded in either order. the distinction I was making is that in an actor model, you don't even have a global order the way CSP defines it. there are various theories and rules about ordering with actors, but it is not well defined or simple the way it is in CSP https://en.wikipedia.org/wiki/Actor_model_theory

2020-11-17T19:10:35.198Z

in an actor model, actors have orderings of events (relativistic), in CSP events have ordering

phronmophobic 2020-11-17T19:11:10.198300Z

but couldn't you explicitly model that within CSP?

phronmophobic 2020-11-17T19:11:52.198500Z

for example, if you made the sending/receiving/dropping of a message an explicit process in CSP

phronmophobic 2020-11-17T19:13:42.198800Z

like CSP doesn't inherently have buffers, but you can model buffers using CSP. you just have to make the buffer an explicit process

2020-11-17T19:17:19.199Z

what I'm saying is that in order to have one CSP system that's distributed, you need something that pretends that you can't read a message then be disconnected indefinitely from the rest of the system

2020-11-17T19:17:32.199200Z

that's an distributed failure mode that CSP doesn't model

2020-11-17T19:18:21.199400Z

you can of course have multiple CSP systems and distribute them, but CSP doesnt' deal with any of the distributed failure mode, that's not what it does or is for

phronmophobic 2020-11-17T19:18:32.199600Z

why can't that be modeled?

phronmophobic 2020-11-17T19:19:22.199800Z

you have a process A (the sender), the process B (the sending) , and process C( the receiver). process B conditionally forwards the message from A to C.

2020-11-17T19:19:48.200100Z

you can model anything in a CSP system, what I'm saying is that none of the CSP features address distribution, it's not a distributed correctness model

2020-11-17T19:20:25.200300Z

in order to model distributed processing, you'd need to implement some other model designed for that use case

phronmophobic 2020-11-17T19:21:58.200500Z

yea, I wouldn't call it a distributed correctness model

2020-11-17T19:22:49.200700Z

so you could a) build CSP on top of an already distrubuted system (it might help with coordination?) b) use individual CSP systems to implement something like raft (would it help here? not sure) but you could swap out CSP with "state machine", or "continuation passing" or "petri net" and get the same result

phronmophobic 2020-11-17T19:23:35.200900Z

I guess i'm not sure what you mean by "build" CSP

phronmophobic 2020-11-17T19:23:41.201100Z

CSP is an abstract model

2020-11-17T19:23:59.201300Z

implement a system which follows the rules of CSP

2020-11-17T19:24:52.201500Z

in order to follow those rules, the failure modes of distribution that CSP isn't designed to account for need to be abstracted away

2020-11-17T19:25:08.201700Z

in other words, something else needs to solve those problems before you start doing CSP

2020-11-17T19:26:28.201900Z

the thing that's important here to me is that we don't confuse the "async programming" problem with the much larger, and harder, "distributed programming" problem

phronmophobic 2020-11-17T19:27:57.202100Z

I agree that fault tolerance has many facets to it.

phronmophobic 2020-11-17T19:28:59.202300Z

I also think that reading and trying to understand CSP will help you understand erlang better

phronmophobic 2020-11-17T19:29:55.202500Z

and learning CSP (the abstract model) can also help you reason about and model distributed systems in a more rigorous way

2020-11-17T19:30:24.202800Z

I'd argue that petri-nets are closer to erlang than CSP is (in a petri net there's no shared clock - two units don't need to operate in any specific timing, which better models network distance)

2020-11-17T19:30:39.203Z

but even petri-nets are not a distributed state model

phronmophobic 2020-11-17T19:31:36.203500Z

CSP can help you model erlang, but erlang processes aren't 1:1 matches to CSP processes

2020-11-17T19:32:30.204200Z

From a practical perspective (as opposed to theory), processes and actors are in a similar solution space: Shared resource management in user space.

2020-11-17T19:32:46.204300Z

erlang doesn't do CSP - but I do agree that learning more models improves understanding

2020-11-17T19:33:07.204900Z

Neither model inherently addresses distributed failure modes.

2020-11-17T19:33:48.205800Z

@potetm it's worse than that though: you can't make the CSP guarantees across a network

2020-11-17T19:34:07.206500Z

(without solving distributed state first, outside CSP)

2020-11-17T19:34:08.206600Z

Erlang built in networked messaging in order to extend the model to multiple machines.

2020-11-17T19:34:19.206900Z

"the model" - which one?

2020-11-17T19:34:25.207100Z

the actor model

2020-11-17T19:34:54.208100Z

But the actor model doesnā€™t inherently address network failures.

2020-11-17T19:34:57.208400Z

right, the actor model makes very few guarantees, so distribution doesn't inherently break it in that way

2020-11-17T19:35:10.208900Z

You have to construct the application in a very particular way to get particular guarantees.

2020-11-17T19:36:35.211Z

So, tl;dr ā€” I think @nathantech2005 is asking for something that isnā€™t built-in to either model.

2020-11-17T19:36:54.211600Z

Nothing magically removes the need to think about various failure modes.

2020-11-17T19:36:59.211800Z

right - but I can call multiple systems on multiple hosts one actor model - that remains coherent

2020-11-17T19:37:26.212400Z

I can't call multiple systems on multiple hosts one CSP model without solving distributed failure first

2020-11-17T19:37:30.212700Z

to me that's a big difference

2020-11-17T19:37:59.213300Z

(of course this doesn't mean the actor model is that much better than CSP - it also doesn't make many promises...)

2020-11-17T19:38:47.213900Z

Iā€™m not sure what distinction youā€™re trying to draw here. This depends on the actor impl iiuc.

2020-11-17T19:39:01.214100Z

I'm saying it's possible

2020-11-17T19:39:07.214300Z

it happens that erlang is async i/o by default, so this is roughly true

2020-11-17T19:39:11.214500Z

the rules for what CSP is make the equivalent impossible under its model

2020-11-17T19:39:28.214900Z

but in practice you can swamp the nic and be in the same boat as CSP ā€” some msgs get delivered, some donā€™t

2020-11-17T19:39:38.215400Z

sure

2020-11-17T19:39:44.215900Z

you can also use dropping buffers in csp to emulate that behavior

2020-11-17T19:39:45.216100Z

or no?

phronmophobic 2020-11-17T19:40:13.216600Z

I think when you refer to CSP, you're assuming a specific implementation running on a set of computers rather than an abstract model

2020-11-17T19:40:15.216800Z

I'm not talking about using CSP to model network failure, I'm talking about implementing one CSP system that spans a network

2020-11-17T19:40:44.217500Z

@smith.adriane was that to me or @potetm?

phronmophobic 2020-11-17T19:40:56.218100Z

to @noisesmith

2020-11-17T19:41:18.219Z

I'm talking about a system that fulfills the rules to be a CSP system, so it offers the guarantees that CSP offers

phronmophobic 2020-11-17T19:41:23.219300Z

it's like the difference between prolog and horn clauses

2020-11-17T19:41:25.219600Z

you know scheme was originally an exploration of the actor model https://en.wikipedia.org/wiki/History_of_the_Scheme_programming_language#Carl_Hewitt,_the_Actor_model,_and_the_birth_of_Scheme

2020-11-17T19:41:30.219900Z

you can't do that in a way that spans a network without solving distributed state first

2020-11-17T19:41:39.220500Z

@noisesmith What difference do you perceive between CSP and ā€œprocesses + queuesā€?

2020-11-17T19:41:45.220800Z

(which isn't inherently distributed or fault tolerant)

2020-11-17T19:42:09.221600Z

And how does modeling a networked queue as a dropping buffer not emulate the actor system?

šŸ‘† 1
2020-11-17T19:42:30.222300Z

absolutely - the distinction I'm drawing is that unlike CSP it doesnt' make promises that are voided once you distribute your algorithm

2020-11-17T19:42:44.222900Z

there are

2020-11-17T19:42:54.223300Z

oh?

2020-11-17T19:43:03.223700Z

the actor model doesn't handle unreliable message sends

jimmy 2020-11-17T19:43:05.223900Z

Just wanted to inject in here that many problems don't need csp, the actor based model, or any thing like it. Many problems work perfectly fine in a stateless model. And if you can get away with that you should. Don't under estimate the usefulness of a stateless api layer, a persistent database, and some external queue/log for communication between subsystems. Debating which concurrency model is better can often take you down a rabbit hole that wasn't really needed. I ran into this as a consultant before where there was a real obsession with elixir's really cool model. But for something that was 100% stateless.

2020-11-17T19:43:14.224100Z

i.e. ā€œAll writes to the network queue go through this dropping-buffer channel. The writer process will handle i/o.ā€

2020-11-17T19:43:35.224800Z

the actor model and csp are both early sort of process formalisms

2020-11-17T19:43:49.225400Z

the pi calculus or petri nets are newer

2020-11-17T19:44:22.226300Z

yeah sorry this has gone way off topic, and I'm not even saying the actor model is better, I'm just trying to assert a distinction (one I was perhaps wrong about...)

2020-11-17T19:44:35.226600Z

well, I guess the pi calculus is contemporaneous with csp, but continued to evolve

2020-11-17T19:45:03.226700Z

once again, I'm not saying "CSP can't model x", I'm saying "you need to solve x before you can have CSP"

2020-11-17T19:45:35.227200Z

CSP describes a set of operations that are possible, they don't really stay coherent across distribution

phronmophobic 2020-11-17T19:46:30.227400Z

there's a difference between the abstract concept of "turing machine" and the implementation

phronmophobic 2020-11-17T19:47:08.227600Z

sure, you can't implement a turing machine because there's no such thing as infinite memory, but the turing machine model is useful for thinking about computers

2020-11-17T19:48:33.228Z

So, just to be clear: Youā€™re saying that, as long as i/o is async, actors need not consider the network directly. You have to do something (like what I just suggested) to get that behavior in CSP.

2020-11-17T19:48:34.228200Z

?

2020-11-17T19:48:43.228400Z

but, pragmatically speaking, we use CSP because it gives us guarantees about async behavior (preventing deadlock, livelock, and race conditions), and network failures undermine all of those guarantees

2020-11-17T19:49:22.228600Z

which is why I'm emphasizing that I can't have one CSP system that spans a network

phronmophobic 2020-11-17T19:50:17.228800Z

i'm saying "one CSP system that spans a network" isn't even a concept

2020-11-17T19:50:21.229Z

Another way of saying that is: You cannot have reliability behind a queue without ACK semantics?

2020-11-17T19:50:24.229200Z

(without first creating an abstraction that lets us pretend network failures don't exist)

2020-11-17T19:50:28.229400Z

(for example)

2020-11-17T19:51:15.229600Z

@potetm it goes beyond that - what do you do with a message that is accepted but the node fails or gets disconnected before any result is calculated

2020-11-17T19:51:37.229800Z

right ā€” ACKing solves this, right?

2020-11-17T19:51:51.230Z

(which formal CSP does not have, I agree)

2020-11-17T19:52:29.230200Z

Wait to be even more clear: ACKing + Retries by the channel

2020-11-17T19:52:33.230400Z

if what you mean by "ACK" includes timeouts, retries, and fail states, sure

2020-11-17T19:52:38.230600Z

right, exactly

2020-11-17T19:52:41.230800Z

ok gotcha

2020-11-17T19:52:53.231Z

yeah I mean, itā€™s a total apples to oranges comparison tho?

šŸ‘† 1
2020-11-17T19:53:17.231400Z

Do actors include those sorts of semantics somewhere?

2020-11-17T19:53:51.231600Z

@smith.adriane I think we've been talking past each other, I wasn't trying to disparage CSP, just point out a very common mistake (one OP was making) of thinking that a distsys failure would somehow be solved or addressed by CSP (or that a failure to handle a distsys state problem was a problem with CSP)

2020-11-17T19:54:04.231800Z

yes, actors include timeouts, retries, etc.

2020-11-17T19:54:25.232Z

I genuinely donā€™t know about formal actor models. I have a passing knowledge of Erlangā€™s model (mostly from reading Armstrong).

2020-11-17T19:54:47.232200Z

really?

2020-11-17T19:54:56.232400Z

erlang does?

2020-11-17T19:55:04.232600Z

or are we not including that as a formal actor model?

2020-11-17T19:55:18.232800Z

I'm actually not an expert on actor systems... I should be more careful about the claim I'm making here

2020-11-17T19:55:46.233100Z

I'm saying that you can do actor system, and add things like retries and timeouts, and still meaningfully use an actor model

2020-11-17T19:56:00.233300Z

partly because the actor model makes very weak guarantees

2020-11-17T19:57:19.233800Z

Okay I think Iā€™m following you. But the end result is roughly the same either way: You have to deal with the problem at every call site somehow. Itā€™s not magically solved.

hipster coder 2020-11-17T19:58:46.234800Z

I was only askingā€¦ because I am talking to CoinBaseā€¦ About Go (CSP) versus Actors on the JVM. Thatā€™s all. if it gives you better context of the domain problems.

2020-11-17T19:58:47.234900Z

But youā€™re point is different: Yes, you have to deal with it everywhere, but the semantics of the system hold. (e.g. the actor is in a HOLD state until it receives the ACK)

phronmophobic 2020-11-17T19:58:48.235100Z

i guess my point is that CSP is meant as a way to model concurrent processes. so the fact that CSP can model a distributed system means that CSP can handle distributed systems. the fact that the model itself doesn't magically turn into a working distributed, fault tolerant system running on AWS is moot

2020-11-17T20:00:42.236Z

@smith.adriane I mean, if youā€™ve ever tried to build in acking semantics to a core.async program, youā€™ll know that itā€™s a real PITA. I think noisesmith has a fair point there.

2020-11-17T20:01:09.236400Z

Like, itā€™s def worth noting if youā€™re gonna compare the approaches.

2020-11-17T20:01:32.237Z

right, I once tried to make core.async in one service communicate with core.async in another, and it was a huge waste of effort and source of unneeded complexity

phronmophobic 2020-11-17T20:01:35.237200Z

i'm thinking of CSP as an abstract reasoning tool, not a library implementation that uses the concepts of CSP

2020-11-17T20:01:35.237400Z

that is a false dichotomy because there are both csp libraries and frameworks for the jvm, and the possibility of using go to implement actors

2020-11-17T20:01:51.237600Z

the failure modes and problems across services are not the ones CSP help me with

hipster coder 2020-11-17T20:02:07.237900Z

I think of fault tolerance as.. a message broker queing up messages if the other side (receiver) fails for a timeā€¦ then that que (channel) can continue sending messages to other receivers, or just waitā€¦ CSP does not offer this by default

phronmophobic 2020-11-17T20:02:15.238100Z

it's like lambda calculus vs. clojure

2020-11-17T20:02:31.238300Z

@smith.adriane right and that's why I waited until my first specific example to mention a specific implementation

hipster coder 2020-11-17T20:02:36.238500Z

if the channel fails on CSPā€¦ the receivers failsā€¦ and thereā€™s no que like GraphQL offers. Not that I am aware of.

2020-11-17T20:02:59.238800Z

the source of complexity wasn't "core.async isn't networked yet" it was "the things CSP does for me didn't help any more at the network boundary"

2020-11-17T20:03:09.239100Z

@smith.adriane The problem isnā€™t with a specific impl. The problem is in the model. Namely, it doesnā€™t include resiliency semantics.

phronmophobic 2020-11-17T20:03:10.239400Z

CSP is to lambda calculus as core.async is to clojure

hipster coder 2020-11-17T20:03:25.239600Z

GraphQL would be slower, over http/tcpā€¦ not as fast as a message broker or concurrency modelā€¦ but itā€™s fault tolerant

phronmophobic 2020-11-17T20:03:30.239800Z

you can model processes that fail

phronmophobic 2020-11-17T20:03:33.240Z

right?

2020-11-17T20:03:46.240400Z

that's why I've been comparing it to actors, something that does mix well with resiliency constructs

2020-11-17T20:04:28.241300Z

@smith.adriane You cannot model channels that ack.

šŸ’Æ 1
2020-11-17T20:04:46.241900Z

Because they are a primitive in CSP and acking is not included in the definition.

2020-11-17T20:05:25.242800Z

the big reason why actors and the actor model get talked about for distributed systems, is in the actor model you don't have channels/queues/whatever as values, you have process addresses, and it is far easier to distribute a simple value like a process address then it is to distribute something as complex as a channel

phronmophobic 2020-11-17T20:05:48.242900Z

is it impossible to model a "sending process" that acks?

2020-11-17T20:05:53.243100Z

@smith.adriane I do agree that you can construct a go block that sends then times out and retries / fails, but I found these sorts of things were a bad match for inter-system failures, and the complexity of this logic bleeds

2020-11-17T20:06:22.243400Z

you can do that - make a channel op that doesn't return until you get an ack

2020-11-17T20:06:29.243600Z

but you can't do that in a go block

phronmophobic 2020-11-17T20:06:53.244300Z

i'm not talking about an actual implementation, I'm talking about using the stuff from Hoare's paper

2020-11-17T20:06:55.244600Z

(right that's an implementation specific problem, but it's indicative of the sorts of mismatches you run into)

hipster coder 2020-11-17T20:07:22.245600Z

. Unfortunately, Golang does not support distributed execution of goroutines on clusters or distributed systems. In this paper, we extend the concurrency capabilities of Golang to a distributed cluster by providing a library called Gluster that is simple and easy to use

2020-11-17T20:07:37.245800Z

all I'm saying is that CSP isn't designed for modeling these sorts of failure states, and other abstractions fit them quite well

hipster coder 2020-11-17T20:07:52.246400Z

They had to use Gluster library to achieve distributed computingā€¦ so if a machine failsā€¦ thereā€™s another machineā€¦ Thatā€™s what I mean by fault tolerant.

2020-11-17T20:07:53.246600Z

not that CSP can't be used in these cases

phronmophobic 2020-11-17T20:08:21.247100Z

like church numerals would be completely impractical for implementing a math library, but you can do arithmetic with church numerals

2020-11-17T20:09:21.249500Z

Iā€™m not sure what youā€™re getting after. The point isnā€™t that you couldnā€™t rig something together that approximates the behavior you want. The point is that doing so runs through your entire program.

hipster coder 2020-11-17T20:09:25.250Z

yes, I just read a paper, Go does not have ability to distribute across machines, by defaultā€¦ have to use a library to do it

2020-11-17T20:09:30.250300Z

right - to distribute CSP you'd need a coherent channel whose behavior / state crosses a network boundary, and that's a distsys complete problem, once you can do that your problem with distribution is solved already anyway

hipster coder 2020-11-17T20:09:36.250800Z

this is also part of what I am getting at with fault tolerance

2020-11-17T20:09:42.251100Z

where actors just need data to cross a network, much simpler

2020-11-17T20:09:49.251400Z

So itā€™s not like, ā€œbuild a CSP thing, tack on some finishing touches.ā€

hipster coder 2020-11-17T20:10:06.252500Z

There is a library called Glusterā€¦ to do that

hipster coder 2020-11-17T20:10:23.253100Z

it seems like a ton of work, to make Go fault tolerant

hipster coder 2020-11-17T20:10:38.253900Z

it seems like the only advantageā€¦ is the control of the order of the workload

hipster coder 2020-11-17T20:11:04.254600Z

example: an assembly line on a factory floor

2020-11-17T20:11:04.254700Z

No oneā€™s saying itā€™s impossible. Itā€™s a fair point that resiliency is not included in CSP by default ā€” itā€™s an exercise for the reader.

mpenet 2020-11-17T20:11:21.255200Z

one big plus of csp is easier flow control management. With actors you have to deal with it backwards, have producers get to know if they can continue to send messages or not via some more complex mechanism (stg like credit based flow) than just blocking or channel op ret check; unbounded mailboxes make it all more convoluted imho

hipster coder 2020-11-17T20:13:29.258900Z

so if I was building a crypto trading platformā€¦ peer to peer sellingā€¦ which model?

hipster coder 2020-11-17T20:13:33.259300Z

CSP or Actors?

uosl 2020-11-19T18:26:40.438100Z

This is the clearest CSP vs Actor model comparison I've read, and I hope you consider turning it into a blog post/article so other's can find it as well! @didibus

2020-11-19T20:01:43.439300Z

Hum... Good idea. I'll add it to my TODO, hopefully I'll get around to it sooner rather than later.

šŸ‘ 1
mpenet 2020-11-17T20:14:03.260800Z

Then usually on the jvm you rely on other means for fault tolerance. Plenty of dist systems on the jvm do just fine without actors

hipster coder 2020-11-17T20:14:08.261Z

do I really care about controlling the order of transactions? I am thinking noā€¦ unless I offer a premium level, complete your order first

hipster coder 2020-11-17T20:14:55.261500Z

yes, thatā€™s also 1 thing I totally love about JVMā€¦ I have access to every possible concurrency model there is

hipster coder 2020-11-17T20:15:08.261800Z

plus, I can run almost any language on it

hipster coder 2020-11-17T20:15:26.262100Z

so all this talk about Java becoming the next Cobolā€¦ blah blah

hipster coder 2020-11-17T20:15:57.262600Z

maybe I donā€™t write so much Javaā€¦ Write Clojure or some Kotlinā€¦ which I love

hipster coder 2020-11-17T20:17:02.262700Z

phronmophobic 2020-11-17T20:17:18.263300Z

the point of learning CSP to me isn't only for making a library that maps closely to the semantics in the paper (like core.async), but also for formal reasoning. For example, Leslie Lamport built on and referenced Hoare's work for reasoning about distributed systems. Just because the model doesn't magically turn into a library that spits out distributed systems doesn't mean it's not useful for thinking about distributed systems. Having good mental models and being able to reason about systems is important too!

hipster coder 2020-11-17T20:17:29.263800Z

thenā€¦ look at Goā€¦ Iā€™d rather write in a JVM langā€¦ This makes my eyes bleed

phronmophobic 2020-11-17T20:18:34.265300Z

for example, https://lamport.azurewebsites.net/pubs/ghl.pdf

2020-11-17T20:18:46.265500Z

did anybody say ā€œdo not learn it, itā€™s garbageā€?

hipster coder 2020-11-17T20:19:01.265800Z

hipster coder 2020-11-17T20:19:06.266200Z

Clojure šŸ™‚ šŸ™‚ šŸ™‚

2020-11-17T20:19:34.266300Z

I donā€™t know what youā€™re responding to. The comment was, ā€œIt has these shortcomings.ā€ The comment was NOT ā€œitā€™s total crap. use actors n00b.ā€

phronmophobic 2020-11-17T20:19:44.266500Z

I didn't try to imply that. I'm sorry if I did. > Iā€™m not sure what youā€™re getting after. I was just trying to clarify what I was getting after

hipster coder 2020-11-17T20:29:13.269Z

my intuition is telling me, use Actors, distributed computing, for a trading system

hipster coder 2020-11-17T20:29:27.269500Z

sending transactions across the world, trading cryptos

hipster coder 2020-11-17T20:32:30.271400Z

I am afraidā€¦ if I commit to using the Go CSP modelā€¦ and this failsā€¦ this will damage my careerā€¦ big decision coming up

hipster coder 2020-11-17T20:36:49.271900Z

yes, thatā€™s also what I am getting atā€¦ I think financial trading systems need distributed systems, so transactions can be sent across machines, to different AWS centers.. this is an international system

hipster coder 2020-11-17T20:37:19.272400Z

I donā€™t see CSP, limited to the machine processesā€¦ and needing libraries and tons of workā€¦ to get it setup for distribution, a good fit for international trading system

hipster coder 2020-11-17T20:37:38.272900Z

I see Go CSP a better fit for sayā€¦ 1 big warehouse

2020-11-17T20:38:50.274300Z

Actors vs CSP is basically akin to ā€œwhat language should I use?ā€ Itā€™s important sure. It has potentially substantial knock-on effects (e.g. hiring filtering and ecosystem). But this decision alone will not sink you. There are about a bajillion more important decisions that are much more likely to sink you.

hipster coder 2020-11-17T20:40:16.274900Z

so I can either make CSP work more like Actors (distributed)ā€¦ but not make Actors work more like CSP?

2020-11-17T20:40:37.275300Z

You can do both or neither.

2020-11-17T20:41:18.275900Z

(You can make a Queue Actor and ape queue semantics.)

2020-11-17T20:41:44.276500Z

(This doesnā€™t magically solve problems of backpressure, but you can do it if you choose.)

hipster coder 2020-11-17T20:43:16.277300Z

ok, good to knowā€¦ thatā€™s what I was worried aboutā€¦ if I go down one path, like CSPā€¦ and it totally failsā€¦ too rigid

2020-11-17T20:43:46.277400Z

I think this is an a/b problem, neither csp nor actors solve distributed state, and what you need is distributed state

2020-11-17T20:44:49.278500Z

This is all pretty vague and amorphous if you ask me. ā€œtoo rigidā€ ā€œsending txns across the worldā€

āž• 1
2020-11-17T20:45:35.279600Z

There is a world where you can be quite confident that your approach will suffice.

2020-11-17T20:45:50.279900Z

What does it take for you to get to that world?

hipster coder 2020-11-17T20:46:14.280400Z

just knowing that I can adjust the concurrency model, or even switch to a different model

hipster coder 2020-11-17T20:46:23.280700Z

and not being locked into one

hipster coder 2020-11-17T20:46:52.281400Z

yes, itā€™s amorphous, because I canā€™t predict the future that far

hipster coder 2020-11-17T20:47:30.282500Z

I am pretty new to concurrency, you geniuses have more experience from the field with this

2020-11-17T20:49:11.283900Z

There is nothing that I know about CSP that you cannot figure out yourself in short order. (e.g. just by reading and experimenting)

hipster coder 2020-11-17T20:49:30.284300Z

yea, I would just write two proofs of conceptā€¦ to test it

hipster coder 2020-11-17T20:49:43.284800Z

I am going to ask the architects if they have done that

2020-11-17T20:50:18.285300Z

But there is something you know that I could never tell you: The particulars of the problem youā€™re solving.

āž• 1
ghadi 2020-11-17T20:51:16.286300Z

Maybe take this conversation to #core-async or #architecture

ā˜ļø 4
2020-11-17T20:51:38.286400Z

Thatā€™s where the meat is. The computer stuff will come as needed, and it will come quickly compared to the particulars of your problem.

hipster coder 2020-11-17T20:54:56.286800Z

k, thanks, I joined #architecture

2020-11-17T22:02:42.287300Z

They are equivalent in terms of what you can do with them. So the choice is more about expressiveness, usability, etc. Which is a bit of a personal preference.

2020-11-17T22:07:05.287500Z

In each model, you have independent processes doing work independently (and thus possibly in parallel). In CSP its called a Process, in Actor its called well an Actor.

2020-11-17T22:08:50.287700Z

So first think, you can have "things that do sequential work on something on their own." In CSP these are called Processes. Each process is a thing doing some work sequentially. In Actor, it is an Actor. Each actor is a thing doing some work sequentially.

2020-11-17T22:09:36.287900Z

That's the building block, and at this point, the two models are the same (minor their different names).

2020-11-17T22:11:55.288100Z

Now, here comes the major part. What if you need to have those things work together collaboratively? What if one thing depend on another? Now you need a way to orchestrate their interaction together so that they are coordinated in what they are each doing. It means they are not fully independent anymore. For example, if one thing is "Sending Emails for each new customer registration" And another thing is "Registering new customers". You can see that the thing sending emails needs to wait for the thing registering customers to tell it that a new customer was registered.

2020-11-17T22:13:15.288300Z

So where CSP and Actors differ, is in how they will coordinate those seemingly independent processes/actors when their operation is dependent on another process/actor

2020-11-17T22:16:49.288500Z

In the case of Actors, they will communicate with each other directly through messages.

2020-11-17T22:17:27.288700Z

In the case of CSP, they will communicate with each other indirectly through a Channel (a queue of messages).

2020-11-17T22:19:10.288900Z

One big difference is that a Channel is a many to many communication. Many processes can put messages in the same channel, and many processes can read from the same channel. You can't do that with Actors, Actors always communicate one on one. So you'd need an Actor who itself would be responsible for fanning out a message from one actor to many others if you wanted a similar kind of one to many or many to many communication.

2020-11-17T22:21:18.289100Z

Another big difference is that Channels can be used to handle back-pressure. Basically, they bugger the communication between Processes. So if one Process is sending messages faster than the other can consume them, you can have the Channel tell the sender to back off. So the sender could wait for the Channel to have more room, before sending more message, or it could choose to go do something else and come back to trying to send messages later.

2020-11-17T22:23:31.289300Z

With Actors you'd need to implement this yourself. So you'd need to either introduce an intermediate Actor that basically replicates this functionality, and it gets a bit more complicated.

2020-11-17T22:24:55.289500Z

Ok, so now I hope you understood why CSP is called: "Communicating Sequential Processes". As you have these Processes that can each be doing their own sequential work, and you have a means to have them communicate with one another if they need to coordinate themselves of exchange data between each other through the Channel abstraction.

2020-11-17T22:29:22.289700Z

Now, you might think... Ok I've only mentioned pros for CSP, and they just seem like they have more features than Actors? So what gives? Well, its true, CSP is more featured. That also means it is harder to implement CSP. Since the processes communicate through Channels, you need to implement Channels, and as Channels have a lot of features: many to many and back-pressure handling. It is much more difficult to build an implementation of Channels, then the direct async fire and forget message passing of Actor communication.

2020-11-17T22:31:46.289900Z

Especially challenging if the Processes are running on different machines. Now you need to implement a robust cross-machine distributed Channel, and that's harder than building a distributed direct message passing that Actors use.

2020-11-17T22:34:31.290100Z

So in general, CSP is used on a single machine, and Actors are used across machines. That said, some people say that taking Actors and distributing them, while "easier", for any robust implementation, will also need to implement on top of them a distributed back-pressure mechanism, and possible distributed many to many messaging. And doing that above Actors might not be any easier than implementing a distributed Channel.

2020-11-17T22:36:53.290400Z

Another thing is, while Channels are more feature-full, if you don't need those features, they can be more cumbersome. Having to go through a Channel indirection if all you need is one to one fire and forget communication is just more convoluted.

A -> B
vs
A -> C <- B

2020-11-17T22:40:59.290600Z

So finally, my advice to you... If you really need to go distributed, what you want isn't Actor vs CSP, but a solid framework that already implemented all the difficult part for distributing your work, may it be based on Actors, CSP or anything else.

2020-11-17T22:42:39.290900Z

This is where Erlang shines, because the Beam VM has already focused on all these hard problems, and the Erlang standard library is full of things specifically to handle distributed systems. And the Erlang community is full of best practices and all just for that problem.

2020-11-17T22:44:39.291200Z

The JVM not so much, but it doesn't mean it has nothing. Akka is quite well built, and a lot of effort have gone into it already to handle a lot of the hard problems of distributed programming for example. Depending what your problem is, Spark or Storm are also good options.

2020-11-17T22:45:10.291400Z

Going solo, and not leveraging an existing framework with years of man power behind it is probably going to bite you.

2020-11-17T22:45:38.291600Z

So when it comes to core.async in Clojure, it is not for distributed systems, but for single machine concurrency. And offers nothing to be able to distribute your Processes and Channels accross machines.

2020-11-17T22:46:13.291900Z

Hope this helps!