Clojurians Log v2

@mpenet How dare you turn down on your hometown?! Especially during « the Escalade »... :) I hoped Facebook Libra would have brought something to Geneva. There is Sixsq and RTS using Clojure in Geneva. I think there are a lot in the biotech scene there though. HPC could be a cool field.

@sam What sort of environment do you find 30m from the center of Zurich? By that I mean is it a small rural town, a contiguous suburb of Zurich, something else?

I forgot sixsq! Great bunch. I didn't know rts is using clojure now, is this just for a few projects (rtslab?) or is it somewhat official?

A rural village in wine country in my case. But you can live in small towns, larger towns (Winterthur), suburbs, you name it. A lot of people do, not only for the quiet, but because Zurich itself has expensive rent. Job-wise I really think it's one of the top places to live. Salaries are high, and taxes moderate, which you can truly benefit from if you manage your living costs accordingly (these can be very high).

@mpenet I think there are even comments on the official webpage of Clojure! https://clojure.org/stories/rts

@paul.legato 30 mins of Zurich can be the middle of litteraly nowhere (especially in the south of Zurich, where you end up in the middle of the mountains).

I am still trying to check who uses Clojure in ZH though.

@neo2551 sorry, not entirely sure if it was mentioned or you are aware, but http://deep-impact.ch does their non-AI backends in Clojure (I used to work there).

They have also done some work with ClojureScript, but the major investment is in the backends.

I hope to use Clojure once my company starts to grow 🙂

http://finity.ai as well (lausanne)

they are also behind http://paper.li, which is also mostly clojure

@sam It would be good to research the AI with Clojure haha, this is where I am mostly interested 🙂 @mpenet Great, I did not know them! Maybe once I have to go back to Geneva, I will think about them.

@sam Where do you work now?

I have my own company, doing consulting and whatever code the client needs, http://saydo.co (very customer-oriented for now)

Did you advise any company on using Datomic?

The companies I have been working for up to now have not been in the context of being able to switch to Clojure or any special DB (in fact, I’m staring at some PHP code right now 😅). We did use Datomic at DI.

I’m quite cautious about recommending major technical movement 🙂 So much value in existing code!

there is also someone from lambdaforge in geneva, @grischoun 🙂

so not datomic, but datahike

@mpenet interesting, you saw datahike use in production?

@mpenet @neo2551 btw, where do you work currently?

They mentioned to me they have production uses. One of which is a project for the Swedish gov

Wow, not bad

I live close to Malmö atm, Sweden

Ah ok! That’s a country I still need to visit 🙂 I nearly went to Linköping for an exchange year, but life came in the way.

I don't really like it here.

we're planning to move back to ch or germany

too cold?

no, other reasons. The weather is not great but it's not the worst of it

We just don't want to remain here for too long, we have small children and we want them to grow up somewhere else

Ah! Yes, that is always a source of big questions being asked. My kids are also a major reason I am currently living here.

I am working at Vontobel

@sam

Yeah, this is also a reason why I am reluctant to move as well. Education is great in CH.

Not that the education system is the best, but when I combine all factors, it is hard to find a better place for a family than where I am..

Yes, it is good! Just not the best 🙂

I am also heistating between crux and datahike.

Where do you think education is better?

@neo2551 Vontobel is doing Clojure?

ch is a great place wrt to education

EPFL was a fun right?

I do remember Satelite time

I think the higher education is incredible here, yes!

Totally, many hours spend there 😄

Basic education is however a bit rooted in the past.

It depends a lot on the school, and we are fairly happy with the school now.

Yes I agree, but I believe that parents have also a big role in education as well, at least in CH I had this feeling.

Absolutely. I think there is a mentality of heavy guidance, and not as much free learning as modern pedagogy seems to advise. But this is arguing at a quite high level.

So to answer the Clojure question: basically, I am a mathematician doing AI/ML/Statistics for finance. I was frustrated with our tools, so I started to build small webapps with python dash/R shiny, and my big boss asked me to make client facing quality web apps, so I told them to it was impossible with his short term deadline to use traditional tools, and he should make the bet to let me use whatever I wanted. I already knew Clojure, so doing ClojureScript was blessing for me 🙂

Since then, we destroyed most metrics of effectiveness / costs, so I can still use ClojureScript and now they want me to build the Kafka infra for our team, so I am still going to do it with Clojure.

It is so much faster to learn JVM tools with Clojure.

Wow, very cool!

Thanks, but nothing crazy.

Just leverage on all the knowledge and kindness of our community.

The trick is our requirements are fairly small compared to real world constraints.

My audience will use a chrome based browser, they will have decent computers, they will probably be in Switzerland or sufficiently near.

Yes but still, congrats on the success of bringing in that degree of innovation to an enterprise env. I have heard a couple of stories that ended differently in that regard.

And in banks, anything bigger than an Excel is big data.

True 😄

yeah about edu, sweden is full of montessori/waldorf, but they just use the name, it's rare to have teachers with actual education/certificates in these

I think I embraced talk from Rich for the 10 years of Clojure: I focused all the arguments on the artefacts and not the constructs: is what I am doing working for the definition of my boss? Can I change? Modify? Debug? Yes to all of this and at crazy speed.

Nothing, absolutely nothing can beat live reloading while maintaing the state of your app in UI.

I am sold on attributes at the bottom, datomic, spec & all, took me a while to accept/understand it but I really think he's onto something. I wish we could use datomic at work, but licensing makes it a bit tricky atm

Maybe you try datomic free?

datahike is still young, and there's always the hope that datomic on-prem gets an open-source version at some point

Especially when you got a lot of that state. @mpenet re education, if you want actual montessory/waldorf you need a private school here, but the ones I heard about are really good. There are a lot of private Steiner schools, which are less expensive.

My biggest problem with Datomic is I fear I would max out the the size for table for on prem.

sam: it's free in sweden, my daughter is in a montessori preschool

Yes, that is not available here!

in ch it would be a montain of money

but quality would be much better for sure too

@neo2551 yes, I think with datomic you have to think very early how to shard data(bases) efficiently

I guess it depends on the volumes and uses cases too

but for instance nubank have multiple db per user

The problem could be (IMHO) circumvented if we could have separated databases on single machine and not loosing too much time when creating the indices.

since you can query cross-db that's also ok'ish

yes, exactly.

but then Datomic does not recommend to have multiple DB per transactor...

Datomic is even harder to push for in enterprise envs than Clojure in my experience. Customers often have pretty orthodox but strong opinions about their data.

I argue that the data is stored in Postgres or Kafka

(then I use Datahike or Crux haha)

crux doesn't scale right now

all your data has to fit on every node

really?

you get replication sure, but no way to scale out for now

I thought they solved it with SQL stored

no

not as far as I know

What about S3 data store?

I gotta head back into my code, was good talking @neo2551@mpenet! 👋

dunno about this

@sam Thanks to you! See you again.

See you!

@mpenet https://opencrux.com/reference/20.09-1.12.1/kafka.html They say you can use S3 for the document store. So I guess it does scale?

I might be wrong though, I am still wondering what is the best solution.

indices have to be stored in lmdb or rocksdb

and iirc these need to be on every node

ok make sense.

and indices are big

yes true.

Did you check Datomic Cloud then? Does it solve that problem?

personally I prefer the design of datomic/datahike

too expensive, not open enough 🙂

haha agreed.

I can't even try it at work 😞

datahike is promissing, we ll see

yep. On the other hand at 5k per year per transoactor (less if you don't update them)

I think it could still be acceptable.

not always

I work for a cloud provider: they wouldn't agree to pay to have something hosted on aws, plus there's the question of data protection given it's an US entity

I was thinking of Datomic on Prem

sorry

sure, I guess we could, but it's not in my hands 🙂

Hi @mpenet, hi guys! Good to hear from swiss clojurers. As @mpenet mentioned, i live in Geneva and work with lambdaforge.

So cool! So you are from the team that works on datahike?

Hi all! Its nice to see some activity in here. Im from close to Winterthur and working for a Startup in Zürich. Im interested in Clojure but sadly didnt find an opportunity to work with it as of yet. Its good to hear that there are some companies in Switzerland using it!

Hi @mpenet 🙂 so when you say "scale" here I assume you only mean "unpredictable horizontal scaling of queries"? Otherwise I have to disagree that Crux does not scale when compared to ~similar alternatives. I think there are many dimensions of scaling other than "unpredictable horizontal scaling of queries" where Crux is arguably very scalable (compared with the mentioned alternatives). For instance: • sustained write throughput (vs. writing to distributed indexes which are inevitably slow) • total index size (RocksDB is very happy handling 10s of TBs and queries remain fast without the need for gigantic in-memory caches) • snapshot-based scaling avoids many cold-cache issues • handling of large unstructured maps (tested with 10-100 MBs) ^ a benchmark report will be published in Q1 to put all this in context. To mitigate the local-index requirement we have added a snapshotting feature that allows new nodes to come online and sync quickly (i.e. they only have to replay the recent transactions since the last snapshot). Large values are never stored in the indexes. And to further mitigate the main concern of index duplication, I have proven the concept of using Redis as a shared index store which can in principle support ~infinite nodes running in Lambdas. Such a Lamdba-based setup is only viable with Crux because the query engine is lazy and therefore has low memory requirements (i.e. it doesn't compute the clauses/joins one-at-a-time, which is a memory-intensive approach): https://github.com/crux-labs/crux-redis (the main trade-off is that queries will be a little slower due to network hops)

Hi @neo2551 I wrote a response to the earlier comment that you may want to skim 🙂 https://clojurians.slack.com/archives/C08QBF7B5/p1607972774115600?thread_ts=1607940371.097800&cid=C08QBF7B5

the statement was a bit too broad sure. I meant size of full dataset/index. Sure you can have boxes with 10tb and put your index on them, but that's not very convenient for a few reasons (costs, operational among others) and you'll still be bound by limits in storage size per box.

but my crux knowledge is quite outdated, so maybe I missing the big picture

write throughput surely is fast, I guess it's essentially benchmarking kafka, write/read roundtrip is probably a different story, even then I expect it's fast enough in most cases

but the one reason I am not considering/using crux right now is because of the sharding issue, it feels like a pain point for long running clusters

about redis: interesting POC, tho redis is also problematic in its own ways, it's also one more dependency to manage/operate

the "doesn't scale" was only for the index issue. Another problem with that would be that spreading write load on indexes is not possible, If you have a lot of writes, surely kafka will keep up but all your nodes will also be busy writing index data. And in turn that would impact querying on all these nodes.

crux is way more mature than some of the other solutions I mentioned tho, both in tooling and the dev behind it, major props on that. I prefer the datomic design personally, but that's also a matter of different use cases.

@taylor.jeremydavid I am actually wondering if you are working on sharding/partitioning at all, other than via redis?

it's a tough nut to crack but there are plenty existing solutions to get inspiration from

Thanks for the responses! It's certainly a subtle and complex discussion. We have it on our backlog to try Crux out with this "cloud-native" fork of RocksDB (by the Rockset team of ex-RocksDB Facebookers): https://github.com/rockset/rocksdb-cloud this is an insightful post https://rockset.com/blog/remote-compactions-in-rocksdb-cloud/ there are a few presentations on it also. Clearly Rockset are using this fork heavily themselves in their cloud offering and I think it would be a good bet for us, if it works like we think it will. This would go a very long way to solving all the issues you mention without needing to change the nature of the sorted-kv index structure of Crux itself. > Sure you can have boxes with 10tb and put your index on them, but that's not very convenient for a few reasons (costs, operational among others) and you'll still be bound by limits in storage size per box. Cost is definitely an important dimension for analysis and critique, and I believe was the biggest motivator for Rockset's fork. I think if "limits in storage size per box" are truly an issue then Crux is probably not the right fit for the problem at hand (e.g. EBS volumes go up to 16TB, and i3en has 60TB of local SSD). > I am actually wondering if you are working on sharding/partitioning at all We are certainly thinking about it often. Did you see Håkan's short segment about the roadmap in our Re:Clojure video? https://youtu.be/JkZfQZGLPTA?t=1325

Interesting. No I didn't have the time to watch the videos yet and I missed part of the qa when it was live