Discussion about all editors used for Clojure/ClojureScript
bozhidar 2018-08-06T06:47:55.000033Z

@pez No, nothing special. We’ve been polishing the Elisp bencode code for quite a while and it’s pretty reliable at this point.

dominicm 2018-08-06T06:52:11.000185Z

@pez just a quick theory, if your network callbacks aren't complete, then perhaps you're receiving chunks? e.g. di6e followed by i3ee

dominicm 2018-08-06T06:52:20.000202Z

I've been wondering about whether this can happen.

dominicm 2018-08-06T06:54:39.000146Z

I mean, it definitely can, that's just TCP.

dominicm 2018-08-06T06:54:58.000247Z

@bozhidar does elisp bencode stream?

bozhidar 2018-08-06T07:01:26.000067Z

I don’t remember at this point to be honest. Haven’t touched this code since the early days of the project.

bozhidar 2018-08-06T07:01:52.000037Z

I just remember it was pretty buggy, we fixed some bugs and forgot about it. 😄

dominicm 2018-08-06T07:14:55.000101Z

python-nrepl couldn't handle negative numbers or empty lists for a very long time.

dominicm 2018-08-06T07:15:30.000056Z

Only a clojure programmer would have made the mistake, empty lists are falsey in python 😄

bozhidar 2018-08-06T07:17:06.000149Z

I do remember that the Emacs bencode also didn’t handle negative numbers in the beginning.

dominicm 2018-08-06T07:17:48.000227Z

I think it's a common mistake to overlook

dominicm 2018-08-06T07:18:20.000226Z

https://github.com/clojure-emacs/cider/pull/616/files I see something here for handling incomplete streams. @pez

pez 2018-08-06T07:29:23.000247Z

t seems that when the result is large it is chunked up, yes, but the chunks are not encoded correctly, by what I can see. That is probably my error, when adding stuff to the buffer. Hmmm… Thanks for that question, @dominicm!

dominicm 2018-08-06T07:30:35.000176Z

I wrote a lot of C code as a teenager, I now have a bit of a sixth sense for network problems and race conditions. It's a little trigger happy, but I like it that way 😄

pez 2018-08-06T07:30:38.000005Z

In fact I get decode errors for some non-large result values as well, but I haven’t looked at that yet, I do think empty lists and negative numbers are handled, though. 😃

dominicm 2018-08-06T07:31:10.000141Z

@pez I found generative testing amazing for this. I can highly recommend pairing your encode/decode functions in a generative test.

dominicm 2018-08-06T07:31:28.000021Z

It ironed out all of the problems in python-nrepl-client (from what I can tell, anyway)

pez 2018-08-06T07:32:22.000044Z

I’ve never looked at generative testing. It seems to be time to open that box now.

pez 2018-08-06T07:34:34.000220Z

(In case it isn’t obvious I am still very much a Clojure (and functional) noob. I can even admit to not having groked functional fully yet. Still waiting for that “click”… But that is why I am rewriting Calva in ClojureScript, eventually it has to happen!)

dominicm 2018-08-06T07:34:53.000129Z

I had been waiting a very long time for a chance to use it 😄 It doesn't come up so often.

pez 2018-08-06T07:35:01.000284Z


dominicm 2018-08-06T07:35:16.000145Z

I suspect that's a function of the way the codebases I work on are written too though.

dominicm 2018-08-06T07:35:30.000252Z

Most of it is side effecting, and very little of it is data pushing & business logic.

pez 2018-08-06T07:36:40.000102Z

I am trying to factor the Calva code so that ClojureScript is doing almost only data pushing and business logic, and then keep the side effecting out in the TypeScript glue into VS Code.

dominicm 2018-08-06T07:37:20.000202Z

that makes sense, that should be amenable to generative testing then 🙂

pez 2018-08-06T07:39:01.000018Z

I am pretty sure it is. But I haven’t even looked at spec yet, so I have some learning to do. Guess I should be happy there is still so little of Calva ported to CLJS yet. 😃

pez 2018-08-06T07:39:25.000172Z

BTW, I heard that @bozhidar hates spec and generative testing. 😃

dominicm 2018-08-06T07:40:00.000209Z

o rly?

dominicm 2018-08-06T07:40:20.000199Z

spec and generative testing do pair, but test.check can be used without spec.

pez 2018-08-06T07:42:01.000001Z

I must find a way to test this programmatically anyway, the manual way I am doing it now is driving me mad.

dominicm 2018-08-06T07:46:51.000098Z

I've just been trying to wrap tests around pack (uberjarring command line tool I wrote). It's surprisingly hard. I wanted something that ran the CLI and tested that the jar behaved correctly. I've found so many bugs in shell-based harnesses. It's really hard to integration test non-http things. Vim does have some harnesses about, but I haven't tried writing tests for replant yet, in the fullness of time I suppose I should though 🙂

dominicm 2018-08-06T07:48:03.000118Z

What does testing VSCode look like?

bozhidar 2018-08-06T07:48:11.000065Z

> BTW, I heard that @bozhidar hates spec and generative testing. 😃

bozhidar 2018-08-06T07:48:14.000302Z

Really? 😄

bozhidar 2018-08-06T07:49:01.000266Z

I don’t like spec in its current form, that’s true, but I don’t hate idea in general. Although I do feel that it goes against the nature of dynamic programming languages to a large extent.

bozhidar 2018-08-06T07:49:45.000255Z

I don’t have anything against gen testing, but in my experience the touted benefits are rarely strongly pronounced.

dominicm 2018-08-06T07:50:31.000065Z

It's either a great fit or a terrible fit

dominicm 2018-08-06T07:53:48.000130Z

https://lispcast.com/testing-stateful-and-concurrent-systems-using-test-check/ That is a lot of effort for just k/v, I suspect the complexity here is exponential, so that's a lot of work for most systems.

pez 2018-08-06T08:29:44.000114Z

I do hope that smiley did its work when I put words in your mouth, @bozhidar ❤️

pez 2018-08-06T09:20:45.000099Z

@dominicm I haven’t looked much at testing in VS Code. For Calva Formatter (a separate extension) I have unit tests on the CLJS side, but I can’t leverage them fully because I have stubbornly stuck to keeping my tests together with the functions they are testing and cljs,test do not discover them. I might give up un that, but have so far now and then tried to figure out how to get the testrunner to see them… Sometimes I wish I was less stubborn about things.

pez 2018-08-06T09:30:12.000275Z

I found Eric Normand’s talk about making composable abstractions very interesting, even though most of it flew over my head. He is specing his way forward in the process: https://youtu.be/jJIUoaIvD20

bozhidar 2018-08-06T12:20:41.000072Z

Btw, @pez, I think that probably you and other people who can work directly with JS or CS in their editor would probably benefit a lot from a native JSON/EDN transport in nREPL.

bozhidar 2018-08-06T12:21:12.000166Z

That should be pretty simple to setup and would save you from bencode completely.

pez 2018-08-06T12:22:02.000013Z

transit even?

bozhidar 2018-08-06T12:22:02.000235Z

It’s as simple as implementing a function like https://github.com/nrepl/nREPL/blob/master/src/clojure/nrepl/transport.clj#L84

bozhidar 2018-08-06T12:22:20.000299Z

Well, we can have a transport in whatever format we want.

bozhidar 2018-08-06T12:22:42.000158Z

Bencode was chosen for the default transport with the assumption it would be easy for many editors to support it.

pez 2018-08-06T12:23:49.000210Z

My experience with JSON and ClojureScript is that it can be slow, but EDN makes total sense.

bozhidar 2018-08-06T12:25:11.000171Z

I think that for the small messages that are typically encoded probably any format will do. 🙂

bozhidar 2018-08-06T12:25:27.000203Z

But yeah - obviously the more efficient the format, the better.

pez 2018-08-06T12:26:52.000299Z

Going from JSON to Transit sped up the initialization of SPA I am working with (for food) magnitudes.

pez 2018-08-06T12:27:39.000272Z

And that was not huge amounts of data, even though larger than typical nREPL results.

pez 2018-08-06T12:28:24.000095Z

Anyway, what would it entail? Updating nREPL?

bozhidar 2018-08-06T12:28:32.000120Z

The problem is that ideally nREPL should not have 3rd party deps, as they can conflict with something in the project of the people using it.

pez 2018-08-06T12:29:21.000213Z

Makes sense.

bozhidar 2018-08-06T12:29:46.000218Z

But yeah, we just need a different transport function that uses say EDN and that’s it.

bozhidar 2018-08-06T12:30:15.000343Z

The encoding/decoding will be trivial then.

pez 2018-08-06T12:30:26.000214Z

Want me to pack a PR?

bozhidar 2018-08-06T12:30:59.000018Z

Yeah, that’d be great is you’re willing to tackle this.

pez 2018-08-06T12:31:38.000014Z

Seems like a Good First Issue ™️

bozhidar 2018-08-06T12:33:18.000257Z


bozhidar 2018-08-06T12:33:53.000093Z

It will also be an opportunity to extend http://nrepl.readthedocs.io/en/latest/design/#transports 😄

bozhidar 2018-08-06T12:34:05.000142Z

It’d be nice if we added some small tutorial on the subject.

pez 2018-08-06T12:34:36.000091Z

Anything I need to know about how to hack on this project?

dominicm 2018-08-06T12:45:56.000343Z

it would be quite neat if there was a cli-friendly way to inject a transport :thinking_face:

dominicm 2018-08-06T12:46:15.000228Z

OTOH, using bencode everywhere is super beneficial as it means you can connect to any arbitrary nrepl server.

bozhidar 2018-08-06T12:47:34.000385Z

@pez Just a normal Clojure project. Nothing special about it. There’s a section on hacking in the manual, but it’s pretty basic.

bozhidar 2018-08-06T12:48:34.000329Z

@dominicm Well, if there were more bundled transports that wouldn’t really change anything. After all each transport defines it’s own protocol, so you can use all of them depending on your preference.

bozhidar 2018-08-06T12:48:46.000342Z

(I might not understand your reservation, though)

dominicm 2018-08-06T12:49:37.000265Z

@bozhidar as long as all are loaded, sure. But you end up with people saying "Oh, emacs uses bencode, I'll start that" and http/edn never gets loaded.

bozhidar 2018-08-06T12:49:43.000001Z

As for the injection of transports - that shouldn’t be hard, but I think something like an json or edn transport should probably be built-in.

dominicm 2018-08-06T12:49:56.000196Z

json without a dependency is very unlikely

bozhidar 2018-08-06T12:50:11.000279Z

I think the JDK has something built-in.

dominicm 2018-08-06T12:50:51.000008Z

huh, so there is

dominicm 2018-08-06T12:52:01.000046Z


bozhidar 2018-08-06T12:52:20.000027Z

I just remember that had something built-in for XML, so probably they caught on the json front as well. 😄

bozhidar 2018-08-06T12:54:00.000228Z

> But you end up with people saying “Oh, emacs uses bencode, I’ll start that” and http/edn never gets loaded.

bozhidar 2018-08-06T12:54:35.000304Z

Well, if the EDN parser for Emacs matures that might change. Time will tell. 🙂

dominicm 2018-08-06T12:55:56.000269Z

Sure, but then vim hangs out on bencode and it's still true 😉

dominicm 2018-08-06T12:56:45.000082Z

Choice is okay, except when you want to try and have a uniform interface. Then you get things like content negotiation. At that point we should use http 😄

bozhidar 2018-08-06T12:58:05.000141Z

Yeah, of course.

bozhidar 2018-08-06T12:58:51.000011Z

There’s another reason why more transports (json in particular) have been on the back of my mind - which such a transport it’d be easy to expand nREPL to an LSP implementation. 🙂

bozhidar 2018-08-06T12:59:04.000298Z

(although you can obviously just proxy an LSP server to nREPL)

dominicm 2018-08-06T13:00:43.000320Z

I'm waiting for the latter to happen before I consider the former.

bozhidar 2018-08-06T13:18:49.000350Z

Yeah, baby steps.

bozhidar 2018-08-06T13:18:53.000382Z


dominicm 2018-08-06T13:21:07.000397Z

Overall, bencode has been surprisingly good. It's got limited expressivity which sucks (transit over bencode anyone?) but writing a bencode decoder/encoder is difficult primarily because of handling streaming in the language, which is quite fundamental anyway, you'd have to do the same with JSON. I'm quite a simple programmer, and bencode is well within grasp for me to write a parser/encoder for. Json, not so much.

bozhidar 2018-08-06T13:54:15.000176Z

No argument here. But there’s also the point it’s unlikely you’ll have to write a json parser yourself. 🙂 Anyways, Let’s bench the json idea for now and only focus on edn.

dominicm 2018-08-06T14:02:16.000496Z

I'm just thinking out loud really. I wonder if an json streaming parser exists for most languages.

pez 2018-08-06T16:04:15.000154Z

Hmmm, there is no write in clojure.edn, right?

pez 2018-08-06T16:10:12.000156Z

My computer goes super hot and then becomes quite unresponsive working with the nREPL project. Seen something like that, @bozhidar?

bozhidar 2018-08-06T18:05:55.000474Z

> Hmmm, there is no write in clojure.edn, right?

bozhidar 2018-08-06T18:06:09.000084Z

I’ve never used it, so I can’t answer that one. 🙂

bozhidar 2018-08-06T18:06:17.000305Z

> My computer goes super hot and then becomes quite unresponsive working with the nREPL project. Seen something like that, @bozhidar?

bozhidar 2018-08-06T18:07:23.000185Z

First time I hear this. Did you notice anything in particular that triggered this? Sounds like some infinite loop or something, but who knows…