Clojurians Log v2

whoa!!!

root@raspberrypi:~# time echo ':cljs/quit' | planck 
ClojureScript 1.10.516

real	0m0.111s
user	0m0.100s
sys	0m0.010s
root@raspberrypi:~# time echo ':cljs/quit' | planck 
ClojureScript 1.10.516

real	0m0.062s
user	0m0.050s
sys	0m0.010s
root@raspberrypi:~# time echo ':cljs/quit' | planck 
ClojureScript 1.10.516

real	0m0.093s
user	0m0.080s
sys	0m0.000s

that's on a raspberry pi 3

that’s pretty good right?

that's really fast!

clojure startup time on the same machine is 16 seconds 🙂

Unfortunately, what you are seeing there is simply the time to shut down without really needing to wait for the ClojureScript (JavaScript) runtime to initialize.

oh ha 🙂

This is part of what makes Planck appear to be fast: The C part of it is up immediately, allowing you to type.

That’s what I expected too. Try time plk -e '(+ 1 2 3)'?

root@raspberrypi:~# time echo '(+ 1 1)' | planck 
ClojureScript 1.10.516
2

real	0m17.065s
user	0m17.210s
sys	0m0.270s

ah

ah well, repl-drive development as they say 🙂

is planck compatible with something like SLIME?

Another test is time echo '(js/PLANCK_EXIT_WITH_VALUE 1)' | planck

For Emacs integration, see http://planck-repl.org/ides.html

Unfortunately I ran into an illegal instruction while building on an OLPC (i586)

### AOT compiling macro namespaces
+ mkdir -p planck-cljs/out/macros-tmp
+ planck-c/build/planck -sk planck-cljs/out/macros-tmp '-e(require '\''cljs.analyzer)' '-e(do (set! cljs.analyzer/*cljs-warnings* (assoc cljs.analyzer/*cljs-warnings* :undeclared-var false)) nil)' -e '(require-macros '\''planck.repl '\''planck.core '\''planck.shell '\''planck.from.io.aviso.ansi '\''clojure.template '\''cljs.spec.alpha '\''cljs.spec.test.alpha '\''cljs.spec.gen.alpha '\''cljs.test '\''cljs.pprint '\''cljs.analyzer.macros '\''cljs.compiler.macros '\''cljs.env.macros)'
./script/build: line 154: 21317 Illegal instruction     planck-c/build/planck -sk planck-cljs/out/macros-tmp -e"(require 'cljs.analyzer)" -e"(do (set! cljs.analyzer/*cljs-warnings* (assoc cljs.analyzer/*cljs-warnings* :undeclared-var false)) nil)" -e "(require-macros 'planck.repl 'planck.core 'planck.shell 'planck.from.io.aviso.ansi 'clojure.template 'cljs.spec.alpha 'cljs.spec.test.alpha 'cljs.spec.gen.alpha 'cljs.test 'cljs.pprint 'cljs.analyzer.macros 'cljs.compiler.macros 'cljs.env.macros)"
+ checkCmdSuccess
+ CMD_RESULT=132
+ '[' 132 '!=' 0 ']'
+ echo 'Build Failed.'
Build Failed.
+ exit 132

The nice thing about what Planck is doing there, is if you start the Planck REPL, oftentimes it takes you a few seconds to actually enter the first form, so it is nice to have the textual UI up and running before you need to evaluate. The latest shipping ClojureScript (non-browser) REPLs now play the same trick.

ah neat

Yeah, if you hit an illegal instruction, then it is probably a compiler defect, and the only recourse is to pull out a debugger and get really low level.

Meaning a gcc or clang compiler defect.

maybe something i686-specific, or SSE, etc?

It could also be an illegal instruction in machine code generated by JavaScriptCore

is script/build --verbose verbose enough to show things like -march= and such?

oh good point

I bet you'd have to revise the build script to see more info

that should be pretty simple

its a bit of an odd beast, so I'm not surprised:

root@olpc# cat /proc/cpuinfo 
processor	: 0
vendor_id	: AuthenticAMD
cpu family	: 5
model		: 10
model name	: Geode(TM) Integrated Processor by AMD PCS
stepping	: 2
cpu MHz		: 430.954
cache size	: 128 KB
fdiv_bug	: no
f00f_bug	: no
coma_bug	: no
fpu		: yes
fpu_exception	: yes
cpuid level	: 1
wp		: yes
flags		: fpu de pse tsc msr cx8 sep pge cmov clflush mmx mmxext 3dnowext 3dnow
bogomips	: 861.90
clflush size	: 32
cache_alignment	: 32
address sizes	: 32 bits physical, 32 bits virtual
power management:

One nice aspect about Planck (for this sort of stuff), is that it is really just a small C wrapper driving JavaScriptCore. That makes it easy to mess around on small boxes.

yeah, very cool

thanks for writing it!

@jasonpepas If it is indeed an illegal instruction caused by machine code generated by JavaScriptCore, some of those optimizations can be disabled by simply setting environment variables.

One interesting related flag that might speed up things on older hardware is to set JSC_useFTLJIT to true. I is currently defaulted to false in Planck to avoid crashing that was introduced in a recent version of JavaScriptCore.

@jasonpepas For your illegal instruction problem, setting an environment variable JSC_useJIT=false would turn off all JIT compilation, and perhaps avoid the issue and allow the build to succeed.

what are you going to use planck for on a raspberry?

Thanks @mfikes I'll give that a shot

I'd really like to see this. One area that we haven't yet "reached" with ClojureScript is extremely tiny computers. For example, what if you wanted to prototype some small bit of hardware but drive it with ClojureScript. It seems like Planck / JavaScriptCore almost gets you there, but one huge issue I don't have an answer for is the fact that the core library is pretty huge.

@borkdude just little personal servers, like a wiki, etc.

@mfikes I was wondering if it would be possible / make sense to perform some sort of just-in-time eval for things like the core library? i.e. somehow insert placeholders into the environment which will trigger that code to get eval'ed the first time it is executed. Or does that not make sense in the context of something like clojure?

(or did you mean simply the disk size of the core library?)

Oh, so if you ask the question: Is is possible to run a ClojureScript REPL in a device that, say, only has 256 MB of RAM, you start to encounter issues where the core library is too much to fit in there.

ok, gotcha, you just don't want the whole library in ram

Yeah, there are nice tiny machines you can get with USB or other really simple connectivity, and it would be interesting to drive them with ClojureScript.

I'm tempted to ask if it would be feasible to implement something where forms could get kicked out of the environment and force them to be re-eval'ed from disk on next call, but I'm not sure that's actually any different than just loading everything into RAM and letting the OS use a swap file

I think if you use :advanced you can produce a sufficiently small artifact that would work on those machines. (I think I did that with the Lego MindStorms brick at one point... but I can't recall). But having a REPL is the challenge with respect to RAM.

Yeah, avoiding loading the entire standard library, but lazily loading stuff via thunks might be a trick that would lead to it working.

Planck does that, for example with parts of the analysis cache. It does that not to save RAM but to reduce startup latency in the case you don't need those portions of the cache.

I wonder if that would speed up startup time as well, or is simply parsing the edn most of the cost?

Yeah, I think that modern JavaScript engines even try to parse JavaScript in a way that defers as much as possible. So, it might not fully parse a big function until you call it. But I really don't know.

oh interesting

All, I know is: There are tiny machines that are very cool to do stuff with, and it would be nice to somehow cram ClojureScript into them 🙂

definitely!

I think I need to buy one and just try it out. 🙂 They aren't too expensive I've heard. 🙂

what's that, a lego mindstorms kit?

Or I can do stuff like limit the RAM in a Unix shell.

Oh, it is a way to essentially build programmable robots with legos.

My son has a couple of these https://www.amazon.com/LEGO-MINDSTORMS-31313-Educational-Programming/dp/B00CWER3XY/ref=asc_df_B00CWER3XY/?tag=hyprod-20&linkCode=df0&hvadid=241989243824&hvpos=1o1&hvnetw=g&hvrand=15996673225011530683&hvpone=&hvptwo=&hvqmt=&hvdev=c&hvdvcmdl=&hvlocint=&hvlocphy=1027188&hvtargid=pla-381559943747&psc=1

yeah, actually I audited an EE course while working for UT and we built a little line-following robot using one of those

It is essentially Lego's take on an Arduino

The visual programming language for it was pretty interesting

Yep. I actually started a project to make an IDE for its language for the iPad. But never carried through with it.

Having already been through college and out in the working world, I had the benefit of being able to plan out a project and stay on task. I'll never forget that one of the teams made a robot which simple ran into a wall and sat there, while playing the imperial death march on its little piezo speaker. They had literally spent the entire project on making it play that song.

An iPad IDE would be fantastic!

> All, I know is: There are tiny machines that are very cool to do stuff with, and it would be nice to somehow cram ClojureScript into them Yeah. There’s also joker, which is a small Clojure interpreter built in Go. I’ve never really done scripting with that, but it might also run in tiny machines.

It’s probably very limited, not as far reaching as ClojureScript. I use it mostly as a linter, which is very cool imo.

I ran strace and I'm not sure if this is definitive, but it pauses at the futex( call for about 3 seconds before hitting the illegal instruction:

open("/root/.planck_keymap", O_RDONLY)  = -1 ENOENT (No such file or directory)
nanosleep({0, 1000000}, NULL)           = 0
ioctl(0, SNDCTL_TMR_TIMEBASE or SNDRV_TIMER_IOCTL_NEXT_DEVICE or TCGETS, 0xbfbec148) = -1 ENOTTY (Inappropriate ioctl for device)
fstat64(0, {st_mode=S_IFREG|0600, st_size=8, ...}) = 0
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb771b000
read(0, "(+ 1 1)\n", 4096)              = 8
open("/root/.planck_history", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 4
fstat64(4, {st_mode=S_IFREG|0644, st_size=0, ...}) = 0
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb771a000
write(4, "(+ 1 1)\n", 8)                = 8
close(4)                                = 0
munmap(0xb771a000, 4096)                = 0
futex(0x843dce4, FUTEX_WAIT_PRIVATE, 1, NULL <unfinished ...>
+++ killed by SIGILL +++
Illegal instruction

(somehow the build had completed, I suppose!)

The build script creates a “stage 1” Planck executable that is then used to AOT compile shipping macros, to be included in a “stage 2” Planck executable. I bet this use of the stage 1 Planck failed during the build. But it’s binary is there nevertheless for you to use, and evidently repro the SIGILL

oh interesting

so if the problem is in JSC it might exist in both stage 1 and stage 2

I tried JSC_useJIT=false but I killed it after a bout 5 minutes (not sure if it was stuck in a loop or if JS is really that much slower without JIT

ah, gdb finally hit something

Reading symbols from ./planck-c/build/planck...(no debugging symbols found)...done.
(gdb) run
Starting program: /tmp/planck-2.20.0/planck-c/build/planck 
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/i386-linux-gnu/libthread_db.so.1".
[New Thread 0xb50c1b40 (LWP 22038)]
ClojureScript 1.10.516
[New Thread 0xb48c1b40 (LWP 22039)]
cljs.user=> [New Thread 0xb3effb40 (LWP 22040)]

Program received signal SIGILL, Illegal instruction.
[Switching to Thread 0xb50c1b40 (LWP 22038)]
0xb2a01475 in ?? ()
(gdb) bt
#0  0xb2a01475 in ?? ()
#1  0xb29fa1c4 in ?? ()
#2  0xb29ff088 in ?? ()
#3  0xb7dd55b4 in llint_entry () from /usr/lib/i386-linux-gnu/libjavascriptcoregtk-4.0.so.18
#4  0xb2a018bf in ?? ()
#5  0xb2ae6b62 in ?? ()
#6  0xb2aebe2b in ?? ()
#7  0xb7dd55b4 in llint_entry () from /usr/lib/i386-linux-gnu/libjavascriptcoregtk-4.0.so.18
#8  0xb7dd5601 in llint_entry () from /usr/lib/i386-linux-gnu/libjavascriptcoregtk-4.0.so.18
#9  0xb7dd12d8 in vmEntryToJavaScript () from /usr/lib/i386-linux-gnu/libjavascriptcoregtk-4.0.so.18
#10 0xb7b57f53 in JSC::JITCode::execute (this=0xb26c3840, vm=0xb4079000, protoCallFrame=0xb50bb524)
    at /build/webkit2gtk-u7EGwB/webkit2gtk-2.6.2+dfsg1/Source/JavaScriptCore/jit/JITCode.cpp:56
#11 0xb7b37c6c in JSC::Interpreter::execute (this=0xb408a5d0, program=0xb349fb00, callFrame=0xb369fa6c, 
    thisObj=0xb365fb60)
    at /build/webkit2gtk-u7EGwB/webkit2gtk-2.6.2+dfsg1/Source/JavaScriptCore/interpreter/Interpreter.cpp:928
#12 0xb7c701e9 in JSC::evaluate (exec=0xb369fa6c, source=..., thisValue=..., returnedException=0xb50bba90)
    at /build/webkit2gtk-u7EGwB/webkit2gtk-2.6.2+dfsg1/Source/JavaScriptCore/runtime/Completion.cpp:82
#13 0xb78e533f in JSEvaluateScript (ctx=0xb50bcd08, script=0xb307a960, thisObject=0x0, sourceURL=0xb307a970, 
    startingLineNumber=<optimized out>, exception=0xb50bbafc)
    at /build/webkit2gtk-u7EGwB/webkit2gtk-2.6.2+dfsg1/Source/JavaScriptCore/API/JSBase.cpp:66
#14 0x080779c5 in evaluate_script ()
#15 0x080746c7 in function_import_script ()
#16 0xb78e6fc8 in JSC::APICallbackFunction::call<JSC::JSCallbackFunction> (exec=<optimized out>)
    at /build/webkit2gtk-u7EGwB/webkit2gtk-2.6.2+dfsg1/Source/JavaScriptCore/API/APICallbackFunction.h:61
#17 0xb7b74a4b in JSC::handleHostCall (execCallee=execCallee@entry=0xb50bcd08, callee=..., kind=JSC::CodeForCall)
    at /build/webkit2gtk-u7EGwB/webkit2gtk-2.6.2+dfsg1/Source/JavaScriptCore/jit/JITOperations.cpp:666
#18 0xb7b778ef in JSC::linkFor (execCallee=0xb50bcd08, callLinkInfo=0xb327cfc0, kind=JSC::CodeForCall, 
    registers=JSC::RegisterPreservationNotRequired)
---Type <return> to continue, or q <return> to quit---

looks like it is definitely happening in JSC

hmm, there are no shortage of hits which go back to debian: https://www.google.com/search?q=libjavascriptcoregtk+illegal+instruction

this guy traced it to a movsd instruction: https://groups.google.com/forum/#!msg/linux.debian.bugs.dist/qTh7bOqmPsY/6emwKMV5BwAJ

looks like I'm hitting the same thing:

(gdb) display/i $pc
1: x/i $pc
=> 0xb2a01475:	movsd  (%ebx,%ecx,8),%xmm0

ah, damn, the patch which was attached to that message is now 404: https://anonscm.debian.org/cgit/pkg-webkit/webkit.git/tree/debian/patches/disable-jit-nonsse2.patch

looks like this might be it: https://sources.debian.org/patches/webkitgtk/2.4.11-3/

Sweet. Nice work. 🙂