Clojurians Log v2

When I run a script using the bootleg pod, I now get a NullpointerException with the following error:

----- Error --------------------------------------------------------------------
Type:     java.lang.NullPointerException
Location: ***/static_website.clj:1:1

----- Context ------------------------------------------------------------------
1: (ns static-website.core
   ^--- 
2:  (:require [babashka.pods :as pods]))
3: 
4: (pods/load-pod "bootleg")
5: (require '[pod.retrogradeorbit.bootleg.utils :as utils])
6: 

----- Stack trace --------------------------------------------------------------
pod.retrogradeorbit.net.cgrand.xml - ***/static_website.clj:1:1
static-website.core                - ***/static_website.clj:4:1

An verbose call with the classpath and main flag brought this additional info:

clojure.lang.ExceptionInfo: null
{:type :sci/error, :line 1, :column 1, :message nil, :callstack #object[clojure.lang.Delay 0x4d51040 {:status :ready, :val ((#object[clojure.lang.AFunction$1 0x39a2d783 "clojure.lang.AFunction$1@108981b80"] "bootleg") (#object[clojure.lang.AFunction$1 0x1c52cd9 "clojure.lang.AFunction$1@108019998"] document? "Document nodes are a parsing impelentation details and should never leak\n   outside of it." [x] (= :document (:type x))))}], :file "***/src/static_website/core.clj", :locals {}}
 at sci.impl.utils$rethrow_with_location_of_node.invokeStatic (utils.cljc:104)

As bootleg hadn't released a new version I suppose, bb had made a change here.

Does it work with bootleg when used not as a pod though? I’m not aware of changes to the pod protocol that would break existing pods

Maybe a question for @retrogradeorbit

Hey what version of bb and bootleg is this with. I will try and reproduce.

@danielgrosse is it possible to get a boiled down reproduction example that triggers the NPE?

I run it with latest released version of bb and bootleg. It occurs with the example from the bootleg repo. But I try to setup something simple

bootleg example should be sufficient. Which example?

https://github.com/cgrand/enlive/blob/be075b124eb8c4e00b271e21ad63b3a8de6ecb87/src/net/cgrand/xml.clj#L22

some how x is going in nil here by the looks

I did have issues when podifying with passing metadata over the pod boundary. Turns out clojure's zipper implementations rely on embedded metadata in the data structure for functioning. Thought I had dealt with those issues (by keeping the zipper calls on the babashka side by sending that code over to be evaluated bb side). Maybe not. Will need to test to see.

ok. have reproduced it. Will investigate. https://github.com/retrogradeorbit/bootleg/issues/72

@danielgrosse Found the bug. It's an issue with babashka.pods and bb 0.2.1. For now use 0.2.0 with the bootleg pod

Babashka seems to use nio.FileVisitor and FileVisitResult to build uberjars, so I thought it would be possible to use FileVisitor with bb, but apparently not. Would it be cheap to add these, since they are already used internally?

@danielgrosse Now added bb 0.2.0 to glam:

$ glam install org.babashka/babashka@0.2.0
$ bb --version
babashka v0.2.0

@hugod I guess so. How would the interop look? I'm not sure if the interpreter already can handle that (if reify etc is involved)

can you post a Clojure example?

the uberjar code is representative of what I was trying to do. Without reify though I guess this is a non-starter.

The thing is, reify does work, but only for protocols at the moment

and some Clojure interfaces which create objects that only work with Clojure, not when being passed back to Java

if you want to have compatibility with both JVM and bb, use reader conditionals in that case and rewrite using map, filter, etc

ok, thanks for explaining

at interpretation time we can't create new classes which is a fundamental limitation of graalvm, that's basically the problem. although we could maybe have support for a limited set of interfaces / superclasses. feel free to post an issue at bb and it might be addressed in the future

It’s a testament to bb that you don’t have to think about the graalvm limitations most of the time.

I think we could add support for it like this:

user=> (defn sci-reify [clazz impls] (when (= clazz Object) (reify Object (toString [this] ((get impls :toString) this)))))
#'user/sci-reify
user=> (reify Object (toString [this] "foo"))
#object[user$eval176$reify__177 0x6a0ac48e "foo"]
user=> (sci-reify Object {:toString (fn [_] "foo")})
#object[user$sci_reify$reify__173 0x38704ff0 "foo"]

so then we will make some pre-defined supported interfaces in sci that you can use with reify

https://github.com/borkdude/babashka/issues/600

are protocol extensions a similar problem?

yes, but because they are not passed back to Java I have enough hooks to work around that

so protocol extensions should work? I tried extending http://clojue.java.io/Coercions onto nio.Path but get a nil pointer exception.

they should work if babashka knows about that protocol. so far it only knows about Datafiable, Navigable, and interfaces IAtom, IDeref

could add support for that protocol. issue welcome

Here's the implementation for Datafiable: https://github.com/borkdude/babashka/blob/master/src/babashka/impl/protocols.clj

Protocols are implemented as multimethods in sci (under the hood), since protocols also do compile time things, but multimethods don't

so all functions that deal with these protocolized objects should be patched as well, to check for a sci-protocolized objects and if not, then defer to the old function

major workarounds, but they do work :)

the flexibility the workarounds give seems worth it

usually the fallback is implemented using a :default multimethod. I was lucky that this all kinda worked

@danielgrosse Fix pushed to master. New bb binaries will be available in #babashka_circleci_builds in a few minutes

Also made an issue for this: https://github.com/borkdude/babashka/issues/601

Seems to work! @retrogradeorbit @danielgrosse

$ glam install org.babashka/babashka@0.2.2-SNAPSHOT retrogradeorbit/bootleg --force
$ cat /tmp/test_pod.clj
(ns mybbscript
  (:require [babashka.pods :as pods]))

(pods/load-pod "bootleg")
(require '[pod.retrogradeorbit.bootleg.utils :as utils])

(-> [:div
     [:h1 "Using Bootleg From Babashka"]
     [:p "This is a demo"]]
    (utils/convert-to :html))
$ bb /tmp/test_pod.clj
"<div><h1>Using Bootleg From Babashka</h1><p>This is a demo</p></div>"

Hmm I guess bootleg could also be useful in the context of the new web server

To serve html

i think babashka might just replace python for me lmao

somewhere in there is a web development framework waiting to get out. Just needing someone to tie it together...

Is there a way to do oauth1 auth in bb? is it preferred to shell out? (the use case is the twitter api)

Although it's not a framework ... https://www.mxmmz.nl/blog/building-a-website-with-babashka.html

Interested to try the bootleg pod soon 🤓

Not sure, but there is an http client and http server now, so it should be possible using those maybe?

and if you figure it out, an example gist would be highly appreciated

that’s what I figured, yeah, I’ve never really used oauth before but I’ll keep at it!

I would say bb has all the ingredients with the client and server

@hugod Are you on mac or linux or do you build bb from master?

on mac. using 0.2.1 at the moment, though I have built bb myself as well.

This script now works:

(java.nio.file.Files/walkFileTree
 (.toPath (java.io.File. "."))
 (reify java.nio.file.FileVisitor
   (preVisitDirectory [this p attrs]
     (prn :pre-dir (str p))
     java.nio.file.FileVisitResult/CONTINUE)
   (postVisitDirectory [this p attrs]
     (prn :post-dir (str p))
     java.nio.file.FileVisitResult/CONTINUE)
   (visitFile [this p attrs]
     (prn :file (str p))
     java.nio.file.FileVisitResult/CONTINUE)))

I've yet to write a unit test for it

I don’t know how you respond so quickly to everything! That’s going to be very useful.

well I've already had the idea for a long time and I've got a day off today ;)

Pushed to master. I'll probably release 0.2.2 somewhere today

The next bb will support reifying http://java.io.FileFilter and java.nio.file.FileVisitor. What other classes are useful to reify in Clojure?

@corasaurus-hex When you were working on the fs lib for bb, there was something like this you were running into right?

Hey. Does bb support reify nowadays? I recall getting bitten by http://java.io.FilenameFilter - https://github.com/clj-commons/fs/blob/642e0e6fc734b0855a4579d4518073b2192ac0ef/src/me/raynes/fs.clj#L8

well, as of today yes, in #babashka_circleci_builds ;)

worth a try again with that lib I'd say

Hmm, I guess it's FilenameFilter, and there's FileFilter. I'll add both

pushed to master. fs seems to work now. btw this is the maintained repo: https://github.com/clj-commons/fs

Exciting to hear about reify!

Yes. Note that reify will work only with a selection of pre-defined classes, so if you need more, let me know

Alrighty, pushed 0.2.2 with the fix for bootleg and some nice reify improvements: https://github.com/borkdude/babashka/blob/master/CHANGELOG.md#v022-2020-09-30

$ which bb
/Users/borkdude/.glam/repository/org.babashka/babashka/0.2.2/bb
$ bb --version
babashka v0.2.2
$ export BABASHKA_CLASSPATH=$(clojure -Spath -Sdeps '{:deps {clj-commons/fs {:mvn/version "1.5.2"}}}')
$ time bb -O -e '(ns foo (:require [me.raynes.fs :as fs])) (map str (fs/glob "*.md"))'
"/Users/borkdude/Dropbox/dev/clojure/babashka/CHANGELOG.md"
"/Users/borkdude/Dropbox/dev/clojure/babashka/README.md"
"/Users/borkdude/Dropbox/dev/clojure/babashka/CONTRIBUTING.md"
bb -O -e   0.03s user 0.02s system 92% cpu 0.057 total