Anyone looked at TileDB? https://twitter.com/tiledb/status/1257721692476116992
They supposedly have a Java API. Would be interesting to see how we ll this meshes with our Clojure abstractions.
It is pretty low level which I like for sure but it will take a lot of learning to get anything going: https://docs.tiledb.com/main/quickstart That is their quickstart...Reminds me of TVM in that sense. Harder learning curve up front plus I would have to push some level of context management to users in a lot of cases. I would invest in this before I would invest in Arrow, personally. But I wouldn't invest in either until there was a really solid reason. We are moving towards jdbc-next for http://tech.ml.dataset as the first sort of external integration. Great find, thanks for bringing it up. Will keep it on the radar.
Word; Thanks for your feedback/perspective on it! I'm not as solid on the lower level details, but aside from that I have a pretty similar reaction re: why not postgres (or whatever)? I can imagine they may have optimized for certain use cases, but I'm sure there's a cost to that as well. Definitely worth keeping an eye on.