Clojurians Log v2

anyone here half tempted to ditch clj/gae and use hy/python/gae ?

@qqq have you tried setting up the env like this http://lambda-startup.com/developing-clojure-on-app-engine/ ?

@qqq just curious, is boot-gae unsatisfactory in some way?

@claudiu_03 : Yes, I actually have something setup based on that exact page.

@mobileink : I think boot-gae is pretty damn optimal given the choice of using clojure on jvm

but (1) Hy is really nice, (2) I want server side to be dumb, so I don't need full power of Clojure, (3) Hy/python doesn't require JVM warm start / keeping a server around at all times [if I want to avoid the JVM loadup + clojure class loading penalty]

yeah, startup time is an issue on gae/java, esp. when the clojure runtime is involved.

Hy looks interesting. Hats off to the dev - competing with Rich Hickey takes brass cohones.

myself, i haven't worked with python enuff to have a clue. people at work like it (data scientist types). the other day i mentioned i did dth in clojure and one of them actually said "I don't like all those parentheses." my heart sank.

curious again: why do you want the server side to be dumb? how dumb? statuc pages dumb?

i'm actually leaning in the other direction - move the processing off my mobile and into the cloud.

@mobileink python is really really nice, but clojure is really something else.

I am switching from python to clojure 🙂 (although python still has it's uses for scripting, fast startup time)

@qqq how big of a deal is it for you that we do not have nice clojure wrappers over gae services?

Found this yesturday https://shaunlebron.github.io/parinfer/ exactly like python, focus on indent and it will take care of the parentheses 🙂

@claudiu_03 you can say that again. i just have to figure out how to convey this msg to my pythonista colleagues without frightening them. 😉

it's a hard sell for clojure vs python. The biggest selling point for me was clojurescript. The fact that I can reuse the code between backend & frontend.

@claudiu_03 parinfer looks useful. i'm an emacs guy, have tried several similar packages. in fact, i think getting people to even consider switching tools (editors) may even be a bigger issue than language. i know i would never even think of switching from emacs.😬

yeah, cljs is awesome! core.async too, but that's a little arcane.

nowadays i'm adding boot as a major factor. nothing else even comes close - and I've tried just about everything else. absolutely no comparison. it's the kilker app for clojure.

@mobileink haven't tried boot yet, still to new sticking to lein since most tutorials are with it

https://shaunlebron.github.io/parinfer/#editor-plugins

it's not a editor, just a nice plugin. I use it with neovim

luv lein, cut my clojure teeth on it, but boot is in a different league altogether. the boot channel is very helpful, fwiw.

I feel your pain. I'm tried at my current workplace. We're rebuilding the website, and tried to sell clojurescript for frontend but no luck 😞

think people are just way to attached to the skills they learned (js, webpack, etc..) and don't really want to start from scratch all over again.

i've had some success getting mindshare at work by writing some boot tasklibs (like boot-gae) that look magical. you can do stuff with boot that just ain't gonna happen with other tools, at least not nearly as fast. easy to support java too. pyrhon i'm not so sure about but why not?

i'm actually toying with the idea of a boot tasklib for C projects. having recently suffered thru the ordeal of dealing with Scons (which is python) i think there is a real opportunity there.

sounds really nice, really curious about boot now. Will definitely take a close look after I get my head around om.next 🙂

@mobileink (1) I'm already heavily invested in Hy. I want to use Tensorflow/Numpy, which Java has no match for. Therefore, all my data analysis / machine learning / AI code is written in Hy. This also means almost all of my Google Compute Engine (for heavy data science work) is all written in Hy as well. (2) I want "GAE Handler" to be dumb -- as dumb as possible. It should be stateless, it should (a) take a client request, (b) check if the client has permission, and (c) do the right thins to datastore / task queue and (d) return to the client. There should be nothing complicated going on here. (3) Given how heavily I'm already invested in Hy, and how much nicer GAE-Python is than GAE-JVM-Clojure, it seems like a obvious choice. (4) -- If I did not have to already use Hy for data analysis -- I'd stick with pure clojure. But I already have to use Hy, and I like it.

@claudiu_03: one thing to keep in mind: boot is not a build system. it would be more accurate to call it a Job Control Language - that's right, JCL, done right. Old mainframers like me will understand. :simple_smile:

@qqq so your number crunching hy/python is on Compute Engine, and you're thinking about GAE as a front-end of some kind?

front-end is in clojurescript/re-frame -- most of the "smart" logic" is there -- GAE is literrally meant to be a dumbass "security checker" which decides whether the client has the right to update the database in a certain way

I think this is also more secure -- complicated server side code scares me -- I want server side to be as short and simple as possible

i guess you could use clojuescript/node on Google Cloud, but i don't think that's (yet) possible on GAE Standard Env.

I think you're in the mindset of "use clj/cljs everywhere" whereas I'm in the mindset of "I'm already using Hy for all my data anslysi; the incremental cost of using it for gae is very small, especially given how small I expct my gae code to be"

well, sth lispy, anyway. :simple_smile:

what does sth stand for? I see you using it all the time, but I'm also guessing at the word

sorry, sth = something, sb = somebody. comes from lexicography.

I should setup those up as emacs abbrevs

I'm currently naming all my file names in such a way that helm can uniquely identify them within 3 keystrokes

that's my main criteron for "what to name this file/function"

you madman!

or madwoman. qqq doesn't say. 😉

madsth!

madman

i gave up on emacs customization years ago after i realized that was all i was doing.

I'm betting full emacs. I have C-8 = helm auto ocmplete on all the .org files I have (3 key strokes to pick), then when I pick a .org file, it fires up helm-auto complete on the subheadings of the file, thus C-8 three-keystrokes ENTER three-keystrokes = reaches any section of any file I care about

C-8 three-keystrokes ENTER three-keystrokes ENTER sorry, forgot the 2nd enter

i repeat: you madman! luv it.

anyway, sounds like you want an ultra lightweight frontend etc. clojure on gae, maybe not the best choice. sth in aws might be more optimal

(1) gae datastore, with entity groups, and transactions, I think beats Aurora/SQL, DhynamoDB, and SimpleDB (2) gae tooling just seems so much better than AWS. I tried AWS everything seems laggy as heck, gae -- even when i get errors is snappy and prompt with what I'm doing wrong.

problem is i could not possibly keep all that emacs stuff in my head. i have my 15 favorite keystrokes, after thst, stack overflow.

with helm/hydra, you don't have to -- it displays all possible completions and narrows it as you type

re aws, intersting. i've never used it in anger.

I tried setting up aurora, was a huge pain.

gae datastore is awesome. the native java api is daunting. but it's just a big map!

I think it's just "server side datastcript / datomic"

you have entity - attribute - value

then you have indexes over it

it's bascially the datascript model

i've only briefly looked at datascript, but it looks to me like datastore is actually much simpler.

it's almost a perfect fit for clojure. entities are just maps with a distinguished key. so the whole shmear can be presented in pure clj. no "defentity" needed.

but now, how do we query things? to ahve efficient queries, we want indexes so we don't have to linearly scan all entries -- and datascript/datomic/datastore basically use the same model

I deeply suspect (having not done this myself) one can get the same query langauge to work for both datascript & datastore

i'd have to study datascript/datamic some more, but i suspect you're right. one major diif: there is only one datastore, no creation of databases, no "conn", etc. just keys to a map.

I spent sometime working through https://github.com/halgari/odin -- where tim baldridge walks the user through creating a basic logic langauge, then implementing a "datomic light" (basically none of the server / persistence) -- just the data model / index / query language

after that, all three of these sorta look the same

i'm thinking the db id in datascript/datomic would just map to an ancestor key in ds.

a database is just a map under a key, iow.

odin looks interesting. but i do not think we need special db dsls or query languages. everything we need is already there in clojure.

I agree. The argument is not "we need a custom special db dsl" -- the sidepoint was "sometime ago, I studied odin ;; and after doing that, all three of dbs look really similar when I squint."

It was just fasicnating seeing how stuff gets implemented under the hood.

true!

the thing is, the only diff between a clj map (for example) and a map in a db is a couple of operations: save, and fetch.

in migae.datastore i tranlate this into 2 ctors: entity-map! and entity-map*. they're both ctors even tho we may not normally think of them like that. the "!" ctor constructs and saves. the "*" fetches and constructs.

think of a query as a constructor.

or a co-constructor.

iow, operations like "get" and "save" are bogus.

they are implementation details leaking into the language.

or look at it like this: (let [m {:foo 1].. ) involves an implicit "save" - save to memory. we don't make that explicit here, why should we need to make it explicit if we're saving to a db instead of memory?

you got me going, you madman.

sorry, slack is distracting, was afk coding

reading msgs now

no prob, i actually called off the coding a few hrs ago when i got clojure/alexa/gae working. i am now enjoying a beverage at the corner bar. hey, it's sunday here!

hang on, let me read through https://github.com/migae/datastore/search?p=2&q=defn+entity-map%5C%21&type=Code&utf8=%E2%9C%93 a bit to see how this entity-map magic works

https://github.com/migae/datastore/blob/5df46f7ef6e1116edbc422a4b0780c9d9cc6ded7/src/clj/migae/datastore/structure/vector.clj#L87 is that the right function or wrong function?

hmm, looks like you ahve a entity-map! for each type

what do you mean "for each type". only one entity-map, just like hash-map: an implemenation of map.

@mobileink : in your docs, "constructor == smaller pieces -> bigger piece", "co-constructor == big piece --> smaller pieces" ?

oh crap, i'm going to have to read my docs.

keyword: https://github.com/migae/datastore/blob/5df46f7ef6e1116edbc422a4b0780c9d9cc6ded7/src/clj/migae/datastore/structure/keyword.clj#L57 vector: https://github.com/migae/datastore/blob/5df46f7ef6e1116edbc422a4b0780c9d9cc6ded7/src/clj/migae/datastore/structure/vector.clj#L87 it appears there are multiple "entity-map!" functions, corresponding to various types (i.e. vector vs keyword)

okay, the examples are clear

I know what you mean by co-constructors, Haskell defines them automatically when you have constructors w/ named fields

think ctor. when you do e.g (def m {:foo 3}) you construct a map, on the surface. but behind the curtain clj ctors a hashmap, or whatever, in memory.

theyr'e sorta like "selectors" or "projections" <-- what you're referring to as co-constructor

and keylinks == namespsaced keywords, keychain == path == list of namespaced keywords

dogtag is last elemen of keychain -- lol

and clj can move that stuff around, your code does not need to know, clj hides that.

question about gae, not boot-migae-datastore -- are datastore "kinds" really the same as "buckets" -- because basically, two objects of the same "kind" don't ahve to be similar at all, so really it's just some bag that we throw stuff into

(def em1 (entity-map! [:Foo] m)) ;; kind="Foo", id is autogenned.
(def em2 (entity-map! [:Foo] m)) ;; em1 and em2 have different key ids
(def em2 (entity-map! [:A/B :C/D :Foo] m)) ;; long keychains ok too

so when you ctor an entity-map, similar, except we unavoidably must expoes the persistence. so we have a "push" ctor that constructs the clj thing but also saves it to ds. the pull ctor is dual: pull the thing from ds, then construct the clj thibgie.

(entity-map* [])        ;; fetch all entities
 (entity-map* [:A/B])    ;; fetch unique entity with key :A/B
 (entity-map* [:A])      ;; fetch all entities with kind :A
 (entity-map* [:A/B :C]) ;; fetch all entities with kind :C and ancestor :A/B
 (entity-map* :pfx [:A/B :C/D]) ;; fetch all entities with keychain prefix (i.e. ancestor) [:A/B :C/D]

auto-enning?

auto genning

auto generating an id

yes, if you give an "improper" keychain (one whose last element is not a namespaced kw) to a ctor it will autogen.

re "kinds" - i think of them as tags. they have no type- or schema-related importance.

i rather like the syntax: (entity-map* <spec>) essentially means "fetch the zero or more ('*') ds entities described by <spec>, and construct the corresponding clj datum (entity-map) for each.

hence "pull ctor".

the multiple entity-map fns are just for convenience.

this is very nice for kv access

but i see there's an error in the last example you cited above.

is there a way to make this support querying?

because with indexes, I can ask questions like "give me all people with last name Turing"

(entity-map* :pfx [:A/B :C/D]) ;; fetch all entities with keychain prefix (i.e. ancestor) [:A/B :C/D]

re: queries: last time i worked on this i had lots of good ideas about query syntax, but i'll have to review to get back up to speed. the critical point: query is a bad concept in FP!

there are no "queries", only expressions and predicates. when's the last time you heard a logician talking about "queries"?!

that's why i say "pull ctor", when i could have said "query". actually "co-ctor" would prolly be better.

what do you mean? I consider prolog/datalog "very functional" lanuages

yet datalog is all about querying, and prolog search is sorta like queries

that's only because people feel comfortable with the concept "query". in logic there is no such thing. it's effectively a narketing slogan. not evil, but also not accurate.

if datalog is not a "query" language, what is datalog?

a "query" is really a predicate, and making/sending a "query" really means "satisfy this predicate, if you can".

i have only briefly looked at datalog, but i suspect it is a kind of logic language.

so that's the point of having push and pull ctors in migae/datastore. no queries, just predicates.

the notion of "query" presumably derives from natural language interactions, and that is prolly why is is so successful. think SQL. but we should not be held back by that.

in (traditional, and most modern) logic, there is no concept of asking a question. all you can do is assert statements.

so if you want to retrieve all ds entities with keychain [:Foo/bar], you don't ask a question, you make a claim: (def x (entity-map* [:Foo/bar]), which states a fact.

that is, "x denotes the list of entity-maps whose keychains are [:Foo/bar]".

it's up to the implementation to make the claim true.

which could mean null.

I don't get it. How do you express something like find me all X where father(x) = abraham, mother(X) = safah?

well, logically (and mathematically), you never say "find all ...". you say sth like "let x denote all ...". so your example just describes a set. there is no "query" there, only a fact: X is in fact the kid whose father is Abe and whose mother is safah. the predicate may be u satisfiable, of course.

sure sure, say I'm writing a tree program, and I want to find all people whose father = Abraham, and Mother = Sarah

this isn't the type of thing I can pull of using just a key path

the point being that all this "get" stuff is an implementation detail

we cannot entirely avoid it, but we can avoid polluting our language with such detail, at least in principle. this is just what clj does by e. g. hiding the implementation of map.

ok, wait, dealing with a tree like that, and dealing with persistent storage are different ideas.

ok, you've motivated me. i need to spend some serious time thinking about how to explain this, because it is a fairly major departure from the usual way of thinking.

re: your example: [:Father/Abraham

crap, try again; do you mean amybody whose dad is Abe AND whose mother is Safah? or dad Abe OR mother Safah?

you can do this in various ways in gae ds, either using keys or fields.