Clojurians Log v2

has anyone tried out the new foreign linker api?

I'm having difficulties setting a callback function in a struct

and i'm unsure if its because I am getting the address of the callback function wrong (it should be a symbol in the library) or something I am doing wrong with struct padding

There are several query DSLs (GraphQL, EQL, pull syntax, Datalog, meander, SQL, specter, etc) that are fairly generic, flexible, expressive, and independent of datastore (ie. crux, datascript, datomic, in memory). It seems like mutations/updates/transactions should be equally flexible and generic, but I'm having trouble finding many good references. The closest thing I could find is https://github.com/redplanetlabs/specter navigators. Generally, most options for updating or inserting new facts seem less expressive, especially when updating info that refers to data that already exists in the datastore. I've looked at several different options, but compared to guides covering querying, the sections covering mutations/updates/transactions seem like the "then a miracle occurs" part of the documentation: https://docs.datomic.com/cloud/transactions/transaction-processing.html https://netflix.github.io/falcor/doc/DataSource.html#set https://relay.dev/docs/guided-tour/updating-data/graphql-mutations/ https://pathom3.wsscode.com/docs/mutations https://day8.github.io/re-frame/api-builtin-effects/#db https://github.com/mpdairy/posh#transact https://opencrux.com/reference/21.02-1.15.0/transactions.html https://github.com/cljfx/cljfx#event-handling-on-steroids https://book.fulcrologic.com/#Mutations https://edn-query-language.org/eql/1.0.0/specification.html#_mutations Are there any good resources or ideas I'm missing?

Asami seems to have a slightly different take on transactions than some of its datalog peers: https://github.com/threatgrid/asami/wiki/Transactions I'm not familiar with it enough to comment on how it compares in terms of expressivity though

I think that graphql and pathom mutations are more general than updating a datastore, although they are often used for this. they can be used for any kind of side-effecting procedure.

That's true. I most interested in just the datastore update part, but having a story around combining side effects with updates (eg. the coordination between Agent's send and STM transactions) would be neat too. Other than specter, I can't find many resources that have much documentation/examples on how to set, update, and compose changes.

The tricky thing is most of those query dsls are based on the semantics of triple stores, and a triple store is a kind of universal encoding

So all sql tables can be represented sets of triples

Even if the references only pertained to triple stores, that would be helpful

And in that model a transaction would be a set of assertions and retractions of triples

But not all sets like that can be mapped to a sql table (for example)

You can't just retract a single attribute, you have to retract the whole row or nothing

Right. Assertions and retractions seems kind of low level (especially compared to the query side). In theory, it seems like it should be possible to • given a specter path, a query, and a schema • automatically translate that into a transaction that targets any of the available datastores

like you can run datalog against any datastore, or just an in memory datastructure

it should be possible to write an update description that is as expressive as specter's navigators that would update any datastore (it already works on in memory data structures)

You will need to make lots of decisions on things like the above mentioned "what if I retract a triple on a sql store?" Is it an error if you don't retract everything else in the same row too? Or does it automatically imply they all get dropped? Or is a nop?

Are there any options that work just on triple stores?

And as you make decisions about that it makes your approach less universal

Datomic is a triple store

Right, but afaict, the transaction interface is much less expressive than the query interface

Whoops

They just have to bottom out/expand into a set of assertions and retractions

right. Are there any tools/libraries that help with that?

Dunno

I would really not go with spectres navigators as an example

Like, basically a navigator is just a database query

To summarize, query feels very well documented (guides, APIs, examples) and expressive. The options for Updates/transactions seem less well documented and expressive.

what else would you like to express?

But tightly coupled to a concrete data structure, where a relational/tuple store query language is not

This is just off the top of my head, so it's not a great example: Given several todo lists (work, home, hobby), toggle all work todo list items created more than a year ago. (Toggle would be logical negation, even though that use case doesn't make sense. I just wanted an example that updated an attribute using a function rather than setting the value) This is just an example, but I'm looking for something that is composable similar to the way queries are composable.

Data has logical relationships with other data

But when we use data in programs it has physical relationships (as things are connected in memory etc) as well

And spectre as a tool is way to into physical relationships

the logical basis for query languages is first order logic, the academic treatments of datalog use first order logic directly, and sql does its best

the problem is logic is atemporal while update/assignment/etc requires time

so you might look at something like dedalus https://www2.eecs.berkeley.edu/Pubs/TechRpts/2009/EECS-2009-173.html which explicitly models time

I'm not saying specter is the right answer, but what I like about it is that 1. paths can be reused for either query or transformation 2. complex paths can be composed of simpler paths. Nothing about specter is tied to concrete data structures (it could work with dbs as well). It's more similar to lens in that you just need to implement select and transform. It seems like you should be able to reuse datalog, EQL, etc. for transformation and I was hoping to find an example.

but that is already the case

can you provide an example?

a datomic query is the "path"

so you have a transaction function that runs a query for the todo items, then asserts and retracts them

for a lot of the abstractions you listed, they really don't have transactions, which is why their updates suck, you can't do a read and expect things to be consistent with that read when you write

maybe I'm just missing something, but I can't find any examples that reify the transaction description. I found this todomvc (which is kind of old), https://github.com/madvas/todomvc-omnext-datomic-datascript/blob/master/src/cljc/todomvc/queries.cljc#L48

1. It's doesn't even do toggle, it reads and then writes in two separate transactions. 2. It's not composed of simpler pieces

one of the things that makes query languages like datalog, EQL, SQL etc. "worth it" are the fact that you can give them to another process or machine to run. you're essentially moving the compute of the result to where the data lives. datomic and datascript IME run under an assumption that whatever you're modifying can fit in memory, and to just tell them the facts that have changed - and then transaction functions fill in the gap when that's not the case.

yeah, I mean, random code on github is terrible 🙂

maybe some other corners of tech that deal with writing/updating large (i.e. won't fit in memory) data would yield some more ideas?

Do you have an example that you would recommend?

grrr

how do I keep hitting that checkbox

I mean, first thing that comes to mind is SQL 😛

it shows the transaction function takes a db as an arg, and that the result of the function is a list of assertions and retractions

yea, SQL actually does have a way to combine the query with an update. I was assuming EQL, datalog, etc would also

the db that the tx function takes can be queried etc

I know that it's theoretically possible. I was hoping to find some references/guides/anything where someone is already doing that

Dunno, I just know that exists from watching datomic talks, I've never used datomic

We do have some code at work that treats a database row as an atom, it implements IAtom and does a compare and set in the database of the row contents

Which is a similar kind of thing in the small

You want to express something like given some formula that is true for this version of the database this other formula must true in the next version

The first formula is used to query, and find all the triples that make it true, the second formula needs to generate a set of assertions and retractions that would make it true

And then you actually need some way to talk about different versions of a database (transactions)

There's a few different types of updates with increasing complexity 1. updating a specific attribute 2. updating multiple attributes on a single entity 3. updating an entity's attribute and the attribute of another entity related through a foreign key attribute 4. Updating several entities related that may be linked through attributes. The first 2 are fairly straightforward. Starting with 3), it gets more interesting

> And then you actually need some way to talk about different versions of a database (transactions) Most databases let you just put everything in a single transaction, right?

I think the insight is that the language needed to specify the kinds of updates you'd want to do is close to the level of clojure.core

All of those updates boil down to the same thing

and at that point, why not ship your database with clojure.core? hence, transaction functions

Assertions and retractions of triples

It depends, but I believe most of those query dsls things can run against anything

I agree. Are there examples of using EQL, GraphQL, datomic, datalog where the transactions updates are constructed of simpler pieces?

EQL and graphQL are not concerned with updating a store

their "mutations" are essentially function calls

ok, for applications that use those query languages, do they compose their transactions of simple pieces?

no

not from the client perspective, anyway

I dunno, maybe ask in #datomic

will do

IME most graphQL and EQL services run some procedure on the backend that executes some SQL, or calls another service that updates some store

Yeah graphql very much does not have transactions

Mutations are the wild west

which makes sense on the whole. a lot of systems are built in a CQRS style and so updating a store might look nothing like querying it beyond the facade of a service call

If queries can be generic, flexible, and composable, is it the wrong question to ask if updates/requests/mutations can be structured to be generic, flexible, and composable?

I think that we generally accept more restrictions on how we express our reads than how we want to update something

I dunno, the whole thing is fraught because many of those tools are not just pass throughs, they can transform and merge data from multiple sources

It is hard enough doing transactional updates against a single store

We used to create REST apis for querying that were bespoke with the same justification. It seems at least plausible that specifying updates could have more structure.

like, EQL is strictly less expressive or powerful than datalog. but it meets a lot of use cases, and that lack of power is useful in many cases

if I were using a language that allowed me to declaratively express updates, does the costs in power outweigh the benefits?

I think it could be possible with a good transactional relational store

But almost all of your links are not one of those

They are middleware for presenting a particular style of query api

I'm sure there's a better approach than starting from scratch and building a custom solution every time. Are there any good examples for handling the update side that let you start with more than low level mutations?

Graphql isn't even relational, and has some many odd bits, I am not even sure how you could infer how to call mutations to update data returned by a query

GraphQL gives you a result. Assuming you have a schema, the original query, and can state what changes you would like, it should be possible to generate a transaction

With something like datalog, a query (a formula) can be used both to search a database for a set of facts that make it true, or (this you see mostly in prolog or core.logic) it can be used to generate the set of facts that make it true

Nah, graphql doesn't work like that

A mutation is basically an arbitrarily named rpc

Right. I'm not saying it helps you do that. I'm saying it would be possible to write a library that allows that

So an update can in theory be a pair of formula (queries),

One query to find facts that match another used to generate changes to be asserted

I think so, you would also need either an update function or a new value. I think you might also need a schema to make sure you include unique ids.

@smith.adriane how would you specify an update like, "For each user, capitalize the first letter of every part of a name except <blacklist words>"

In theory no, because the bindings from first query should fill in any unknowns in the second

I'm imagining some syntax like: (update user-name-query capitalize-parts)

how would the remote server know what capitalize-parts is?

what remote server?

I guess that's not a req. then I'll ask slightly differently, how is capitalize-parts implemented?

regular clojure code

#(->> % split (map capitalize-if-not-blacklisted) join)

ah. i thought you were after something more declarative / composable

it's really being able to specify which entities/attributes that I was looking for composability

it wold be something like (update <query> (fn [<bindings from query that includes names>] <some-new-query-that includes the names as capitalize>))

I don't understand the new-query part. How does that work?

a query in datalog is a statement is first order logic

wouldn't the function just return the names capitalized

the result of the query is the set of data that makes that formula true

but you can reverse that and say, from this statement in first order logic, generate the data that makes it true

so the first query is run "forwards" as a query to find the data you want to make changes to, the second query is run "backwards" to generate the changes you need to make

I guess I was thinking more like:

(update user-name-query (map capitalize-parts))

maybe, it depends a lot on the structure of your results

I think you also need some way of validating that the result of the query matches whatever the update fn expects

a query is self validating

an update is destructive. invalid updates could trash your data

e.g. if the result doesn't match it is a bug in your query, so fix it

same with update-in

sure, I'm talking more operationally

I guess I'm stuck in this thinking that, most of these query langs (EQL, GraphQL, datomic datalog) exist in systems that have a server-client relationship. often, GraphQL and EQL are exposed to the internet. so there's some adversarial thinking that goes into the design as well.

maybe phronmophobic doesn't need that in his design

almost none of these things is strictly a first order logic langauge, so you may only be able to support subsets of them

I don't need that in my design

yeah, I think the question is flawed, in the way it conflates things that are not at all alike (databases and api middleware layers)

for api middleware layers, the fact that mutations are only exposed in an adhoc way is almost a feature

the other consideration in EQL and GraphQL are that often they are often layers in front of many different stores, and it takes special care to ensure the relative correctness of what is basically a distributed transaction 😅 (which is basically what hiredman is saying)

I agree that receiving updates from an untrusted client requires more adversarial thinking. Building mutations from scratch doesn't seem like a bonus

It would be great to have a solution that is as flexible and generic as EQL, datalog, etc. I would settle for a solution that works locally with a trusted client with datascript

like what if the way that your system reads data is from a SQL store, but the way you write is by putting a bunch of messages in kafka?

then I'm in agreement that the documentation for datascript is woefully lacking 😛

If there's an example for any of crux, autonormal, datomic,etc, that would be a good start as well.

you could probably do something like this in datascript:

(defn transact-q! [conn q f]
  (swap! conn (fn [db] (ds/db-with db (f (ds/q q db)))))

datascript also has transaction functions

where q is some datalog and f is some function that takes the result of the query and returns tx-data

ah I tried googling if datascript had arbitrary tx fns and couldn't find it

yea, that's more or less the plan. I was hoping there would be some prior art that I could cheat off of

autonormal you could do basically the same. autonormal doesn't have the notion of a conn, so you can handle the transactional guarantees however you like

but yeah, pathom and GraphQL aren't really meant to be fronts to a single store. autonormal is more of a utility library for reading, it doesn't deal with the whole time thing that hiredman was talking about

what I like about pathom is that it focuses on just specifying what data you need rather than how to get it. I still think it's possible to use something that's pathom like to say what changes you want without specifying how to run them.

in addition to queries being generic and composable

I don't need a generic way to handle multiple data stores, but I do plan on implementing a way to have some state in memory with an option to have some state stored more durably.

Interesting convo. I don't have much to add but I did think of another example of a query/transform library that might be useful as another point of reference: Odin https://github.com/halgari/odin#transforming-data