Clojurians Log v2
biff
I am interested in whether Biff might fit my use-case. I am building a Tuple Spaces service. If you’re not familiar essentially a tuple is an ordered list of values (e.g. [1 2 3 4] or [42 “What is six times seven?”]) and a tuple space supports operations such as out(…) to place a tuple into the space, read/take( template ) to get tuples where the template defines rules about what can appear in each position of the tuple. At the moment I have implemented a space using next.jdbc/honey SQL. While it works it’s not a very natural fit as and I am making use of joins in a way that I think is problematic. Now I am thinking about how to expose the service and maybe HTTP isn’t a great fit since tuple operations can block waiting for a matching tuple. I just came across Biff and I’m wondering if it plausibly sounds a good fit for this kind of problem?
It sounds like this would be an API for other services/apps to consume, i.e. it wouldn't be a user-facing app itself, is that correct? A lot of Biff's features (e.g. query subscriptions, authentication, authorization) are specifically for dealing with users and might not be needed. But it does sound like Crux could be a good fit, with web sockets instead of http for communication. So actually, if you did want to experiment at least with Crux + websockets, Biff does have those set up for you. You could use the example app as a template to get started, and then decide what parts you want to keep (if you just want Crux + websockets, might be easiest to copy and paste the relevant bits from Biff's source).
It is offering a space as an API for other apps/services to consume. However there are wrinkles.
1. I want both authenticaion and authorization (for example the ability to create non-expiring tuples would be a privilege in a certain space) 2. Tuple operations like read and take can both be blocking so this isn't a great fit to a regular HTTP API (I am also looking into Server Sent Events) and "template subscriptions" which I suspect are equivalent to query subscriptions might be an interesting idea. 3. Of course I need this at the API layer so you may be right if Biff is providing a web interface to some/all of this.
I'm having a quick look at Crux. I am not very familiar with document databases although I broadly understand the concept.
A document in my context could be [42,"What is six times seven?","See also: Answers to the ultimate question"] or ["=6*7"] or ["Douglas","Adams"] and so on and so forth
Where a query to match it might be ["?integer","?","?string~=See also:"]
I have a feeling that document databases tend to be optimised for JSON and index field: value type stuff where in my case there are no fields which might mean a whole database scan for a query.
hm, yeah. That could be difficult. Here's the best I can think of at the moment if you were to use Crux.
You could use separate attributes for each tuple value, so [42, "foo", "bar"] could be modeled as {:crux.db/id #uuid "...", :0 42, :1 "foo", :2 "bar"}. Then a simple match like [42, "?", "?"] could be queried for with {:find '[doc] :where '[[doc :0 42]]} .
To ensure that you only get tuples with three elements, you could do {:find '[doc] :where '[[doc :0 42] [doc :1] [doc :2] (not [doc :3])]}
To query for "?integer", you could add type info, e.g. model the document like {:crux.db/id #uuid "...", :0 42, :type/0 :integer, :1 "foo", :type/1 :string, :2 "bar", :type/2 :string} . Then a query of ["?integer"] would be {:find '[doc] :where '[[doc :type/0 :integer] (not [doc :1])]}
Querying for "?string~=See also:" would require a scan over all the tuples with string values in the appropriate positions. If this needs to be indexed, the options I see are:
• ask on #crux about how hard it would be to add an index for text search
• use datomic (https://docs.datomic.com/on-prem/query.html#fulltext)
• roll your own indices
• keep using SQL (if it's workable, might be the easiest option)
In general, I bet the crux people would have a lot better advice about how you might go about implementing this.
The general approach Biff takes to authn/authz and subscriptions might be helpful, though it does sound like it won't match your use case out of the box. Might be worth reading through the docs/source and playing with the example project just for understanding the approach.
@foo thank you for such a well thought through response. It should have occurred to me that I could easily transform my tuples into a form more readily searchable in a doc db. That's good, I will have a think about that.