datascript

Immutable database and Datalog query engine for Clojure, ClojureScript and JS
kiemdoder 2018-05-22T05:55:16.000237Z

@ronb you mentioned that your version of Datascript is not backwards compatible. What are the breaking changes?

2018-05-22T09:39:23.000502Z

@kiemdoder Attributes are stored as Integers (d/datom 0 :attr :val) is not allowed anymore Queries return attributes as Integers, not Keywords You need to define the Attribute order in the schema -> {:attr {:db/order 0}} Transaction Id’s are created from 2^24 -1 downwards New range for Entity ID’s is from 0 to 2^24 New range for valid Transaction ID’s is from 2^24 to (2^24) - 2^20

2018-05-22T09:40:39.000348Z

Every attribute is represented as an ID (similar to Datomic) All attributes need a db/order key in the schema

2018-05-22T09:40:57.000393Z

{:attr {:db/order 0} :attr2 {:db/order 1}}

2018-05-22T09:41:12.000581Z

More Details: https://github.com/tonsky/datascript/pull/263

2018-05-22T09:42:14.000403Z

Will do some benchmarking and better Docs. @tonsky Thanks for the reply on the pull request. I will reply later

2018-05-22T09:43:20.000271Z

I’d say, semantically, biggest setback for existing users would be: — lack of datom call with keyword attribute — unexpected integers in query results (although I believe impl can be changed to return keywords) — the need to specify all attributes in schema (:db/order could be probably calculated, but it requires all attributes to be specified explicitly)

2018-05-22T09:45:29.000343Z

@tonsky I completely agree. Right now the backwards incompatibilities are too big. If I address the issues you mentioned, would you be willing to merge the pull request or do you prefer a seperate project?

2018-05-22T09:47:00.000513Z

most incompatibilities could be addressed with code in db/-search and (transact-add), but changes in how (datom) constructor works would remain and probably be confusing to users

2018-05-22T09:48:14.000055Z

I don’t see how we can work around requiring that every attribute must be specified in schema. So far it’s not required and I believe most users rely on that

2018-05-22T09:48:38.000318Z

And then there’s JS story, which lacks 64-bit numbers so that schema wouldn’t work as well there :(

2018-05-22T09:49:47.000173Z

it feels like jvm-only, separate project with no backwards compatibility guarantees would be best

2018-05-22T09:50:22.000043Z

Yeah it would require additional logic in transaction handling to add new attributes on demand and is a bit messy 😄

2018-05-22T09:51:34.000077Z

i use 64 bit floats to guarantee compatibility with JS environments. All tests run successfully on JVM and JS as far as I can see

2018-05-22T09:52:00.000467Z

No problem. I will close the PR later. Thanks for the feedback

2018-05-22T09:58:48.000415Z

I was under impression JS bit operations only work on 32-bit ints? https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Operators/Bitwise_Operators

2018-05-22T10:01:06.000422Z

You can cheat if you do the following: (/ num (js/Math.pow 2 32)) -> gives access to the upper part of float numbers in the lower 32 bit region

2018-05-22T10:02:18.000042Z

for changing the upper part you can do (+ (* upper (js/Math.pow 2 32)) lower-bits)

2018-05-22T10:03:09.000482Z

isn’t it like SUPER slow?

2018-05-22T10:03:34.000199Z

as far as I can see this is in conformance with IEEE Floating point spec

2018-05-22T10:05:43.000569Z

Will test the performance in the evening, but i compared this with bit-shift operators and they have similar performance. Looks like JS engine optimize (/ num some-binary-exponent) into (bit-shift-right )

2018-05-22T14:03:04.000520Z

^^ maybe helpful?

2018-05-22T14:04:29.000141Z

btw maybe instead of trying to fit everything into one 64-bit value, it might be easier to fit everything in two 32-bit values? memory consumption will be the same

2018-05-22T14:04:51.000842Z

that's what goog.math.Long and therefore ^^ does

2018-05-22T14:05:25.000515Z

No, I mean, there’s no reason to pretend it should look like single 64-bit at any point

2018-05-22T14:05:47.000599Z

depends what you want to do

2018-05-22T14:06:03.000005Z

what @ronb is doing

2018-05-22T14:08:43.000721Z

yeah

2018-05-22T16:35:30.000317Z

@thedavidmeister Looks interesting, thanks for the link. I looked into using goog.math.Long based solution. but if i understand correctly how js engines work, it would use twice the memory of a single double number

2018-05-22T16:36:11.000710Z

although maybe i am mistaken and engines actually detect that you only use 32bits of the number and optimize the rest away

2018-05-22T16:39:31.000447Z

this benchmark compares a bitshift operation with division. and shows a 30% performance advantage for shift based operation. which is not significant enough in my use case to be detectable, as performance is largely memory bound

2018-05-22T20:11:11.000617Z

just tested memory usage of 32bit numbers and values that go beyond 32bits. they both use the same amount of memory