Clojurians Log v2

Onyx being open source was a huge win for me because I could understand what was going on and fix things, which made it an easier sell to clients. Are there any plans to open source Pyrostore?

First time I see this word “hyperpartitioning”. Very curious about how it is done in pyrostore (which I haven’t really followed - I though it was “just” a hosted version of onyx with a sleek cljs UI?). In your examples, you show a geographic hyper partitioner - am I right in saying that this is basically the same as reading all records from all parts and creating another queue (ie. this hyper partitioning does not require a database to work)? What are the advantages of a geohash in this case? Like, if you create a consumer listening to “9q9jh” ( South Bay, SF in your post), can you also listen for “9q9" (removed last two chars)?

@nha thats how i interpreted it as well. basically it’s creating another dimension on top of existing topics, hence the “hyper”

@jcf Not at the moment.

@nha you can assign as many hyperpartitions to the consumer as you want, so yes you can do that. We don’t currently have a way to assign a glob pattern for hyperpartitions, but we do have a method to list all the hyperpartitions, so filtering the list yourself before assigning would be easy.

Beat me to it @lucasbradstreet 😛

@nha with respect to the database question, your hyperpartition consumer would be directly reading the new partitions from S3. You don’t need another database. The data is just laid out in an alternative log to be efficiently read in-order for the hyperpartitions you built.

One interesting point is that you can go from kafka -> pyrostore archived topic, and then build hyperpartitions by reading back the archived topic directly from S3 without putting any more load on your main kafka brokers.

Does pyrostore require Kafka for working?

@richiardiandrea yes, we’re a Kafka add on.

Ok thanks