Clojurians Log v2
clojure-uk
morning
think i was the opposite, quite liked ruby, came to resent rails
either way i just mean in terms of 'becomes the one way to do things, then becomes obsolete'
but i spose that happens to everything
Good Morning!
Morning
'Mornin all
Bore da :welsh_flag:
…a bore da hefyd! :flag-wales:
måning
mogge
Hello everyone 🙂
Mornin
anyone had a good or bad experience with aws athena ?
We had a few queries that just broke it (nullpointer exception or something). And then we had to wait for aws support to tell us what was broken so we could stop doing that... But we kinda needed to do that.
@joetague was it getting expensive with plain csv or json, or with parquet ?
i guess i'll try it out and see... i've got a kafka topic with telemetry data - it looks easy enough to dump that to parquet on s3 with kafka-connect, and if that turns out to lead to criminally expensive queries then i'll dump it to CSV and load into redshift
Just had a peek in the S3 bucket, we were coping GA/BigQuery data from GCS -> S3 as json and left it in Standard-IA class
Guesstimate/ballpark most of the files were about 600-800mb in size, they weren't well partitioned so we ended up having to load in a few GB of data a day
both: overall it’s a super useful and easy to use service. Occasionally it has latency issues
as in: queries stay in starting state and AFAIK there is little you can do with it. Happened to me just once
cool, thanks
did you convert your data to parquet before dumping to S3 ?
yes
for other reasons than performance as well, eg handling of multiline strings
if you need it just for performance reasons and csv/json serde works fine for you, there is an option to do the conversion within athena as well
+1 to all the points above.
For our usage it started get expensive as well