Clojurians Log v2

https://www.rfc-editor.org/rfc/rfc8941.html "Structured Field Values for HTTP" about headers is related and hot off the press

Anyone have recommendations for screencast software and workflows? I'd like to record some simple screencasts with keystrokes displayed, on a linux machine.

Hiccup (not quite) in Rust https://github.com/naomijub/hiccup

Thanks for sharing.

her github is full of awesome repos

I've a question about how people handle enterprise ETL. We get several hundred files from banks and other counterparts every morning. csv, txt, xls, large diversity of internal structures, but all tabular. We parse the content (including maybe header sections), run some validations, do simple transformations and chuck it in a DWH, to be used by downstream applications. I'm sure this is a super common thing, but I'm curious whether people tend to roll-their-own and write software to do this vs. buying in ETL box-and-arrow tools. My concern with the latter is that most of the ones I've seen tend to be very 'data-engineer'-y, with not much support for, I guess I'd call it 'production hardening' - change control over a large number of job definitions etc. Like the way to change the ETL job is some person just goes in to prod and fiddles with the boxes and arrows until it works, which scales poorly. Anyone have any experience with this?

I believe stu halloway gave a talk about ETL with core.async at the conj in austin in 2016. worth watching for how his experience went. but i suppose that would put him in the roll-their-own category. I've never used one of the ETL tools as you're describing though so i can juxtapose them

code here: https://github.com/Datomic/mbrainz-importer readme includes youtube link to talk

I do think if someone were to go down the roll-your-own route that Clojure would have some neat USPs, in terms of declaratively defining validations/transforms as data. Like Malli for spec validations and meander for transform. They get saved as resources and you can edit/version as needed.

I’ve used SSIS in the past and would avoid box-and-arrows for anything going forward. Its difficult to share logic, review logic, etc. I have not used it, but Airflow https://airflow.apache.org/ looks nice if you are open to using python for the glue code

Anyone know why Nightcode https://github.com/oakes/Nightcode was archived?

Thanks! What was it about the box-and-arrows approach that turned you off?

Emailed the maintainer to see if they care to share the backstory for that.

logic is sometimes buried in multiple levels of UI, debugging can be a nightmare and it’s difficult to share logic. I would much rather use data structures and functions to describe pipelines. There is also this but I have not reviewed it in depth http://andrewberls.com/blog/post/introducing-overseer--data-pipeline-management-in-clojure-pt-1

my guess is some people feel like that is the responsible thing to do with projects they don't have time for anymore

Cool, thanks for the insight!