Clojurians Log v2
cljdoc
is http://cljdoc.org down for everyone or just me ?
Down for everyone. @martinklepsch?
So the behaviour is this way because it allows building jars locally?
@carkh @seancorfield thanks for the ping, looking into it, seems DigitalOcean had to restart the instance and it didn't start up correctly again
Aaaaand... we're back! π
post-mortem analysis for the curious? π
most likely clj-kondo docs brought the whole thing down =D
luckily I had my local Docker instance running π
lucky you =)
@borkdude DigitalOcean migrated the Droplet to another physical machine and on that machine one of the important services (Consul) failed to start up. I'm not entirely sure why, the logs seem to indicate that the port was already in use which makes me think that maybe something went wrong during the migration Β―\(γ)/Β―
aaah. you use Docker to run cljdoc there?
I do (via http://nomadproject.io)
I see. Glad itβs fixed π
same π took 5 minutes which is great
And I got alerted but was sleeping of course π
Time to get funds together to hire a 24h support team
π
Alternatively, setup pager duty to call you in the middle of the night π
yeah... I think I have better things to do in the middle of the night π
I'd be happy to onboard more maintainers/admins in different timezones though
canβt help there Iβm afraid (NL)
I'm also a little curious as to how often this occurs with DigitalOcean vs. Linode. I've been using Linode for production since 2013, and usually the pattern is that they will give you plenty of head's up if a migration needs to occur because of a potential hardware issue. (Thus giving you a chance to do the migration ahead of time yourself.) Truth be told, I think there may have been a handful (4 or 5) times where they needed to migrate thing "on the spot" because things were dire with hardware. In those cases, even though there was an outage, it resolved itself because the software servers came up. I suppose I'm lucky in that it is just a Clojure / nginx / mysql setup, and they always come back up by themselves.
I don't have enough experience to really comment on that. I think that things didn't boot up again is probably my fault. cljdoc's ops infrastructure is a little more involved to enable continuous deployment without downtime