biff

A web framework + self-hosted deployment solution for Clojure. Repo: https://github.com/jacobobryant/biff. Docs: https://biff.findka.com
2020-11-20T13:27:40.125300Z

> How long has it been since you pointed your domain to digitalocean's nameservers? It can take a little while for the change to take effect. it would have been wed night, so about 30 hours, still under the max. > You should be able to ssh into the droplet since it has your public key. is that how you got in? yep. > standalone in prod does standalone mean its using an in memory database? > What output do you get if you run this on the droplet?

root@biff-webserver:~/snap# systemctl stop biff
root@biff-webserver:~/snap# sudo -u biff /home/biff/run-as-biff.sh 
Agent pid 320659
Error loading key "/home/biff/.ssh/id_rsa": invalid format
sed: can't read deps.edn: No such file or directory
so the id_rsa is invalid? If i cat it, it looks like starts with "begin openssh private key". The key in ~/.ssh/authorized keys is the "ssh-public-key" which wasn't what I had stored in github. I went ahead and added that. still get invalid format. However, thats not the public key for deploy-key, so thats likely a mismatch. I added the deploy-key-pub key to ~/.ssh/authorized keys and that still gives me an invalid format. Now i suspect maybe its corrupt? I

2020-11-20T13:30:55.126200Z

if i copy paste my deploy-key (private key) to the server over top what seems to be the same key i dont get invalid format. Though the rest of the errors persist.

2020-11-20T13:42:32.128500Z

Hypothesis: the private key isn't copied sufficiently for the setup in some way thats not detectable. Potentially not consistently reproducible. The other issues is that ./run-as-biff is running as biff which doesn't find the .ssh/id_rsa because biff not root has that. Im running run-as-root and its downloading my github files. And were live: http://urbardner.com/ 🎉

2020-11-20T15:31:32.129200Z

and were down 🙂. I need to have some long lived process that restarts turn the server on. Probably have to re apply/deploy

2020-11-20T19:06:24.130600Z

The deploy-key is used to ssh in and so needs to be copied to the ~/.ssh folder. I just did this and it let me ssh to the droplet (which i lost after i updated my key).

2020-11-20T19:30:25.134300Z

I think two things happened. 1. the deploy-key was misformated on transfer as evidenced by the fact that copy pasting said key manually fixed an invalid format error. 2. The deploy-key needs to be copied a users ~/.ssh folder so it can be used to ssh in. i'll turn that into a ticket if it seems reasonable. Is the latter in any way disputable? The first thing is questionable though. It make take someone else running into it to be sure.

2020-11-20T20:24:16.142400Z

> does standalone mean its using an in memory database? No, it means storing the entire database on the local filesystem. > sed: can't read deps.edn: No such file or directory oops, I should've had you run cd /home/biff before sudo -u biff /home/biff/run-as-biff.sh > The key in ~/.ssh/authorized keys is the "ssh-public-key" which wasn't what I had stored in github. I went ahead and added that. still get invalid format. However, thats not the public key for deploy-key, so thats likely a mismatch. This is expected. authorized_keys contains your personal public key, which allows you to ssh into the droplet. on the server, id_rsa is the private key for the deploy key which is separate, and the public key for the deploy key goes in github (so what you already did is correct). > if i copy paste my deploy-key (private key) to the server over top what seems to be the same key i dont get invalid format. > Hypothesis: the private key isn't copied sufficiently for the setup in some way thats not detectable. Potentially not consistently reproducible. very interesting... I'll keep an eye out for this > And were live: http://urbardner.com/ awesome! it looks like the letsencrypt certificate didn't get set up though. it likely failed earlier, and the way I have it run is brittle. if you rm /home/biff/certbot-ran on the droplet and then systemctl restart biff, it should provision a certificate, then your site will be available on https (and http requests will get redirected to https) > I think two things happened. > ... Agree with your assessment on #1. #2 is unnecessary--as I mentioned, the deploy key is just needed by the droplet to clone your git repo, whereas your personal ssh key is used to ssh in. The only reason you were able to ssh in using the deploy key is because of this: "I added the deploy-key-pub key to ~/.ssh/authorized keys ..."

2020-11-20T20:27:34.144900Z

thanks for your patience in going through this whole process 🙂 I think my main takeaway is that I need to add more error checking/failure case handling during the deploy process. e.g. check the ssh keys, handle failed certificate provisioning, make sure that prerequisite ./task commands have been run, etc. I'm planning to work on this some time next week