Yeah, we all have.
What I went thru many of you (I hope!) wont…but ever faced that “Lose your PEM file and you’re dead” from AWS?
Been there.
Turns out, it is survivable. AND do you even know what the files mean? I did not.
It happens. Sometimes you lose or trash your PEM file from AWS (or other). And the threats…”gone, you lose”, well, not entirely so.
I *have* hurt or lost my PEM files in the past, but recently had a reverse problem. I hurt my server-side of the PEM/ssh files. What? Yeah, it can happen (and should not have).
I was trying to get my AWS EC2 to export an NFS share. It wasn’t going well, and I’m not a deep devops guy, but know enough, and decides to “chmod 777” more and more things till i got my Mac-client NFS mount of my AWS EC2 to work! HOOOORAY!…NOT.
All was good, until, i tried to ssh back into my instance. “Public key error”. We all know, on the client side, it has to be 700/600 protection, right? And it was. Hasn’t changed. But what they dont tell you, is the other side cares about its side…and if THAT is too open then you fail (and I accidentally) did that.
So, Problem is: HOW do I get a new PEM or resurrect my PEM on an AWS instance???
If you FEEEEAR AWS (as do I) you’d think: Sorry, you’re dead, out of luck. But your instance and volume is good, right? In my case, I could even NFS mount that volume!! But not SSH. Uhg.
So here’s what you do. Your volume is an EBS, right? Must be. So you can visit the AWS volume page and un-assign/unmount it. You should/need to shut it down too. Do that. Then, create a new EC2 (even a cheap one).
And assign the EBS to it. On boot, mount it. (look at Ubuntu/etc docs for such, but easy…she’ll mount on a normal point).
NOW you can go into it, like “cd /myMount/home/ubuntu/.shh”
Look closely, dot-ssh, right?
First, in my case, ls -ld . and be SURE the perms are “drwx” (700) and “authorized_keys” is “-rw” (600). THIS was my problem, but probably not yours.
So, WHAT is “authorized_keys” and your .PEM file and .RSA files??? SO confusing.
So for HIGHLY secure access to a site (say, via ssh or sftp) we sometimes need a pub/private security technique (which is HIGHLY secure). A .PEM file or the equiv RSA files hold those. The file format (rsa vs pem) is really a file-format concern, and not an encryption one. In fact, you can cover from one to the other. The key/core/important-thing is: THE PRIVATE KEY IS KING!
So, your “home” “.shh” is used/interrogated for a variety or reasons, but doesn’t have-to–be. In fact, when I do a ssh or sft, i *always* specify “-i filename” but you dont have to.
On the outbound side, using ssh/sftp, the app has to know your PRIVATE key, hence you need to have a .pem or rsa file etc either in your ~/.shh folder OR using the “-i” flag. Good, so that’s your private key now trying to talk to the remote sever.
(and, per above, that file/folder MUST be secured per the apps expectations or it just wont work).
BUT, there is the other side. It is going to get ping’d with an ssh/sftp/etc and needs to know WHO that is, and if that person is valid. Ah, so, here we learn about “./.ssh/authorized_keys”. This file, text, MUST have a line with each PUBLIC key of the “attacking” client. Or, denied, or, if this “authorized_keys” file isn’t secured or folder as-expected, failure (THIS was my probably, long story)
So, in my case, i mounted my errant volume and changed the foldler protections to be 700/600 as required, re-assigned the volume back to my former EC2 instance, and all was well.
BUT, you might have lost your PEM, AND created a new one (via AWS or command line). All good. All you have to do is let your EC2 instance know about this potentially new client. You need to ADD that new PEM’s public key to your “authorized_keys” file.
Easy, peasy.
Chose your technique, i use: ssh-keygen -f private.pem -y > public.pub
Then take that .pub “TEXT” output, and add it to the end of your server’s authorized_keys file. And the unmount and re-mount that volume back on your first EC2 instance, and ssh using your new .pem file. Life should be good!!!