Search My Techie Guy

Monday, August 25, 2014

I deleted my glibc using "rpm -e --nodeps" rhel 6.2 "/lib64/ld-linux-x86-64.so.2: bad ELF interpreter: No such file or directory"

This is by far the lamest thing i have ever done as a linux sys admin, but believe me you; "It happens to the best of us" :-(
someone on the internet equated deleting the glibc module to "surgically removing your kidneys, heart or brain."
And that's true, my system came crushing after deleting glibc!

This is how events unfolded, i was honestly trying to upgrade my glibc module but "yum update" was not helping so i became impatient and decided to remove the current version of glibc and replace it with a higher version (i know the gurus are shaking their heads and saying "how could you even think like that?" But, yes, this uncouth method worked for me when i was upgrading the openssl module, and i thought the glibc module wasn't anyhow different! my fault).

Problem:

so i thought i would do these two little steps to upgrade my glibc from glibc-2.12 to glibc-2.14

# rpm -e --nodeps glibc-2.12-1.47.el6.x86_64
# rpm -ifvh glibc-2.14.1-14.28.3.x86_64.rpm

As soon i finished entering the first command, all the cli commands went black! i.e.

# grep
-bash: /bin/grep: /lib64/ld-linux-x86-64.so.2: bad ELF interpreter: No such file or directory
# vi
-bash: /bin/vi: /lib64/ld-linux-x86-64.so.2: bad ELF interpreter: No such file or directory
# ls
-bash: /bin/ls: /lib64/ld-linux-x86-64.so.2: bad ELF interpreter: No such file or directory
# /usr/bin/yum
-bash: /usr/bin/yum: /usr/bin/python: bad interpreter: No such file or directory

More symptoms:
1. No new remote login sessions can be established
2. Even the console login fails

So i went for a reboot hoping for a miracle to happen but still no luck, it was showing a black screen with some error like "Kernel Panic - not syncing: Attempted to kill init!"
Kernel Panic
Am sure you landed on this page because you are going through a similar problem, so to spare your time, here is how i managed to recover my server (by the way this server had cacti and speedtest services running on it, we are talking about 3 months of hard work about to be put to waste, i was in panic mode but the good news is that i recovered :-)).

Problem Solved:

what do you need?
1. Bootable Linux DVD, preferable the same version as your broken installation.
2. Physical access to your server (sorry if your server is in Iraq and you are in America, you might need a friend in Iraq :-) )
3. Flash disk
4. glibc rpm file, same as the one you just messed up; you can download it from here.

What did i do?
1. Change the boot sequence of your server to boot from CD
2. Insert the linux bootable DVD
3. Restart your server
4. Select "Rescue Installed System"
Rescue installed system
5. Follow that simple procedure
6. Your server installation will be mirrored to /mnt/sysimage/
7. Put the glibc rpm file on a flash disk
8. Insert the flash disk and mount it
#dmesg (to see the name of the device so you can mount it, e.g. sdb1)
#mkdir /mnt/sysimage/media/flashdisk/ (create a mount point in your rescued system image)
#mount -t vfat /dev/sdb1 /mnt/sysimage/media/flashdisk/
#rpm -Uvh --root=/mnt/sysimage --force /mnt/sysimage/media/flashdisk/glibc-2.12-1.47.el6.x86_64.rpm

9. Command (8) should run properly with no serious errors.
10. Confrim that the module was installed properly #rpm -qa | grep glibc 
11. Restart the server #reboot
12. Your server will reboot twice and you should be good to go.
13. Big Relief

No comments: