I had to re-IP a couple of remote servers last night, as our organization changed the network scheme at the site. In the process, I managed to mistype the network portion of the address - typing ‘10.80.11.173’ instead of ‘10.81.11.187’ - for both the IP and the gateway. That left me with a server that was connected to the network, but unreachable. I logged into the iDRAC and launched the console, but the local credentials stored in our vault weren’t correct, and domain credentials didn’t work because:
- there was no reachable AD controller to authenticate them, AND
- our organization uses a GPO to block cached credentials
In cases like this, I usually use ntpasswd, an excellent (if somewhat dated) utility that can reset local user passwords, add users to the administrators group, and unlock\enable accounts including the built-in administrator account. I downloaded the ISO, mounted it as virtual media, selected it from the UEFI boot sequence (press F11 when prompted) and booted from it. After about 30 seconds, the server responded with a notice that the boot media wasn’t valid. Time for Plan B!
I decided that I’d have to use the utilman.exe trick - basically, replace the utilman.exe file with a copy of cmd.exe so that when I click the accessibility button on the login screen, a command prompt will load.
I connected the iDRAC virtual media to the Windows 2008 R2 (x64).iso on my laptop, then rebooted the server. When the BIOS boot screen appeared, I pressed F11 and selected ‘virtual disc’ from the UEFI boot choices; when prompted to ‘press any key to boot from this CD/DVD’, I did. The Dell server started loading the Windows setup program, which went much faster than I anticipated, considering it’s a 2.9GB image and I was remotely connected to a site several timezones away. I clicked on System Recovery Options from the first GUI screen, which will let me access a command prompt.
DENIED! “This version of System Recovery Options is not compatible with the version of Windows that you are trying to repair. Try using a recovery disk that is compatible with this version of Windows.” Um, I was using the correct recovery disk. A quick search revealed that this message is due to the setup program not being able to find the Windows installation… because drivers for the RAID controller aren’t included on the Windows DVD. Leaving that error message on the screen, I looked for the drivers.
I went to the Dell site and downloaded drivers for the PERC H710 mini. After extracting them, I used powerISO to create an ISO image of the files and directories. Back in the setup GUI, I started a Windows install. NOTE: I did not re-install the OS! After picking the OS version to install (Windows 2008 Enterprise with GUI), I accepted the EULA, clicked Custom Install, then clicked ‘Add Drivers’. At this screen I unmounted the Windows.iso, and mounted my drivers.iso via. the Virtual Media option in iDRAC. I browsed to the virtual DVD, and selected my driver before clicking NEXT. Windows was then able to correctly mount/discover the drives attached to the RAID controller. DO NOT CLICK NEXT or you’ll start installing overtop of your existing installation. Use the blue back arrow in the GUI to go back a few screens (just to be safe), then click the X in the upper right to close the setup program. At this point, I was back at the first screen, when I could choose to ‘Repair your Computer’. This time, Windows presented me with options to repair, as it could find my Windows installation. I picked ‘Use Recovery Tools….’, then clicked through until I could load ‘Command Prompt’ to replace utilman with cmd.exe
X:\Sources> g:
G:\> cd windows\system32
G:\Windows\System32> copy utilman.exe utilman.exe.bak
1 file(s) copied.
G:\Windows\System32> copy cmd.exe utilman.exe
1 file(s) copied.
At this point, I rebooted, unmounted the Virtual Media and let the server come up normally. At the login screen, I clicked on the Accessibility icon 
, and a command prompt loaded. Success!
Now, I needed to give myself administrator access. I decided to create a new user and add it to the Administrators group, just in case the local administrator account is used by a process - I don’t want to reset the password and cause an outage!
C:\Windows\System32> net user /add greg *
The command completed successfully.
C:\Windows\System32> net localgroup administrators greg /add
The command completed successfully.
I logged in with my new account and fixed the IP typo. After the server started responding via. ping, I logged in via. Remote Desktop and a domain account, then deleted my temporary administrative account ‘greg’.
References
A site I visited when trying to figure this out was:
This version of System Recovery Options is not compatible with this version of windows