iDrac6 Recovery Through TFTP and Serial

The History:
This week I had a Dell PowerEdge R510’s iDrac completely die on me; I attempted repairs with several utilities that Dell gives out on their site and all of them ended with failure. I thought it might have been because I upgrade the iDrac from an old version to the latest, without components like the BIOS or NIC, that the iDrac communicates with, being upgraded as well. After upgrading everything, iDrac still was not working, after a few days of messing with it, I found out through piecing together several sites how to force the iDrac in recovery mode to do a TFTP repair, writing a new image to it.

The symptoms:
The system used the Windows iDrac Updater, which stated the update had competed successfully. I then, remotely, told the system to reboot; it shut down and never came back up. When I physically went to the server, it was at the BIOS start screen stating “Error Communicating with iDrac. Press F1 to continue, or F2 for System Setup.” In restarting the server I found that “System Services” were disabled. Then the system would go through normal boot sequence, but when it tried to communicate with the iDrac it would fail then restart the server. After restarting, it would allow a full boot, but would give that same “Press F1 to continue, or F2 for System Setup” message. Thus the server would not boot without physical intervention at the machine.

This is a Dell PowerEdge R510, I attempted to upgrade the iDrac from 1.3.* to 1.6.5.

The Fix:
We need to get to the iDrac’s serial recovery mode, and then we can recover the system.

  1. Reboot the system, and after the system resets itself for not being able to reach iDrac go into “System Setup”, the F2 key
  2. Hit down until you select “Serial Communication”, enter that menu
  3. Set the following settings:
    • Serial System Setup Settings
    • Serial Communication : On With Console Redirection via COM2
    • Serial Port Address : Serial Device 1=COM1, Serial Device2=COM2
    • External Serial Connector : Serial Device 1
      • This could be Remote Access Device, but that gave me problems (I may have had a bad serial cable)
    • Failsafe Baud Rate : 115200
      • For the 11G servers this is the default baud rate
    • Remote Terminal Type : VT100/VT220
    • Redirect After Boot : Enable
  4. Then rebooted the system. I got Windows to start by manually hitting F1
  5. At this point you need to go to support.dell.com, lookup downloads for your system, then under “Embedded Server Management” there is “iDRAC6 Monolithic Release 1.97” (or whatever version is newest)
  6. There are several versions, for my system I got “iDRAC6_1.97_A00_FW_IMG.exe (50 MB)”
  7. After downloading, running this file will extract “firmimg.d6” and a readme file.
    • The readme has no useful information in it, it just tells you to search for the user guide
  8. The “firmimg.d6” file needs to be placed on a TFTP server that the iDrac can hit
  9. Using Putty in Windows I connected the COM2 at 115200 Baud, this is the iDrac being redirected. Connect to your systems Com2 however you can
    • Note all this is being done on the server and nothing is done on a other machine, I had TFTP running on this Windows system
  10. Hitting enter should show a recovery menu
    • Unfortunately I did not save pictures of the recovery screen, some of the next menu options may not be the exact wording
  11. I had DHCP on the network my iDrac was sitting on so I hit 9 to get a IP address, this can also be set manually
  12. Hit 7 to change the TFTP server IP address
  13. Now hit the option that says “Firmware Upgrade”, this will go to the TFTP server specified, download the firmware, and reinstall all pieces of the iDrac from that file. It takes about 5 minutes.
  14. Keep in mind you are in your OS, for me Windows, while the iDrac and its system upgrades and reboots
  15. After it reboots successfully the recovery console stops getting data, I was next to the server, when the iDrac reboots the fans go to full speed then calm back down. That’s how I was able to tell it restarted
  16. Now you can use the RACADM commands if open manage/iDrac tools are installed, or reboot and you should see “System Services” back online, then you can change the IP of the iDrac like normal

Everything should work now and the world is happy!

Advertisements

22 comments

  1. Thanks you saved my day!
    I ran the update live dvd (om74-firmware-live) to update the firmware on a R710 server.
    everything wend fine, till it startet hanging at the iDRAC6 update (while the fans where on full speed) waited for +3 hours. Than restarted the server.
    The Server showed the same problem you described better jet your solution worked!
    I didn’t have a OS on the box, so i used the live dvd to access the serial console (“screen /dev/ttyS1 115200”). It had some issues displaying all of the serial output correctly but thanks to you i knew that 9, 7 + ip and 10 would do the trick. In case some of the characters that are send are not display ignore that, and assume they are there.

    Thanks again!!

    1. hey, i copied the recovery menu,
      With the path i took:

      ==== Firmware Recovery Option ====

      1. Bootloader Version ……. 1.13.7
      2. Ethernet Adapter ……… AMEA
      3. MAC Address ………….. bc:30:5b:e8:2c:1b

      4. IP Address …………… 192.168.0.120
      5. Subnet Mask ………….. 255.255.255.0
      6. Gateway IP Address ……. 192.168.0.1

      7. TFTP Server IP Address … 192.168.0.100
      8. Image File Name ………. firmimg.d6

      9. Enable DHCP Client
      10. Execute Firmware Upgrade
      11. Reset

      12. Bypass Firmware Recovery
      – This option reverts serial device 2 to be connected
      to the external serial connector.

      ECOVER> 9
      Wait for auto-negotiation complete … OK 100MB – Full Duplex
      BOOTP broadcast 1
      *** Unhandled DHCP Option in OFFER/ACK: 28
      *** Unhandled DHCP Option in OFFER/ACK: 28
      DHCP client bound to address 192.168.xx.yy

      RECOVER> 7

      Please enter TFTP Server IP Address ==> 192.168.xx.zz

      RECOVER> 10

      Downloading image from TFTP server …

      Using AMEA device
      TFTP from server 192.168.xx.zz; our IP address is 192.168.xx.yy
      Filename ‘firmimg.d6’.
      Load address: 0x1000000
      Loading: 52 MB
      Bytes transferred = 55364664 (34ccc38 hex)

      Checking and flashing N-1 image …

      Checking partition of N-1 image … Done!

      Checking image header CRC … OK
      Checking platform ID …….. OK
      Checking kernel image CRC … OK
      Checking rootfs image CRC … OK
      Checking u-boot image CRC … OK
      Skipping u-boot update …… YES
      Copying kernel image …….. OK
      Copying rootfs ………….. OK
      Done!

      Syncing up persistent storage …

      Checking and flashing N image …

      Checking partition of N image … Done!

      Checking image header CRC … OK
      Checking platform ID …….. OK
      Checking kernel image CRC … OK
      Checking rootfs image CRC … OK
      Checking u-boot image CRC … OK
      Skipping u-boot update …… YES
      Copying kernel image …….. OK
      Copying rootfs ………….. OK
      Done!

      Erasing private storage …

      Un-Protected 12 sectors

      Erasing Bank 0: 100%
      Erased 12 sectors
      Protected 12 sectors

      Erasing bootloader environment …

      Un-Protected 1 sectors

      Erasing Bank 0: 100%
      Erased 1 sectors

      Writing Bank 0: 100%
      Protected 1 sectors

      Hit any key to stop reset: 0

      ÿ
      ——
      Then the menu stops. and after about 2 Minutes the fan speed goes down.

  2. Hi Guys,

    After 2 day’s (many long reboots) trying to fix this nasty issue, I found this solution and it works! Thanks a lot.
    All other solutions failed. On a T610 I had ESX 5.5 installed. To fix it, I installed a windows server 2008 evaluation to see if the windows solutions worked. There are quit some people having a issue with drac with no working solution. Installing the TFTP server and putty and simply let DHCP assign a IP and and use my own IP as TFTP server, the process was completed within minutes. It’s funny that even Dell does not mention such a great solution.

    Thanks a lot for sharing

    Best Regards

    Henk

  3. I’m so glad I found this guide, however there is a little issue.
    After setting the Serial config in the BIOS, as shown here: https://buildingtents.files.wordpress.com/2014/04/screen-shot-2014-04-24-at-12-21-03-pm.png?w=620&h=124

    The COMS2 port dissapears from Windows Server 2008 upon booting up.
    COMS2 shows up again when It’s set to “On Without Console Redirection”.

    I’ve tried connecting via The External (Serial port 1), and Virtual Serial port in Windows, still nothing.

    I’m going to try around with the Serial settings in the BIOS to get it to show up.

    1. Did the serial settings give you any luck? I was thinking you could go into Device Manager on Windows and see what serial ports it is giving you. Perhaps for some reason it is creating COM3 or something similar?

      1. Well, as it turns out, Even tho I can’t see COM2, apparently It’s being used!

        So, that made me think:
        Maybe It’s hidden.

        And sure enough, It is.

        Unfortunately, It thinks It’s unplugged, so I think iDrac6 has kicked the bucket.

        Looks like I’m going to have to replace the Motherboard in my Dell T610.

  4. Hi, im so glad to find this article.. Everythinks works, But when i start the Firmware Update, the output says *** device is not ready ***

    Any more ideas? Replace Mainboard??

    Output:

    RECOVER> 10

    Downloading image from TFTP server …

    Using LOM0 Gb1 device
    TFTP from server 192.168.25.8; our IP address is 192.168.25.183
    Filename ‘firmimg.d6’.
    Load address: 0x1000000
    Loading: 52 MB
    Bytes transferred = 54574136 (340bc38 hex)

    Checking and flashing image …

    *** device is not ready ***

    Erasing persistent storage …

    Un-Protected 12 sectors

    Erasing Bank 0: 100%
    Erased 12 sectors
    Protected 12 sectors

    Erasing bootloader environment …

    Un-Protected 1 sectors

    Erasing Bank 0: 100%
    Erased 1 sectors

    Writing Bank 0: 100%
    Protected 1 sectors

    Hit any key to stop reset: 0

    1. One thing you could try. Unplug the server, the hold the power button for ten seconds. I know that sounds crazy, and a classic IT thing to say; but I spoke to a dell systems engineer and he told me todo that and it fixed my problems before. By holding the button for ten seconds when the system doesnt have power. It tells the battery powered iDrac and BIOS to flush everything running and clear all settings, then reboot when they get power again.

      1. Hi,
        I had the same problem (“*** device is not ready ***”). Tried to hold power for 10 seconds with AC unplugged, but it got no better.
        I tried to post in the Dell community but nobody has any idea yet.
        Any ideas or hints appreciated.

      2. Unfortunately I dont have any recommendations 😦 That should reset the chip and then the console should allow recovery mode to work. Sorry :/

      3. I replaced the DRAC (the little module with plastic cover that plugs into the motherboard, not the one that has the RJ-45 connector) and it works now. Hope this helps someone googling around for this problem.

  5. Hi,
    I am so glad to find this article after i been toss a few days.

    I have the same problem on R610. I do every step i could done from here, but i can’t enter iDrac recovery mode!!

    What can i do ?

    Here is what i do with:
    1. set the BIOS like this article describe.
    2. using a usb to serial port adapter on a laptop running windows 8.1 and the serial port adapter works fine.
    3. using null modem cable connected R610 COM2(serial, NOT VGA) and laptop’s usb to serial port adapter.
    4. I could see R610 booting up information, but if the server not power on and booting up, i can see nothing and i can’t enter recovery mode.

    Question:
    1. Do i need another PC connect to the Server (R610) in COM2?
    2. what cable should i use for connection of the console redirection?
    3. Is the Server(R610) needed a OS? such as running on Windows or Linux?

    Thanks..!

    1. To answer your questions in order 🙂
      1. No I used the server itself to read the comm port.
      2. No cable its virtual!
      3. I had windows on my server, but Linux should work too, just need to get the Linux idrac package.

      1. Quick Question, after the redirection, how did you connect to coms port 2?
        With Putty in Windows on the Server itself?
        I’ve done everything however it shows up, however it is ‘disconnected’ and so putty simply can not connect to coms port 2.
        Maybe I’m missing something….

      2. Mine was comm 2, you may want to check device manager, it could be a different number for you. I used putty with the settings listed.

      3. THANKS FOR YOUR REPLY.
        I will install a windows Server 2008 on that Server then try again with your way.
        Whether succeed or failed, I will reply your here.

        I better take a rest for now, it’s 5:09am here.
        hehe

      4. Same problem like here others reply, I can see COM2 on my windows server 2008 on that Server.

        There’s even no hidden COM2! I installed OMSA on windows server 2008, it can’t detect iDrac too (No idrac options).

        Is that something I missed?
        Do I have to replace any hard part on that Server ?

      5. I didn’t replace any part. I was able to communicate over com 2 then, make sure you hit enter after connecting to see if it gives you anything? Also you could try unplugging the box, and holding the power button for 10 seconds; then plugging back in. That hard resets idrac.

  6. Hi, I used Dell’s live DVD 7.3 to update the server r610 and got it hanged on DRAC update, after reboot front LCD went dark and it fails to communicate to drac now.
    Whatever I do, there is nothing related to drac recovery from ANY com ports. Been switching those com settings in bios back and forth without success, the only output I can get is booting text, nothing else there! In windows there is NO COM2 etc. This is so frustrating.

  7. Just got it fixed!
    Dell R610. Real pain.

    Set in BIOS:

    Serial Communication: On without Console Redirection
    Serial Port Address: Serial Device1=COM2, Serial Device2=COM1 (NOTE, reversed then in author’s post)
    External Serial Connector: Serial Device 1
    115200

    Find in google a live Windows (any kind). I used some creepy Gandalf’sWin7x86PE.iso and it was good enough. Write to USB or connect ISO directly via KVM (my case).

    Boot to Windows, configure networking, download Putty. If in Device Manager you see Unknown Device, just do right click and click update driver (setup networking upon doing this).
    Note: live iso works in RAM only, all the files or drivers you added will be lost after reboot.

    You should see COM1 and COM2 both unhidden!

    Connect to COM1 via putty, selecting Flow Control: None and Rate: 115200 !!!!
    If port is empty, try COM2. In my case COM1 was the one.

    You’ll see the recovery menu. FINALLY!!!

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s