Firmware Best Practices and Update Schedules

Question

Firmware receives very little attention when it comes to updates. Out of sight, out of mind.

Many devices: RAID controllers, NICs, chipsets, and even hard drives, get some benefit from being updated. Better features, security/bug fixes, etc.

Most SA say, "Whenever it breaks, update the firmware." But this can lead to difficulties down the road; Several times, when contacting Dell about a failed hard drive, I've been asked if my hard drive firmware is up to date. All of my servers use some type of RAID configuration. If I already have a single drive failure, should I even consider trying to upgrading the firmware on the rest of the drives or the RAID controller? I would say no. But Dell seems to have a different view.

• What's a realistic update schedule for system firmware?
• Do you have any best practices to share?

(I am aware that Dell has a nice utility called Server Update Utility, which checks for all new firmware on any Dell server.)

Solution 1

I update firmware in two key instances.

1. When staging up a server.
   • When I just get the server, I'll check the the HP web-site for the date of their latest "Firmware Update CD". If it's new enough, I'll run it against the server before bringing it up to production.
   • When I repurpose a server. Typically, this server is 2-5 years old and probably hasn't had a firmware update in that entire time. Since I'm reformatting it anyway, I'll update all the various firmware on the server.
2. When there is a vendor identified need to do so.
   • Sometimes there are major stability problems identified, like an inability to rebuild a RAID5 array after the wrong kind of failure, or a major performance bug in the TCP-offload engine on the NIC.
   • Sometimes when calling in for support, the support tech will request I update the firmware. I will do so then.

There is a third instance that I didn't list above, because it hasn't happened yet:

• When putting a much newer component into an older server. Sometimes the system BIOS will need updating to handle it.

Solution 2

We use HP SIM (System's Insight Manager) to roll out firmware, we do it by platform - test first, then development, then integration, then reference then finally production - usually about a week or so per platform so we have a 5/6 week release to production window. Seems to work but one thing we NEVER do is roll out firmware at the same time as other updates like drivers/code etc. - saves a lot of finger-pointing.

Solution 3

I might be going against the grain here, but if it's not broken don't fix it. If it's not a security issue then I leave it alone.

I've never had Dell deny service for that reason.

Perhaps you can schedule the utility once a month, then update them with other reboots?

Solution 4

You should treat firmware updates as you treat patches, although these updates are typically harder to deploy so you may want to check the release notes to help deciding if it is worth the effort (but then again, you may be doing this for patches as well). Firmware updates are just as likely to contain a new nasty bug as they are to fix one.

Updating firmware should be done when commissioning (or re-commissioning the hardware) as it's an easy time to do so.

The deployment should roughly follow something along these lines:

1. Test on lab machines
2. Deploy to unimportant systems
3. Wait
4. Deploy to important/production systems

Then again, some things cannot follow this. In particular, it's pretty crucial to very quickly push Microsoft patches to Windows desktops, and testing is difficult to do quickly without significant resources.

Solution 5

I can't speak on Dell, but I was told by folks inside the IBM storage group many years ago that the first levels of their RAID firmware is NEVER the best performer. They shoot for stability first, and then crank up the performance with future rev levels. Unfortunately, few of us would think (or perhaps have the moxie) to go back later and mess with the RAID firmware of healthy system unless there were problems. So, our strategy has been to upgrade the RAID firmware levels every time we set up a new server, assuming the RAID adapter isn't a brand spankin' new model. That way we at least get the best performance levels available at the time. If it is a new mode adapter, we try to make a mental note to go back in a few months and check for updates, but we're not religious about it.

On motherboards, we DO NOT mess with them unless vendor tech support tells us. Our experience over the years has been that unless there is specific problem that needs fixing, the risks vastly outweigh the hard-to-measure benefits.

//spk

View more solutions

Comments

[Joseph Kern]

Tell me more ...

[Joseph Kern]

How often do you check for "security issues" on all your different firmware?

[Joseph Kern]

+1, finger pointing, an all too common occurrence ... it seems like every Tuesday ...

[RainyRat]

+1 for speaking the truth. IF there's no good reason to upgrade (like the vendor no longer supporting your kit unless you update the BIOS/firmware), then why are you doing it?

[Mitch]

We followed the Dell releases, quarterly I think, to schedule the changes to apply the updates. We would apply them to all Dell machines across all of the clients we supported. Perhaps we were lucky but having had very few issues I think it also came down to having a plan and keeping things up-to-date. When we took on new servers one of the first things we did was bring all the goods up to the current level.

[Richard Slater]

Virtually all of our kit is HP, by registering the product I sign up a generic account to "notifications about firmware, drivers and security" issues, if I leave I will re-direct that account to my sucessor.

[Ben Dunlap]

I don't work with Dell Support but I am still stuck on a problem that Intel won't support until I update my BIOS -- only I can't get the dang thing to update and I sometimes wonder if it's because the only available update is too many versions ahead of what my server is running.

[Chopper3]

In principal I agree with you but we have some servers that are significantly faster and more stable now than when bought, purely due to firmware improvements.

[Madhu Cheluvaraju]

Agreed. I guess performance issues could be considered "broken" in this case.

[Joseph Kern]

I think that strikes an excellent balance.

[Ryan Bolger]

For the record, I recently actually had Dell deny a hard drive replacement on a PowerEdge server until I updated the raid controller firmware to the latest version.

[Madhu Cheluvaraju]

Interesting. Did the upgrade fix the issue?