Should we be installing Windows Security updates?

Question

I just RDP'd into one of my companies servers, was alerted to windows updates, so I click. Then I see 62 high priority updates, with the last update (according to update history) was installed on Thursday, January 16, 2014, more than one year ago.

What actions need to be taken here?

Solution 1

Short answer - yes. Most of the Windows Updates are security related. Not having the patches means you're vulnerable.

Longer answer - you need a procedure that covers this sort of thing. It's more rare these days, but sometimes a patch can break things, or change behavior in such a way that it's broken as far as your company is concerned. You should be evaluating each patch when it's released (there's a monthly schedule plus some urgent ones), determine if you need the patch (probably yes), do some testing on test/staging servers to do some diligence about potential breakage, and then do the installs.

You should also exercise some care about the deployments, because OS patching often means rebooting, which often means there's service downtime, unless you've got some good HA for all your services. If you think you'll be clever and patch during the day and then postpone the reboot, that's not a great idea - some files will be updated but others won't.

Microsoft offers a free product called WSUS that can make patch management a little easier than doing approvals and deployment all one-by-one.

FYI, you should be doing this sort of thing for all classes of device you have. Network device firmware, server hardware firmware, VMware ESXi, etc. Those patches don't come out for the fun of it, almost all of them address bugs, and many of them can be security related.

Further - you should be asking someone who's more senior than you on your technical team. If you're the only admin there, you and your organization are not doing too well. Don't take that personally, we all need to start without knowing everything we should - but if this is your question, you shouldn't be the only person managing these servers.

Solution 2

The generic answer is it is a good practice to keep your servers updated.

But pay attention to a few things:

1. Updates may cause the server to be sluggish during installation, or even cause some downtime if they require reboot(s). You should plan to do them out of office work hours.

2. Updates have some risk associated. They might break your server, or cause some incompatibility. They are usually fully uninstallable, but with 62 of them you should also consider if you have a trustworthy backup (you should, anyway).

3. Is there a reason why you are one year late on upgrades? Is this your first log in to that server in a year, or is something else broken?

4. Pay special attention to the infamous Excel bug that comes with some December updates of Office, if your company uses Excel macros, but this probably doesn't apply to a server that shouldn't be running Office.

5. Many sysadmins wait a few days or weeks before installing updates, just to see if anything bad comes up on the Internet regarding those updates. When deciding if you need to wait, consider security risks of leaving the server unpatched for more time.

Solution 3

I know mfinni beat me to the punch, but I'm just going to +1 for WSUS. Specifically:

Let's assume that you have multiple servers, including test and production. Let's also assume that test has similar hardware to production (which isn't a safe assumption, I know, but let's go with it--it's nice but not necessary). You could set up the following scenario in WSUS:

1. Test servers in their own OU. Group policy says to install updates and reboot at some non-inconvenient time, like Sunday at 3am.
2. Prod servers in a different OU or OUs. Group policy says to download and notify.
3. Patches approved, and deadlined to install and reboot the servers during your scheduled maintenance window, several days or a week after the test/dev servers apply patches.

What this does, if it's not obvious, is it approves all critical/security patches for your servers, applies them to test first, and then applies them later to production. I've only seen an update critically break something once, but this would give you a chance to roll back the patch if it fails in test before it applies to prod.

As for the big pile of updates on the server in question, patching is a lower risk than not patching, but I would verify my backups before applying them all just in case because there are so many. If it's a VM, you might want to take a snapshot first.

Solution 4

This is entirely up to your business and the policy that you have set out for updating your servers.

At the very least you should install security updates and perform any other patches like .NET framework updates in a testing environment first before updating production servers.

View more solutions

Comments

[kralyk]

Consider yourself fortunate that mfinni and others are actually answering this. It is akin to one of us coming to SO and asking "when I write code should I debug it?"

[HopelessN00b]

@TheCleaner The answer to that question is "after you upsell the customer on your code-debugging services."

[MonkeyZeus]

Before I read any of the answers I just want to say that you should contact the server's admin. If this person is you then brush up on the potential pitfalls of performing updates and proceed with caution. Upper-management will blame you as soon as they put their pen in the wrong drawer and wonder where it went. If it ain't broke...

[MonkeyZeus]

@AndréDaniel "If it ain't broke, don't rush to break it". Hence the entire first three sentences of my comment :)

[user2338816]

"If it ain't broke, don't fix it" and "If it ain't secure, don't secure it" express essentially opposite ideas.

[kralyk]

@Lilienthal - "useful for many other developers" has no bearing on this site. This site isn't designed as a helpdesk for SO users. Call it cruel if you want, I didn't make the site's scope.

[Wesley]

@Lilienthal Beginner and non-expert questions are perfectly acceptable on the network. Each site has it's own rules, and this is below the threshold for participation on ServerFault. ServerFault is not SO and does not have the same rules as SO.

[Wesley]

This question is being voted for closure because the author does not show a level of technical understanding or appropriate due diligence in researching the topic that the community judges as being a minimum barrier to participate.

[Lilienthal]

@TheCleaner Make that short for "developers in charge of maintaining their own servers". For better or worse (likely the latter) the OP apparently has the rights to manage the business systems he's using and as such his question is perfectly on topic. Whether he put in enough effort before turning to the site is an entirely different but valid question. Remember that Topicality ≠ Quality.

[HopelessN00b]

Fast-typing bastard. >:/

[mfinni]

Snow day baby. Trying to get VPN access into the office.

[mfinni]

But the rest of your answer is correct - there should be policy and process for this, that weighs security against service interruption/business need/etc.

[HopelessN00b]

1. Too slow. You got beat to the punch by two other, better answers. 2. There's nothing opinion-based about whether to install patches/security updates or not. The only scenario I can envision where you wouldn't want to install patches would be one where you're stealing from your employer. 3. "Patch management" is most definitely a Server Fault topic, though it might also be topical at super User.

[mfinni]

Lots of things are opinion-based, but that doesn't mean that asking the questions about forming those opinions aren't in the domain of professional sys-adminery. Almost everything is a trade-off, and one's opinion is what matters. If this stuff was cut-and-dry, we'd just need to write one book and we wouldn't need professionals.

[Vasili Syrakis]

If I knew my server admins were asking this on SF I'd be terrified for my infrastructure. The core of the question is "What should I do?" not something akin to "How do I manage/automate/improve?" which would fall under the category of patch management and so on. I thought this place was for professionals, maybe I am wrong about that. Just seems like it belongs on SU to me!

[mfinni]

The asker is clearly fairly junior, because he/she is asking this question. They need help; that's why this site exists. Both other answers are "Yes, here's more details and nuance."

[OpenCoderX]

I'm not managing them, I'm an app developer who happened to need to view some event viewer logs on the host, I've actually noticed update alerts before but this time I missed the little 'x' and clicked the bubble, leading me to the summary page. My dilemma now is what sort of flag do I raise to senior management, because it appears to me that the work is simply not being done. We actually have WSUS. Up until today I just assumed that any update notice I saw would be taken care of that weekend.

[Michael Hampton]

I would be more worried about the server admins who did not ask and did not update for a year.

[OpenCoderX]

I'm not managing them, I'm an app developer who happened to need to view some event viewer logs on the host, I've actually noticed update alerts before but this time I missed the little 'x' and clicked the bubble, leading me to the summary page. My dilemma now is what sort of flag do I raise to senior management, because it appears to me that the work is simply not being done. We actually have WSUS. Up until today I just assumed that any update notice I saw would be taken care of that weekend.

[mfinni]

Talk to management immediately. Do you have systems administrators? If you do, then they might not be doing their job, unless your company policy is "don't install updates." If you don't have systems administrators, get management to hire some or contract it out. As you can probably guess, devs don't have the same goals or skillsets as sysadmins and most can't/shouldn't play both roles.

[kralyk]

"My dilemma now is what sort of flag do I raise to senior management, because it appears to me that the work is simply not being done. " - no dilemma, you tell your boss via email what you noticed and are concerned. There may be a legit reason, or it may simply be laziness. Either way, it isn't your fault it's not been done, but you should at least voice concern.

[OpenCoderX]

I am basically auditing the existing IT manager. This is what I found and before I walk in and say hey, you no do your job, I want to know if there are legit reasons to not have this many updates installed, BTW, since I posted I've hit many more servers, and I find the same situation.

[Vasili Syrakis]

That's a different scenario altogether. As you said, the work might not be getting done, or there could be something preventing the updates from being pushed out from WSUS. Although, whoever is managing WSUS should notice if things are not getting updated.

[Vasili Syrakis]

It's definitely something that should be raised; there have been some fairly critical security updates in the last 12 months.

[MDMoore313]

I agree, however rare is a relative term, relative to $environment. I say this b/c if a shop is running non-MS software (almost every environment), it's more likely a patch could break something than if that environment was pure MS.

[Katherine Villyard]

@shadowadmin I would avoid accusing language and say, "Hey, did you know that there are 62 unapplied updates on [Servername]? I thought WSUS took care of that." It might be some kind of accidental oversight rather than, you know, apathy (although it suggests that whomever runs WSUS doesn't have the cute little reports going to their email).

[Katherine Villyard]

Also, +1 for thinking patch management is on-topic.

[nobody]

What "infamous Excel bug that comes with some December updates of Office" are you talking about?

[Shiv]

"For some users, Form Controls (FM20.dll) are no longer working as expected after installing MS14-082 Microsoft Office Security Updates for December 2014." as per Technet blog post blogs.technet.com/b/the_microsoft_excel_support_team_blog/…

[Shiv]

To add to your short answer, the most readily available stream of attack vectors are the monthly updates themselves. Hackers simply reverse engineer the fixes and produce a new set of attack vectors for unpatched machines every month. Think of the implications for outward facing servers.

[pgr]

@Shiv: thanks, I edited the answer to include your link.

[Pacerier]

@pgr, Aren't there like tons of these infamous bugs?

[pgr]

@Pacerier: eheh, sure. Usually all you have to do is rollback the update. Not this one. Files can get "infected" with the bug, i.e., somebody opens them after the bad update, and suddenly the file stops working on a different computer. It's been a real PITA dealing with this one, and it's not over yet. Notice that the issue has got so complex (for the worst cases, when the problem travels with the file) that Microsoft is STILL working on it, and a definitive solution is still to be achieved... but, of course, each sysadmin will have his own nightmare story, this is mine... :-)

[Pacerier]

@pgr, So how did you manage to solve the nightmare eventually?

[pgr]

The Microsoft links teach the solution (actually install the update everywhere, but clean up the msforms.exd files, recompile VBA where files are broken), it's just a lot of work, and I was able to do it because I had only a handful of computers and files gone crazy. It would be really ugly if I had hundreds, or thousands. And I'm not sure everything is 100% solved, I might get complains about other files later...