Background
- Our servers are Server 2016 Version 1607
- Our servers have been created at varying times, some have been up for a few months, others a few years
- We deploy our patches via SCCM
The Issue
During our maintenance windows when we apply the latest and greatest patches to our Windows Server 2016 VMs, we noticed that occasionally, (more often as time goes on), Cumulative Update
(CU) patches end up either stuck, or taking hours to apply. We believe that the reason for these long times are the presence and amount of superseded updates on the server.
So that leaves the question – why are these superseded updates accumulating and not being cleaned up? And how can we clean them up?
Best Answer
TL:DR
In short, the superseded updates were being caused by a Windows Owned scheduled task failing. This scheduled task is responsible for cleaning up superseded updates. To fix this, you're able to run a
Dism.exe
command to manually cleanup these superseded updates. This allows for faster, more reliable patching during maintenance windows. Find more information about these scheduled tasks andDism.exe
commands here.Superseded Updates In The Event Log
So firstly lets talk about how we know there are CUs and other updates that are being cleaned up and delaying patch installation. For people new to investigating patching on Windows Server, there is a Windows Event Viewer log called
Setup
which has events related to patch installation. In this event viewer log, we can see an event denoting that a KB was marked as superseded and is to be removed. Below is what such an event looks like.These events would always occur right as we are starting patching during our maintenance window. This means that much of our time spent patching is actually spent handling these superseded updates, causing our patches to take MUCH LONGER than we planned for.
So why are these superseded updates lingering and only being cleaned up during patching?
Windows Task Scheduler
StartComponentCleanup
While researching the issue, I came across an article detailing Windows built-in capability to clean up out-of-date components. It turns out that this is exactly what I was looking for. The cleanup that this article discusses is cleaning up old components, this correlates to the removal of the superseded updates.
Clean Up the WinSxS Folder - Task Scheduler
This article gives information about a built-in Task Scheduler task:
Library\Microsoft\Windows\Servicing\StartComponentCleanup
Upon reviewing this task on our servers, I found many instances of this task either failing to run or being stopped before it was able to complete. This I believe was caused due to the time limitations discussed in the excerpt from Microsoft above, and I believe that this task failing on many of our servers is the reason we are experiencing surplus of superseded updates. When this task can't run, superseded updates can't be cleaned up on a regular basis.
Dism.exe
The article previously mentioned then goes on to discuss
Dism.exe
commands that can be used to perform this manual cleanup.Clean Up the WinSxS Folder - Dism.exe
Effectively, you can run the same command that the Scheduled Task was running to clean up out-of-date components, albeit, without the limitations imposed by the Scheduled Task.
I found it beneficial to first run the
Dism.exe
command with theAnalyzeComponentStore
flag. This returns whether or not there are any out-of-date components and whether or not you should run the cleanup command.Dism.exe /online /Cleanup-Image /AnalyzeComponentStore
If you see this in your result:
Component Store Cleanup Recommended : Yes
then you can likely proceed with a component cleanup using the following command. Note that this can take quite some time to run.Dism.exe /online /Cleanup-Image /StartComponentCleanup
And running this command did exactly what I was looking for! Running this command kicked off the process of Windows removing the superseded updates (as seen by the screenshot below). This is important because it gives us the ability to proactively remove superseded updates prior to maintenance windows. This allows our maintenance windows to be more predictable, with less chance of exceeding the window or getting stuck.
Conclusion
In short, the superseded updates were being caused by a Windows Owned scheduled task failing. This scheduled task is responsible for cleaning up superseded updates. To fix this, you're able to run a
Dism.exe
command to manually cleanup these superseded updates. This allows for faster, more reliable patching during maintenance windows.Bonus Info Regarding Patching Windows Server Version 1607
Below is an article talking about the mechanisms used by Microsoft to patch the operating system. It gives information related to how and why the 1607 version of Windows takes longer to patch than others. If your on this version, it may be worth a look.
How to improve Windows cumulative update installation times