So by now you have likely heard of the April 20th anti-virus signatures released by McAfee which had negative effects on various systems (mostly it seems Windows XP SP3). More details here
Judging from the responses from administrators, etc. this seemed to have caused a number of outages of various levels (including at least one 911 call center documented in an article on isc.sans.org).
This is not the first time that automatic updates of software (either patches, anti-virus updates, etc.) have caused unexpected results on systems. This should be a reminder to all who are responsible for the maintenance of systems or software that you still can not trust that software from a vendor will always work on all of the environments that you are responsible for.
In previous engagements I've had to respond to outages caused by patches or updates, and after going though this a couple times it was decided that the solution was to centralize the deployment of patches/Anti-virus updates and to have at least a high level testing process to verify that the update works (is deployed correctly), and that there are no obvious stability issues at the OS level due to the patch/update.
This centralized deployment and testing has saved me a number of headaches and I strongly encourage this approach, especially if the environment you are responsible for is one that has a low tolerance for outages. Another benefit of centralizing the anti-virus deployments is that you can more accurately be aware of systems that are not being updated (i.e. you know how many systems will be vulnerable in the event you have to push out an emergency Anti-virus update).
->Pierre
Judging from the responses from administrators, etc. this seemed to have caused a number of outages of various levels (including at least one 911 call center documented in an article on isc.sans.org).
This is not the first time that automatic updates of software (either patches, anti-virus updates, etc.) have caused unexpected results on systems. This should be a reminder to all who are responsible for the maintenance of systems or software that you still can not trust that software from a vendor will always work on all of the environments that you are responsible for.
In previous engagements I've had to respond to outages caused by patches or updates, and after going though this a couple times it was decided that the solution was to centralize the deployment of patches/Anti-virus updates and to have at least a high level testing process to verify that the update works (is deployed correctly), and that there are no obvious stability issues at the OS level due to the patch/update.
This centralized deployment and testing has saved me a number of headaches and I strongly encourage this approach, especially if the environment you are responsible for is one that has a low tolerance for outages. Another benefit of centralizing the anti-virus deployments is that you can more accurately be aware of systems that are not being updated (i.e. you know how many systems will be vulnerable in the event you have to push out an emergency Anti-virus update).
->Pierre
Comments