Support Tip: Hyper-V hosts fail and log Event ID 5120 when being backed up

Michael Vargo| Senior Support Escalation Engineer

Some customers are experiencing failures on Windows Server 2012 Hyper-V hosts that utilize Cluster Shared Volumes (CSVs) when backing up virtual machines (VM’s) using System Center 2012 Data Protection Manager (DPM 2012) with Service Pack 1 (SP1).

When this occurs the following error is logged on the Hyper-V hosts.

Log Name: System
Source: Microsoft-Windows-FailoverClustering
Event ID: 5120
Logged:
Details: Cluster Shared Volume 'Volume2' ('ClusterStorage Volume 2') is no longer available on this node because of 'STATUS_CLUSTER_CSV_AUTO_PAUSE_ERROR(c0130021)'. All I/O will temporarily be queued until a path to the volume is reestablished.

You will also see the VM’s pause temporarily, go in to a paused state, shut down or see memory spikes on the Hyper-V hosts. Configuring serialized backups doesn't help the problem.

It has been determined that any backup product using shadow copies results in the same errors, thus this isn't necessarily a DPM 2012 SP1 issue.  The Windows Server 2012 cluster CSV team is investigating this problem.

The Windows team has released a fix to address CSV backup issues and it is available for download.  This will address the known memory leak issue along with some other issues that were discover during testing. You can find the hotfix here:

KB2813630 – Virtual machine enters a paused state or a CSV volume goes offline when you try to create a backup of the virtual machine on a Windows Server 2012-based failover cluster (http://support.microsoft.com/kb/2813630)

The Windows team is also investigating other issues found during testing that not included in this release.  However, they wanted to get this update published since the memory leak issue is fixed and will provide immediate relief.

The following Knowledge Base article lists KB2813630 as well as other updates recommended for Windows Server 2012-based Failover Clusters:

KB2784261 – Recommended hotfixes and updates for Windows Server 2012-based Failover Clusters (http://support.microsoft.com/kb/2784261/EN-US)

To avoid CSV failovers, you may have to make additional changes to the computer after you install the hotfix. For example, you may be experiencing the issue described in this article because of the lack of hardware support for Offloaded Data Transfer (ODX). This causes delays when the operating system queries for the hardware support during I/O requests.
In this situation, disable ODX by changing the FilterSupportedFeaturesMode value to 1 for the storage device that does not support ODX by using the following command on all nodes of the Cluster using Windows PowerShell.

Set-ItemProperty hklm:\system\currentcontrolset\control\filesystem -Name "FilterSupportedFeaturesMode" -Value 1

For more information about how to disable ODX, see the following Microsoft website:

General information about how to deploy ODX : http://technet.microsoft.com/en-us/library/jj200627

Michael Vargo| Senior Support Escalation Engineer | Microsoft GBS Management and Security Division

Get the latest System Center news onFacebookandTwitter:

System Center All Up: http://blogs.technet.com/b/systemcenter/
System Center – Configuration Manager Support Team blog: http://blogs.technet.com/configurationmgr/
System Center – Data Protection Manager Team blog: http://blogs.technet.com/dpm/
System Center – Orchestrator Support Team blog: http://blogs.technet.com/b/orchestrator/
System Center – Operations Manager Team blog: http://blogs.technet.com/momteam/
System Center – Service Manager Team blog: http://blogs.technet.com/b/servicemanager
System Center – Virtual Machine Manager Team blog: http://blogs.technet.com/scvmm

Windows Intune: http://blogs.technet.com/b/windowsintune/
WSUS Support Team blog: http://blogs.technet.com/sus/
The AD RMS blog: http://blogs.technet.com/b/rmssupp/

The Forefront Endpoint Protection blog : http://blogs.technet.com/b/clientsecurity/
The Forefront Identity Manager blog : http://blogs.msdn.com/b/ms-identity-support/
The Forefront TMG blog: http://blogs.technet.com/b/isablog/
The Forefront UAG blog: http://blogs.technet.com/b/edgeaccessblog/