Unable To Delete Hyper-V Root Snapshot in Hyper-V Manager

During a build-out for a customer it became necessary to move some virtual machines between a Hyper-V 2012 cluster and a Hyper-V 2012 R2 cluster but when trying to do so, all sorts of nasty errors came cropping up:

Live Migration Error Due To Differencing Disk

Error (12700)
VMM cannot complete the host operation on the host1.contoso.com server because of the error: Virtual machine migration operation for ‘MachineToMove.contoso.com’ failed at migration destination ‘host2.contoso.com’. (Virtual machine ID 1D5042AA-1A93-4635-9F0A-F7C7B0D10BDD)

Failed to access disk ‘C:\ClusterStorage\Volume2\MachineToMove.contoso.com\Windows Server 2012 DC with SP1_disk_1_3F40B5A6-E8DC-4752-873C-D9742C9419F4.avhdx’: ‘The system cannot find the file specified.'(‘0x80070002’).
Unknown error (0x800b)

Error (23753)
The virtual machine or tier load balancer configuration requires an IP pool and there are no appropriate IP pools accessible from the host.

Recommended Action
Select a host with access to an appropriate IP pool and try the operation again.

Live Migration Error Due To Differencing Disk 2

Error (12700)
VMM cannot complete the host operation on the MachineToMove.contoso.com server because of the error: Virtual machine migration operation for ‘MachineToMove.contoso.com’ failed at migration source ‘Host1’. (Virtual machine ID 1D5042AA-1A93-4635-9F0A-F7C7B0D10BDD)

Virtual machine migration for ‘MachineToMove.contoso.com’ failed because configuration data root cannot be changed for a clustered virtual machine. (Virtual machine ID 1D5042AA-1A93-4635-9F0A-F7C7B0D10BDD)
Unknown error (0x8005)

Recommended Action
Resolve the host issue and then try the operation again.

You may notice in the top error that the disk path is pointing to an odd file name. Looking at the settings for the machine in Hyper-V Manager and inspecting the disk, we find:

Live Machine Properties

Lo and behold, it’s a differencing disk. Let’s try removing the snapshot that created it:

Hyper-V Snapshot Missing Delete

And there’s the problem – no delete option!

Let’s look at the snapshot in PowerShell. To do so, open an elevated PowerShell session on a Machine with the Hyper-V PowerShell tools installed and run:

Get-VMSnapshot -VMName MachineToMove.contoso.com -ComputerName host1.contoso.com | fl

Here’s the output for the above VM:

SnapshotType : Recovery
VMId : 1d5042aa-1a93-4635-9f0a-f7c7b0d10bdd
VMName : MachineToMove.contoso.com
State : Off
Key : Microsoft.HyperV.PowerShell.SnapshotObjectKey
IsDeleted : False
ComputerName : host1.contoso.com
Id : 4382dc53-2fdd-476f-91b8-81963c292d24
Name : MachineToMove.contoso.com - Backup - (1/16/2014 - 6:00:19 PM)
Version :
Notes : #CLUSTER-INVARIANT#:{434c76e7-5581-463a-b1b4-71027d39770f}
Generation :
Path : C:\ClusterStorage\Volume2\MachineToMove.contoso.com
CreationTime : 16/01/2014 20:22:24
IsClustered : True
SizeOfSystemFiles : 49254
ParentSnapshotId :
ParentSnapshotName :
MemoryStartup : 8589934592
DynamicMemoryEnabled : False
MemoryMinimum : 536870912
MemoryMaximum : 1099511627776
ProcessorCount : 4
RemoteFxAdapter :
NetworkAdapters : {MachineToMove.contoso.com}
FibreChannelHostBusAdapters : {}
ComPort1 : Microsoft.HyperV.PowerShell.VMComPort
ComPort2 : Microsoft.HyperV.PowerShell.VMComPort
FloppyDrive : Microsoft.HyperV.PowerShell.VMFloppyDiskDrive
DVDDrives : {DVD Drive on IDE controller number 1 at location 0}
HardDrives : {Hard Drive on IDE controller number 0 at location 0, Hard Drive on SCSI controller
number 0 at location 0}
VMIntegrationService : {Time Synchronization, Heartbeat, Key-Value Pair Exchange, Shutdown...}

Time to remove it:

Get-VMSnapshot -VMName MachineToMove.contoso.com -ComputerName host1.contoso.com | Remove-VMSnapshot

You can run this command while the machine is running and if you look in Hyper-V after running this you’ll see that the differencing disk will quickly merge into the parent and then the recovery-point snapshot will be removed. Migrating the VM in this state should go without a hitch.

What caused this?

In this instance, the environment is running HP Data Protector 8.0 which is HP’s incredibly powerful (albeit rather old-looking) backup platform. The environment had been configured to back up the machines in the Hyper-V cluster using the HP StoreVirtual P4000 VSS/VDS Providers along with Application Aware Snapshot Manager. As I understand it, this uses the differencing disks so that incremental backups can be achieved – they’re merged and renewed during each Full backup. This is why you see the word “Backup” in the snapshot name along with the data and time that Data Protector took the backup.

HP Systems Insight Manager Firewall Rules

As part of a deployment of HP Systems Insight Manager (HP SIM) onto Windows Server 2012, it was necessary to allow the required ports through the Windows Firewall to pass the pre-requisites check.

Unfortunately, HP SIM seems to be somewhat annoying and has a list of 52 (yes, you read that right!) ports that need opening. What’s more, you can’t use ranges as this causes the pre-requisites checker to fail. HP’s solution is to disable the firewall (they also tell you to disable UAC – which is similarly mad but that’s another rant) but obviously this isn’t good enough for any half-way secure infrastructure.

The solution is to use netsh to create the required rules. With a bit of magic, here are the commands you need to run to add all 52 rules in one easy copy and paste:

netsh advfirewall firewall add rule name="HP SIM - FTP (TCP 21)" dir=in protocol=tcp localport=21 action=allow
netsh advfirewall firewall add rule name="HP SIM - SSH (TCP 22)" dir=in protocol=tcp localport=22 action=allow
netsh advfirewall firewall add rule name="HP SIM - HP Insight Control Server Deployment (TCP 67)" dir=in protocol=tcp localport=67 action=allow
netsh advfirewall firewall add rule name="HP SIM - HP Insight Control Server Deployment (TCP 68)" dir=in protocol=tcp localport=68 action=allow
netsh advfirewall firewall add rule name="HP SIM - HP Insight Control Server Deployment (TCP 69)" dir=in protocol=tcp localport=69 action=allow
netsh advfirewall firewall add rule name="HP SIM - HTTP (TCP 80)" dir=in protocol=tcp localport=80 action=allow
netsh advfirewall firewall add rule name="HP SIM - SNMP (TCP 161)" dir=in protocol=tcp localport=161 action=allow
netsh advfirewall firewall add rule name="HP SIM - SNMP Trap (TCP 162)" dir=in protocol=tcp localport=162 action=allow
netsh advfirewall firewall add rule name="HP SIM - Web server for HP Systems Insight Manager; Web agent auto-start port (TCP 280)" dir=in protocol=tcp localport=280 action=allow
netsh advfirewall firewall add rule name="HP SIM - HP Insight Control Server Deployment (TCP 401)" dir=in protocol=tcp localport=401 action=allow
netsh advfirewall firewall add rule name="HP SIM - HP Insight Control Server Deployment (TCP 402)" dir=in protocol=tcp localport=402 action=allow
netsh advfirewall firewall add rule name="HP SIM - HP Insight Control Server Deployment (TCP 405)" dir=in protocol=tcp localport=405 action=allow
netsh advfirewall firewall add rule name="HP SIM - HP Insight Control Server Deployment (TCP 406)" dir=in protocol=tcp localport=406 action=allow
netsh advfirewall firewall add rule name="HP SIM - HP Insight Control Server Deployment (TCP 407)" dir=in protocol=tcp localport=407 action=allow
netsh advfirewall firewall add rule name="HP SIM - HP Insight Control Server Deployment (TCP 415)" dir=in protocol=tcp localport=415 action=allow
netsh advfirewall firewall add rule name="HP SIM - Harris Stat Scanner Engine (TCP 443)" dir=in protocol=tcp localport=443 action=allow
netsh advfirewall firewall add rule name="HP SIM - HP Insight Control Server Deployment (TCP 505)" dir=in protocol=tcp localport=505 action=allow
netsh advfirewall firewall add rule name="HP SIM - HP Insight Control server migration (TCP 1080)" dir=in protocol=tcp localport=1080 action=allow
netsh advfirewall firewall add rule name="HP SIM - HP Insight Control virtual machine management Control (TCP 1124)" dir=in protocol=tcp localport=1124 action=allow
netsh advfirewall firewall add rule name="HP SIM - HP Insight Control virtual machine management Agent (TCP 1125)" dir=in protocol=tcp localport=1125 action=allow
netsh advfirewall firewall add rule name="HP SIM - HP Insight Control virtual machine management Agent (TCP 1126)" dir=in protocol=tcp localport=1126 action=allow
netsh advfirewall firewall add rule name="HP SIM - HP Insight Control Server Deployment (TCP 1758)" dir=in protocol=tcp localport=1758 action=allow
netsh advfirewall firewall add rule name="HP SIM - HP Insight Control Server Deployment (TCP 1759)" dir=in protocol=tcp localport=1759 action=allow
netsh advfirewall firewall add rule name="HP SIM - HP Insight Control Server Migration (TCP 1779)" dir=in protocol=tcp localport=1779 action=allow
netsh advfirewall firewall add rule name="HP SIM - HP ProLiant agents (TCP 2301)" dir=in protocol=tcp localport=2301 action=allow
netsh advfirewall firewall add rule name="HP SIM - HP SIM RMI connection (TCP 2367)" dir=in protocol=tcp localport=2367 action=allow
netsh advfirewall firewall add rule name="HP SIM - HP ProLiant agents (TCP 2381)" dir=in protocol=tcp localport=2381 action=allow
netsh advfirewall firewall add rule name="HP SIM - HP Insight Control Server Deployment (TCP 4011)" dir=in protocol=tcp localport=4011 action=allow
netsh advfirewall firewall add rule name="HP SIM - HP Insight Control Server Deployment (TCP 5001)" dir=in protocol=tcp localport=5001 action=allow
netsh advfirewall firewall add rule name="HP SIM - HP Insight Control Server Deployment (TCP 5002)" dir=in protocol=tcp localport=5002 action=allow
netsh advfirewall firewall add rule name="HP SIM - WBEM Service (TCP 5988)" dir=in protocol=tcp localport=5988 action=allow
netsh advfirewall firewall add rule name="HP SIM - WBEM Service (TCP 5989)" dir=in protocol=tcp localport=5989 action=allow
netsh advfirewall firewall add rule name="HP SIM - HP Insight Control Server Deployment (TCP 8080)" dir=in protocol=tcp localport=8080 action=allow
netsh advfirewall firewall add rule name="HP SIM - HP Insight Control Server Deployment (TCP 8081)" dir=in protocol=tcp localport=8081 action=allow
netsh advfirewall firewall add rule name="HP SIM - Matrix Operating Environment (TCP 9143)" dir=in protocol=tcp localport=9143 action=allow
netsh advfirewall firewall add rule name="HP SIM - Matrix Operating Environment (TCP 9617)" dir=in protocol=tcp localport=9617 action=allow
netsh advfirewall firewall add rule name="HP SIM - Matrix Operating Environment (TCP 9618)" dir=in protocol=tcp localport=9618 action=allow
netsh advfirewall firewall add rule name="HP SIM - HP Matrix infrastructure orchestration (TCP 16443)" dir=in protocol=tcp localport=16443 action=allow
netsh advfirewall firewall add rule name="HP SIM - HP Insight Control Virtual Machine Management (TCP 40420)" dir=in protocol=tcp localport=40420 action=allow
netsh advfirewall firewall add rule name="HP SIM - HP ProLiant agents (TCP 49400)" dir=in protocol=tcp localport=49400 action=allow
netsh advfirewall firewall add rule name="HP SIM - HP Systems Insight Manager web server (TCP 50000)" dir=in protocol=tcp localport=50000 action=allow
netsh advfirewall firewall add rule name="HP SIM - HP Systems Insight Manager SOAP (TCP 50001)" dir=in protocol=tcp localport=50001 action=allow
netsh advfirewall firewall add rule name="HP SIM - HP Systems Insight Manager SOAP with client certificate authentication (TCP 50002)" dir=in protocol=tcp localport=50002 action=allow
netsh advfirewall firewall add rule name="HP SIM - HP Systems Insight Manager SOAP (TCP 50003)" dir=in protocol=tcp localport=50003 action=allow
netsh advfirewall firewall add rule name="HP SIM - WBEM event receiver (configurable) (TCP 50004)" dir=in protocol=tcp localport=50004 action=allow
netsh advfirewall firewall add rule name="HP SIM - WBEM Events (TCP 50005)" dir=in protocol=tcp localport=50005 action=allow
netsh advfirewall firewall add rule name="HP SIM - HP Insight Control virtual machine management Web Service (TCP 50010)" dir=in protocol=tcp localport=50010 action=allow
netsh advfirewall firewall add rule name="HP SIM - Matrix Operating Environment (TCP 51001)" dir=in protocol=tcp localport=51001 action=allow
netsh advfirewall firewall add rule name="HP SIM - HP Insight Control server migration (TCP 51124)" dir=in protocol=tcp localport=51124 action=allow
netsh advfirewall firewall add rule name="HP SIM - HP Insight Control server migration (TCP 51125)" dir=in protocol=tcp localport=51125 action=allow
netsh advfirewall firewall add rule name="HP SIM - HP Insight Control server migration (TCP 51126)" dir=in protocol=tcp localport=51126 action=allow
netsh advfirewall firewall add rule name="HP SIM - HP Matrix infrastructure orchestration (TCP 51443)" dir=in protocol=tcp localport=51443 action=allow

Add those and the pre-requisites checker should no longer fail on the firewall step.