How to Fix Event 1135 FailoverClustering: Cluster node was removed

The following two errors may show in the CSV node’s Event Viewer System logs:

Event 1135 FailoverClustering
Cluster node 'NODE1' was removed from the active failover cluster membership. The Cluster service on this node may have stopped. This could also be due to the node having lost communication with other active nodes in the failover cluster. Run the Validate a Configuration wizard to check your network configuration. If the condition persists, check for hardware or software errors related to the network adapters on this node. Also check for failures in any other network components to which the node is connected such as hubs, switches, or bridges.

and

Event 1177
The Cluster service is shutting down because quorum was lost. This could be due to the loss of network connectivity between some or all nodes in the cluster, or a failover of the witness disk. 
Run the Validate a Configuration wizard to check your network configuration. If the condition persists, check for hardware or software errors related to the network adapter. Also check for failures in any other network components to which the node is connected such as hubs, switches, or bridges.

It turns out, there are many scenarios that may cause these kinds of errors.

Basically the errors mean the Cluster Management service removed NODE1 from the cluster, thinking it has stopped functioning; however, the cause may actually be:

A. there was a network glitch and connectivity to the node was lost. The delay in communication or packet loss makes the cluster manager believe the node is gone.

B. we have seen these errors occur when data and cluster management traffic aren’t perfectly isolated. The way Ethernet works, the more activity you have on the bus, the more collisions occur, and the actual bandwidth possible decreases rapidly. This in turn causes cluster heartbeats to arrive too late.

You may want to check the infrastructure of the CSV is set up correctly: use separate NICs and network switches for cluster traffic and data traffic. Ideally traffic generated by your Hyper-V backup software should also be isolated.

If data or backup traffic spikes and a node on the CSV disappears, it’s a good indication packets collide and heartbeats don’t arrive on time.

To test it’s not something else, try copying a very large file from the node to the backup folder. On most SANs there is no file locking; hence, you could simply copy and paste a live VHD file to the backup folder to test if bandwidth and network traffic are an issue. Note, this file will end up being corrupted as it’s actively being written to while the copy command is being processed. Do not use this method to make copies of your VHDs unless the VM is off!

Another very effective way to test your cluster is to use BackupChain’s Speed tab and set a low speed limit for read and write operations, such as 10,000 KB/s. If running a Hyper-V backup works with speed limits but not without, chances are there is an issue with the CSV setup and management traffic of the cluster collides with backup and other data traffic. In that case, you’ll need to review the CSV infrastructure and change the hardware configuration accordingly.

More about BackupChain’s Features

Whether you need a Hyper V Cluster Shared Volume backup software, a pro-level backup utility, or the entire feature set of a must-have best backup solution, BackupChain is the clear winner.
BackupChain bundles live backup, VM backup, HyperV cloud backup, Server 2012 backup, and much more in one single solution that is flexible and configurable to match your unique needs. Backup with versioning, for example, are standard and use in-file deltas, also known as file-level deduplication. Deduplication is a great space saver for Hyper-V CSV backups, but also for VirtualBox backup, VMware backup, and over FTP as well. By the way, BackupChain is the only FTP backup software offering incremental and differential backups over the plain FTP.
Beyond conventional file backup, EDB backup for Microsoft Exchange, SQL backups, and DIY remote backup capabilities, there are also major granular backup and recovery features in BackupChain: Granular recovery restores folders and objects inside VM images, while granular backups permit access to the virtual machine file system for backup purposes when the VM is running live.
Find out why our customers are convinced BackupChain is the best backup software for their needs and download your trial today! Don’t forget to check out our freeware for Windows section and DriveMaker, a free tool to map FTP to a drive letter.