How to customize Windows Failover clustering to trigger on failure of custom window service

Being that it's a proprietary product I don't think you're going to binary-level analysis of the protocol or even flow diagrams, but I think you can glean some helpful information from:

• The Design and Architecture of the Microsoft Cluster Service - This paper, co-authored by one of my CompSci heroes Jim Gray (RIP, Jim), published in the 1998 IEEE Proceedings of FTCS, describes the basic design of the Microsoft Cluster Server product as it was in Windows NT 4.0, which the later products all evolved from. There's even a flow diagram here, though it's probably wildly out of date with respect to the current generations of the product.

• Network failure detection and recovery in a two-node Windows Server 2000 cluster - This article, while dated, does describe the low level details of the state machine used to determine network availability of nodes in a cluster.

• [MS-CMRP]: Failover Cluster: Management API (ClusAPI) Protocol Specification - This document makes statements about how the product works internally thought its purpose is not specifically documenting the intra-cluster communication protocols and flows.

• This functionality has gotten more configurable in later versions of the product. Searching for the name of the command-line management tool, cluster.exe, and the word "heartbeat" targeted against Microsoft's site seems to give a lot of potentially good results (one of the first hits I found has a paragraph containing the phrase "...here each node has an interface on each cluster network, there are N * (N - 1) unicast heartbeats sent per network every 1.2 seconds...", which sounds just like the kind of things you're looking for-- albeit describing Windows Server 2003.)

And, of course, if you want to see more of the low levels yourself set it up and throw a sniffer at it.

I actually like giving the remote node a vote because it'd make planned failovers that much easier. You could migrate databases & resources over to the remote datacenter, then gradually shut down the nodes in the primary datacenter, and you wouldn't have to monkey with the voting in order to get it to work. Plus, you're not worried about high availability on the file share.