.NET Architecture – Custom .NET Apps and Clustering Best Practices

Architectureclusternet

So for a clustered environment – how would this work with your apps?

What about your own custom .NET apps? Would there be a special way to develop them? I know that you could create a simple Hello world app, and cluster that, but that wouldn't be something you could see in terms of the UI or anything. So they would effectively need to be developed as a Windows Service, or even, perhaps, as a standard Console app, which runs and does not wait for user input. But you wouldn't see any output from it (unless you redirect output to somewhere else).

What I'm getting at here is… for those who have experience or developed a cluster application in .NET, how did you do it? What are the things to be aware of?

For example, we have the cloud service, which is fundamentally built on clustering. If there is an outage, another node takes its place, and the service resumes as normal. However, we don't really see much of that downtime.

Best Answer

I've written a lot of .NET clustered applications over the years (exclusively for Active/Passive clusters) and can share what I've learned.

I've always written my apps as Windows Services and include them as clustered resources in the Windows Cluster Administrator. When the cluster fails over from Node1 to Node2, the Cluster Administrator shuts down the service on Node1 and starts it up on Node2. That's the simple part and Windows takes care of the heavy lifting.

Aside - There is no technical reason why you couldn't write a console app and have it kicked off by the Scheduled Tasks service (which can also be setup as a clustered resource) or some other means. When I first started writing clustered apps the requirements often required the process to be kicked off by the arrival of a file and that seemed to be a better fit for a service than for a console app. For consistency sake, we just continued to write them as services.

The design of your app is going to depend quite a bit on what exactly its task is. But conceptually, you'll need some way to keep track of what unit of work your app is executing so that if the cluster fails over, your app on Node2 knows where to pick up where the app on Node1 left off.

If your app is file based, meaning it is performing some work on files or the data contained in one or more files, those files will need to be on a drive that either fails over or is visible to both nodes. Your app will need to keep track of what file it was working on and potentially what record within the file as well. How it stores this progress is really up to you and what resources are available, but the important thing is that this progress is stored in a manner that will survive the failover (ie, in a DB table or a file on a drive). Storing this progress on a local drive of Node1 will do you no good when the app starts up on Node2, if Node1 is dead.

While you're designing/coding, keep asking yourself - if the cluster failed over at this exact point in the process,

What data would be lost? (and would it matter?)
Where would the other node deteremine it should start from? (and is it correct?)

There's no magic to it really, just a lot of thought in the details. Should your app do all of the work in transaction (DB or COM+)? That depends on the complexity of the task and requirements. Can you just tag records in a DB as your app completes them and have the other node work on untagged records? Sure, again it depends on the requirements.

Related Solutions

Architecture – How to implement a lightweight clustered architecture for a distributed application

Since you are moving away from a single master node (which is appropriate) you will have to change a few things. You will need to setup a Quorum. Since you already have 9 nodes, you are in good shape. For a Quorum to work you need 2n+1 nodes where (n) is the number of nodes that can go down and the system will still work. Within the Quorum a vote will take place on who the leader is, and what transactions are successful. This can be used to pass around configuration information and ensure everyone is synchronized without a database.

There are existing technologies out there that can help you with this. One of thos is ZooKeeper. It is an open source Apache v2 product for Distributed Coordination. You will need something along these lines. Whether it is using ZooKeeper or rolling your own their white papers will be invaluable. It can also be used to maintain your configuration information about each node.

ZooKeeper is written in Java but I have created a project (ZooKeeperNet that will allow it to be embeded within .NET application using IKVM. If this isn't acceptable then you'll want to read about Leader Elections when determining who will be the current Master node. I suggest reading all their Wiki pages and Recipes to get an idea of what you need to account for in a proper distributed system.

Just so you have a good understanding. ZooKeeper is the backing coordination system of Hadoop and HBase. Hadoop is a distributed Map/Reduce framework.

If you already aren't, you can use WCF adhoc or registry discovery information when attempting to find the current master node in your system. If only a single Master node is alive it will be the only registered to support IMaster features. Then your slave nodes will listen on each other's znodes for each other to go away, picking up being the Master almost immediately.

Keep in mind that in order to be high efficient, the data each node needs to work with has to be close (i.e. on the node itself) to the node. If one node acts as a data intermediary you won't be as efficient as you could if the nodes could pull data in a distributed fashion.

C# – S.O.L.I.D., avoiding anemic domains, dependency injection

I would say that neither a person nor a vehicle should know whether a payment is due.

reexamining the problem domain

If you look at the problem domain "people having cars": In order to purchase a car you have some sort of contract which transfers ownership from one person to the other, neither the car nor the person change in that process. Your model is completely lacking that.

thought experiment

Assuming there is a married couple, the man owns the car and he is paying the taxes. Now the man dies, his wife inherits the car and will pay the taxes. Yet, your plates will stay the same (at least in my country they will). It is still the same car! You should be able to model that in your domain.

=> Ownership

A car that is not owned by anyone is still a car but nobody will pay taxes for it. In fact, you are not paying taxes for the car but for the privilege of owning a car. So my proposition would be:

public class Person
{
    public string Name { get; set; }

    public string Surname { get; set; }

    public List<Ownership> getOwnerships();

    public List<Vehicle> getVehiclesOwned();
}

public abstract class Vehicle
{
    public string PlateNumber { get; set; }

    public Ownership currentOwnership { get; set; }

}

public class Car : Vehicle {}

public class Motorbike : Vehicle {}

public abstract class Ownership
{
    public Ownership(Vehicle vehicle, Owner owner);

    public Vehicle Vehicle { get;}

    public Owner CurrentOwner { get; set; }

    public abstract bool HasToPay();

    public DateTime LastPaidTime { get; set; }

   //Could model things like payment history or owner history here as well
}

public class CarOwnership : Ownership
{
    public override bool HasToPay()
    {
        return (DateTime.Today - this.LastPaidTime).TotalDays >= 30;
    }
}

public class MotorbikeOwnership : Ownership
{
    public override bool HasToPay()
    {
        return (DateTime.Today - this.LastPaidTime).TotalDays >= 60;
    }
}

public class PublicAdministration
{
    public IEnumerable<Ownership> GetVehiclesThatHaveToPay()
    {
        return this.GetAllOwnerships().Where(HasToPay());
    }

}

This is far from perfect and probably not even remotely correct C# code but I hope you get the idea (and somebody could clean this up). The only downside is that you would probably need a factory or something to create the correct CarOwnership between a Car and a Person