C# – Deploy a .NET Windows Service with Amazon Elastic Beanstalk with no Web Application

amazon-elastic-beanstalkamazon-web-servicescnet

I want to create an Elastic Beanstalk configuration that allows me to deploy a .NET Windows Service but without deploying a web application.

I have just read this blog post which explains how to use .ebextensions to deploy a Windows Service alongside your web application, but is there a scenario for which the .ebextensions can be run without deploying a Web Deploy package for a web application?

Is my only option to create an empty web application that contains the .ebextensions directory and then deploy the Web Deploy package?

The Elastic Beanstalk FAQ mentions the ability to deploy non-web applications (here) and I have found a similar (unanswered) question on the AWS developer forums (here).

Update

Due to the lack of activity on this question and my inability to find any other information on the internet, I just assumed that the answer to this question is "No" (at least for now).

I ended up creating an empty web application and used that to deploy my Windows Service via the .ebextensions YAML config.

As a side note, I'd like to highlight this page from Amazon's documentation which I found to be a very helpful guide to creating those special config files.

Another Update

After implementing the approach mentioned above, I discovered that Elastic Beanstalk was not executing my .ebextensions scripts for new Beanstalk instances. As a result, the Windows Service failed to be installed when new instances were created. I had to jump through several more hoops to finally arrive at a scalable solution. Please let me know if you want the details of the final solution.

Ultimately, it just seems like Elastic Beanstalk wasn't meant for deploying scalable Windows Services.

Basic Solution

I'm not comfortable releasing the source code since it was not for a personal project, but here is the basic structure of my current deployment solution:

  1. A custom EC2 AMI contains a 'bootstrap' program that runs on startup. The program does the following:

    1.1. Download a 'zip' archive from a (configurable) 'deployment' S3 bucket

    1.2. Extract the downloaded zip file to a temporary directory

    1.3. An "install.bat" script is located/executed (the name of the script is also configurable). This script installs and starts the windows service.
  2. The Elastic Beanstalk "Instance AMI" is set to the custom AMI with the bootsrap program (see: this article)

To deploy new code: upload the installation .zip archive (that contains the windows service and install.bat file) to the S3 bucket and terminate all EC2 instances for the Elastic Beanstalk application. As the instances are re-created the bootstrapping program will download/install the newly updated code.

Of course, if I were starting over, I would just skip using Elastic Beanstalk and use the standard AWS auto-scaling along with a similar deployment scheme. The bottom line is that if you don't have a web application, don't use Elastic Beanstalk; you're better off with the standard AWS auto-scaling.

New AWS Deployment Tools

Amazon recently announced several new code deployment/management services that seem to address deployment issues: http://aws.amazon.com/blogs/aws/code-management-and-deployment/

I have yet to use these new services (I'm not even sure if they've been released yet), but they look promising.

Best Answer

Since this question has been around for a while and still have no answer, but continues to draw interest, let me share my solution to a very similar problem - installing a Windows service on a EC2 instance. I'm not using Beanstalk though, since that service is designed more for quick deploy of web applications. Instead, I'm using directly CloudFormation which Beanstalk uses underneath to deploy resources related to the web application.

The stack expects existing VPC (our spans through several availability zones), a S3 bucket that stores all service build artifacts and a EC2 key pair. Template creates EC2 instance using Windows AMI and few other resources like IAM User with access keys and a working S3 bucket just for illustration of how to create additional resources that your service might need. Template also takes as a parameter the name of a zipped package with all service binaries and configuration files that's been uploaded on the build artifacts S3 bucket (we use TeamCity build server that makes that for us, but you can create and upload the package manually of course). When you build new version of the service, you simply create new package (for example service.v2.zip), update the stack with the new name and the service will be updated automatically. Template contains ids of AMIs in 4 different regions, but you can always add other regions if you wish. Here's the stack template:

{
    "AWSTemplateFormatVersion": "2010-09-09",
    "Description": "Service stack template.",
    "Parameters": {
        "KeyPair": {
            "Type": "String",
            "Default": "MyDefaultKeys",
            "Description": "Name of EC2 Key Pair."
        },
        "ServicePackageName": {
            "Type": "String",
            "Default": "service.zip",
            "Description": "Name of the zip package of the service files."
        },
        "DeploymentBucketName": {
            "Type": "String",
            "Default": "",
            "Description": "Name of the deployment bucket where all the artifacts are."
        },
        "VPCId": {
            "Type": "String",
            "Default": "",
            "Description": "Identifier of existing VPC."
        },
        "VPCSubnets": {
            "Default": "",
            "Description": "Commaseparated list of existing subnets within the existing VPC. Could be just one.",
            "Type": "CommaDelimitedList"
        },
        "VPCSecurityGroup": {
            "Default": "",
            "Description": "Existing VPC security group. That should be the ID of the VPC's default security group.",
            "Type": "String"
        }
    },
    "Mappings": {
        "Region2WinAMI": {
            "us-east-1": { "64": "ami-40f0d32a" },
            "us-west-1": { "64": "ami-20601740" },
            "us-west-2": { "64": "ami-ff4baf9f" },
            "eu-west-1": { "64": "ami-3367d340" }
        }
    },
    "Resources": {
        "ServiceInstance": {
            "Type": "AWS::EC2::Instance",
            "Metadata": {
                "Comment": "Install Service",
                "AWS::CloudFormation::Init": {
                    "configSets": {
                        "default": [ "ServiceConfig" ]
                    },
                    "ServiceConfig": {
                        "files": {
                            "c:\\service\\settings.config": {
                                "source": { "Fn::Join": [ "/", [ "https://s3.amazonaws.com", { "Ref": "DeploymentBucketName" }, "deployments/stacks", { "Ref": "AWS::StackName" }, "templates/settings.config.mustache" ] ] },
                                "context": {
                                    "region": { "Ref": "AWS::Region" },
                                    "accesskey": { "Ref": "IAMUserAccessKey" },
                                    "secretkey": { "Fn::GetAtt": [ "IAMUserAccessKey", "SecretAccessKey" ] },
                                    "bucket": { "Ref": "BucketName" }
                                }
                            },
                            "c:\\cfn\\cfn-hup.conf": {
                                "content": {
                                    "Fn::Join": [
                                        "",
                                        [
                                            "[main]\n",
                                            "stack=",
                                            { "Ref": "AWS::StackId" },
                                            "\n",
                                            "region=",
                                            { "Ref": "AWS::Region" },
                                            "\n",
                                            "interval=1"
                                        ]
                                    ]
                                }
                            },
                            "c:\\cfn\\hooks.d\\cfn-auto-reloader.conf": {
                                "content": {
                                    "Fn::Join": [
                                        "",
                                        [
                                            "[cfn-auto-reloader-hook]\n",
                                            "triggers=post.update\n",
                                            "path=Resources.ServiceInstance.Metadata.AWS::CloudFormation::Init\n",
                                            "action=cfn-init.exe -v -s ",
                                            { "Ref": "AWS::StackName" },
                                            " -r ServiceInstance --region ",
                                            { "Ref": "AWS::Region" },
                                            "\n"
                                        ]
                                    ]
                                }
                            }
                        },
                        "sources": {
                            "c:\\tmp\\service": { "Fn::Join": [ "/", [ "https://s3.amazonaws.com", { "Ref": "DeploymentBucketName" }, "deployments/stacks", { "Ref": "AWS::StackName" }, "artifacts/Service", { "Ref": "ServicePackageName" } ] ] }
                        },
                        "commands": {
                            "Install Service": {
                                "command": "call c:\\tmp\\service\\install.bat",
                                "ignoreErrors": "false"
                            }
                        },
                        "services": {
                            "windows": {
                                "cfn-hup": {
                                    "enabled": "true",
                                    "ensureRunning": "true",
                                    "files": [ "c:\\cfn\\cfn-hup.conf", "c:\\cfn\\hooks.d\\cfn-auto-reloader.conf" ]
                                }
                            }
                        }
                    }
                }
            },
            "Properties": {
                "ImageId": { "Fn::FindInMap": [ "Region2WinAMI", { "Ref": "AWS::Region" }, "64" ] },
                "InstanceType": "t2.micro",
                "KeyName": { "Ref": "KeyPair" },
                "SecurityGroupIds" : [{ "Ref": "VPCSecurityGroup" }],
                "SubnetId" : { "Fn::Select": [ "0", { "Ref": "VPCSubnets" } ] },
                "UserData": {
                    "Fn::Base64": {
                        "Fn::Join": [
                            "",
                            [
                                "<script>\n",
                                "if not exist \"C:\\logs\" mkdir C:\\logs \n",
                                "cfn-init.exe -v -s ",
                                { "Ref": "AWS::StackName" },
                                " -r ServiceInstance --region ",
                                { "Ref": "AWS::Region" },
                                " -c default \n",
                                "</script>\n"
                            ]
                        ]
                    }
                },
                "BlockDeviceMappings": [
                    {
                        "DeviceName": "/dev/sda1",
                        "Ebs": {
                            "DeleteOnTermination": "true",
                            "VolumeSize": "40",
                            "VolumeType": "gp2"
                        }
                    }
                ],
                "Tags": [
                    { "Key": "Name", "Value": { "Fn::Join": [ ".", [ { "Ref": "AWS::StackName" }, "service" ] ] } }
                ]
            }
        },
        "BucketName": {
            "Type": "AWS::S3::Bucket",
            "Properties": {
                "AccessControl": "PublicRead"
            },
            "DeletionPolicy": "Retain"
        },
        "IAMUser": {
            "Type": "AWS::IAM::User",
            "Properties": {
                "Path": "/",
                "Groups": [ "stack-users" ],
                "Policies": [
                    {
                        "PolicyName": "giveaccesstobuckets",
                        "PolicyDocument": {
                            "Version": "2012-10-17",
                            "Statement": [
                                {
                                    "Effect": "Allow",
                                    "Action": [ "s3:*" ],
                                    "Resource": [ { "Fn::Join": [ "", [ "arn:aws:s3:::", { "Ref": "BucketName" }, "/*" ] ] } ]
                                }
                            ]
                        }
                    }
                ]
            }
        },
        "IAMUserAccessKey": {
            "Type": "AWS::IAM::AccessKey",
            "Properties": {
                "UserName": { "Ref": "IAMUser" }
            }
        }
    }
}

As you can see, after copying the artifacts, we execute install.bat batch file (included in the zip file) that will move the files to the correct location and register the service. Here is the contents of the file:

@echo off
sc query MyService > NUL
IF ERRORLEVEL 1060 GOTO COPYANDCREATE
sc stop MyService
waitfor /T 20 ServiceStop
echo D | xcopy "c:\tmp\service" "c:\service\" /E /Y /i
GOTO END
:COPYANDCREATE
echo D | xcopy "c:\tmp\service" "c:\service\" /E /Y /i
sc create MyService binpath= "c:\service\MyService.exe" start= "auto"
:END
sc start MyService

Template also creates config file (from the settings.config.mustache which also resides on the artifacts bucket) containing information about the other resources that has been created for the service to use. Here it is:

<appSettings>
    <add key="AWSAccessKey" value="{{accesskey}}" />
    <add key="AWSSecretKey" value="{{secretkey}}" />
    <add key="AWSRegion" value="{{region}}" />
    <add key="AWSBucket" value="{{bucket}}" />
</appSettings>

You create and later update the stack either from the AWS web console or CLI.

And that's pretty much it. You can visit the AWS CloudFormation website to get more info about the service and how to work with templates.

P.S.: I realised that it would be better if I also share the template that creates the VPC. I keep it separate as I have one VPC per region. You can integrate it with the Service template if you wish, but that would mean that every time you create new stack a new VPC will be also created. Here is the VPC template:

{
    "AWSTemplateFormatVersion": "2010-09-09",
    "Description": "VPC stack template.",
    "Mappings": {
        "Region2AZ": {
            "us-east-1": { "AZ": [ "us-east-1a", "us-east-1b", "us-east-1d" ] },
            "us-west-1": { "AZ": [ "us-west-1b", "us-west-1c" ] },
            "us-west-2": { "AZ": [ "us-west-2a", "us-west-2b", "us-west-2c" ] },
            "eu-west-1": { "AZ": [ "eu-west-1a", "eu-west-1b", "eu-west-1c" ] }
        }
    },
    "Conditions": {
        "RegionHas3Zones": { "Fn::Not" : [ { "Fn::Equals" : [ { "Ref": "AWS::Region" }, "us-west-1" ] } ] }
    },
    "Resources": {
        "VPC": {
            "Type": "AWS::EC2::VPC",
            "Properties": {
                "CidrBlock": "10.0.0.0/16",
                "EnableDnsSupport" : "true",
                "EnableDnsHostnames" : "true"
            }
        },
        "VPCSecurityGroup": {
            "Type": "AWS::EC2::SecurityGroup",
            "Properties": {
                "GroupDescription": "Security group for VPC.",
                "VpcId": { "Ref": "VPC" }
            }
        },
        "Subnet0": {
            "Type": "AWS::EC2::Subnet",
            "Properties": {
                "VpcId": { "Ref": "VPC" },
                "CidrBlock": "10.0.0.0/24",
                "AvailabilityZone": { "Fn::Select": [ "0", { "Fn::FindInMap": [ "Region2AZ", { "Ref": "AWS::Region" }, "AZ" ] } ] }
            }
        },
        "Subnet1": {
            "Type": "AWS::EC2::Subnet",
            "Properties": {
                "VpcId": { "Ref": "VPC" },
                "CidrBlock": "10.0.1.0/24",
                "AvailabilityZone": { "Fn::Select": [ "1", { "Fn::FindInMap": [ "Region2AZ", { "Ref": "AWS::Region" }, "AZ" ] } ] }
            }
        },
        "Subnet2": {
            "Type": "AWS::EC2::Subnet",
            "Condition": "RegionHas3Zones",
            "Properties": {
                "VpcId": { "Ref": "VPC" },
                "CidrBlock": "10.0.2.0/24",
                "AvailabilityZone": { "Fn::Select": [ "2", { "Fn::FindInMap": [ "Region2AZ", { "Ref": "AWS::Region" }, "AZ" ] } ] }
            }
        },
        "InternetGateway": {
            "Type": "AWS::EC2::InternetGateway",
            "Properties": {
            }
        },
        "AttachGateway": {
            "Type": "AWS::EC2::VPCGatewayAttachment",
            "Properties": {
                "VpcId": { "Ref": "VPC" },
                "InternetGatewayId": { "Ref": "InternetGateway" }
            }
        },
        "RouteTable": {
            "Type": "AWS::EC2::RouteTable",
            "Properties": {
                "VpcId": { "Ref": "VPC" }
            }
        },
        "Route": {
            "Type": "AWS::EC2::Route",
            "DependsOn": "AttachGateway",
            "Properties": {
                "RouteTableId": { "Ref": "RouteTable" },
                "DestinationCidrBlock": "0.0.0.0/0",
                "GatewayId": { "Ref": "InternetGateway" }
            }
        },
        "SubnetRouteTableAssociation0": {
            "Type": "AWS::EC2::SubnetRouteTableAssociation",
            "Properties": {
                "SubnetId": { "Ref": "Subnet0" },
                "RouteTableId": { "Ref": "RouteTable" }
            }
        },
        "SubnetRouteTableAssociation1": {
            "Type": "AWS::EC2::SubnetRouteTableAssociation",
            "Properties": {
                "SubnetId": { "Ref": "Subnet1" },
                "RouteTableId": { "Ref": "RouteTable" }
            }
        },
        "SubnetRouteTableAssociation2": {
            "Type": "AWS::EC2::SubnetRouteTableAssociation",
            "Condition": "RegionHas3Zones",
            "Properties": {
                "SubnetId": { "Ref": "Subnet2" },
                "RouteTableId": { "Ref": "RouteTable" }
            }
        },
        "NetworkAcl": {
            "Type": "AWS::EC2::NetworkAcl",
            "Properties": {
                "VpcId": { "Ref": "VPC" }
            }
        },
        "AllowAllInboundTCPAclEntry": {
            "Type": "AWS::EC2::NetworkAclEntry",
            "Properties": {
                "NetworkAclId": { "Ref": "NetworkAcl" },
                "RuleNumber": "100",
                "Protocol": "6",
                "RuleAction": "allow",
                "Egress": "false",
                "CidrBlock": "0.0.0.0/0",
                "PortRange": { "From": "0", "To": "65535" }
            }
        },
        "AllowAllInboundUDPAclEntry": {
            "Type": "AWS::EC2::NetworkAclEntry",
            "Properties": {
                "NetworkAclId": { "Ref": "NetworkAcl" },
                "RuleNumber": "101",
                "Protocol": "17",
                "RuleAction": "allow",
                "Egress": "false",
                "CidrBlock": "0.0.0.0/0",
                "PortRange": { "From": "0", "To": "65535" }
            }
        },
        "AllowAllOutboundTCPAclEntry": {
            "Type": "AWS::EC2::NetworkAclEntry",
            "Properties": {
                "NetworkAclId": { "Ref": "NetworkAcl" },
                "RuleNumber": "100",
                "Protocol": "6",
                "RuleAction": "allow",
                "Egress": "true",
                "CidrBlock": "0.0.0.0/0",
                "PortRange": { "From": "0", "To": "65535" }
            }
        },
        "AllowAllOutboundUDPAclEntry": {
            "Type": "AWS::EC2::NetworkAclEntry",
            "Properties": {
                "NetworkAclId": { "Ref": "NetworkAcl" },
                "RuleNumber": "101",
                "Protocol": "17",
                "RuleAction": "allow",
                "Egress": "true",
                "CidrBlock": "0.0.0.0/0",
                "PortRange": { "From": "0", "To": "65535" }
            }
        },
        "SubnetNetworkAclAssociation0": {
            "Type": "AWS::EC2::SubnetNetworkAclAssociation",
            "Properties": {
                "SubnetId": { "Ref": "Subnet0" },
                "NetworkAclId": { "Ref": "NetworkAcl" }
            }
        },
        "SubnetNetworkAclAssociation1": {
            "Type": "AWS::EC2::SubnetNetworkAclAssociation",
            "Properties": {
                "SubnetId": { "Ref": "Subnet1" },
                "NetworkAclId": { "Ref": "NetworkAcl" }
            }
        },
        "SubnetNetworkAclAssociation2": {
            "Type": "AWS::EC2::SubnetNetworkAclAssociation",
            "Condition": "RegionHas3Zones",
            "Properties": {
                "SubnetId": { "Ref": "Subnet2" },
                "NetworkAclId": { "Ref": "NetworkAcl" }
            }
        }
    },
    "Outputs": {
        "VPC": {
            "Description": "VPC",
            "Value": { "Ref": "VPC" }
        },
        "VPCSecurityGroup": {
            "Description": "VPC Security Group Id",
            "Value": { "Fn::GetAtt": [ "VPCSecurityGroup", "GroupId" ] }
        },
        "Subnet0": {
            "Description": "Subnet0 Id",
            "Value": { "Ref": "Subnet0" }
        },
        "Subnet1": {
            "Description": "Subnet1 Id",
            "Value": { "Ref": "Subnet1" }
        },
        "Subnet2": {
            "Description": "Subnet2 Id",
            "Condition": "RegionHas3Zones",
            "Value": { "Ref": "Subnet2" }
        }
    }
}
Related Topic