How to automate production deployments without experiencing extreme anxiety

To expand on Robert's answer,

Automation is definitely the correct approach. The following should provide a good starting point:

• Ansible: this should be used to actually provision the server. If the Ansible tasks are written correctly, they can be idempotent (run multiple times with no worries), and will only update/change whatever needs to be changed. The typical way to use Ansible is to run it "remotely" using SSH, but I recommend using Ansible in "local" mode (explained below).

• Shell scripts: if anything specific/complex needs to occur on boot, or at the end of the Ansible run, it's best to move it to a shell script which can be called separately. When writing shell scripts, ensure they are POSIX compliant and don't contain too much crazy logic, this way they can run safely on various operating systems, and won't be too difficult to debug down the line.

• Pull, not push: since your customers need to download binaries, you can host them on a local/private HTTPS server, and have your Ansible scripts "pull" (download) the binaries. You can even add an sha256 hash in your Ansible YAML config files to verify the integrity of the downloads. Of course development will be a bit slower if you always need to update the hash in the config files, but if you don't change the binaries too often (ex: once a month), it's an acceptable approach.

• Virtual machines: if you setup a virtual machine, you can test your Ansible and shell scripts on there without affecting your customer's servers. This allows you to catch errors and avoid destroying a production server.

My approach is the following:

• Ensure the server has Git and Ansible installed.
• Ensure the server has an SSH private key to git pull from either GitHub or a private git+ssh server.
• SSH into the server, and perform a git pull, then ansible-playbook -c local

Ansible in "local" mode requires your server to have Ansible installed, but it has the advantage that provisioning is much faster, and doesn't require a network connection (except for the initial git pull).

This method allows your customers to manually decide when a server gets updated, and allows you/them to verify code pushed to the Git repo, before running Ansible on the server. If you're using GitHub to keep track of your Ansible scripts, then you can create a bot account for pulling the repo changes (allowed as per GitHub's ToS), or use your server's public SSH key as a deploy key.

I've listed a few other best practices for building software for on-premises, which you can read about here for additional tips.

Normally you wouldn't just have a dedicated build machine, but also run a build server on that dedicated machine. A dedicated build machine merely offers the advantage of never blocking the work of a developer and deploying from a centralized machine.

A build server offers much more. A build sever allows for CI (continuous integration), meaning that it will automatically build on every push to your VCS (like git), might even execute unit tests if you have them and allows for "one click deployment". Build servers can notify you per mail if builds or tests fail. They offer historic data and trends on what happened.

Build servers can generally be accessed by multiple users or teams at once, by using a web gui that runs in a browser.

In the Java world one of the most used build servers is Jenkins. Jenkins works perfectly fine with C++ builds as well (Since you seem to use those two languages). Jenkins calls itself automation server, since it can run all kinds of tasks that don't have to be related to programming and building.