You've certainly done your research...
From all of my experience with ansible what you're looking to accomplish, isn't supported. As you mentioned, ansible states that it does not require passwordless sudo, and you are correct, it does not. But I have yet to see any method of using multiple sudo passwords within ansible, without of course running multiple configs.
So, I can't offer the exact solution you are looking for, but you did ask...
"So... how are people using Ansible in situations like these? Setting
NOPASSWD in /etc/sudoers, reusing password across hosts or enabling
root SSH login all seem rather drastic reductions in security."
I can give you one view on that. My use case is 1k nodes in multiple data centers supporting a global SaaS firm in which I have to design/implement some insanely tight security controls due to the nature of our business. Security is always balancing act, more usability less security, this process is no different if you are running 10 servers or 1,000 or 100,000.
You are absolutely correct not to use root logins either via password or ssh keys. In fact, root login should be disabled entirely if the servers have a network cable plugged into them.
Lets talk about password reuse, in a large enterprise, is it reasonable to ask sysadmins to have different passwords on each node? for a couple nodes, perhaps, but my admins/engineers would mutiny if they had to have different passwords on 1000 nodes. Implementing that would be near impossible as well, each user would have to store there own passwords somewhere, hopefully a keypass, not a spreadsheet. And every time you put a password in a location where it can be pulled out in plain text, you have greatly decreased your security. I would much rather them know, by heart, one or two really strong passwords than have to consult a keypass file every time they needed to log into or invoke sudo on a machine.
So password resuse and standardization is something that is completely acceptable and standard even in a secure environment. Otherwise ldap, keystone, and other directory services wouldn't need to exist.
When we move to automated users, ssh keys work great to get you in, but you still need to get through sudo. Your choices are a standardized password for the automated user (which is acceptable in many cases) or to enable NOPASSWD as you've pointed out. Most automated users only execute a few commands, so it's quite possible and certainly desirable to enable NOPASSWD, but only for pre-approved commands. I'd suggest using your configuration management (ansible in this case) to manage your sudoers file so that you can easily update the password-less commands list.
Now, there are some steps you can take once you start scaling to further isolate risk. While we have 1000 or so nodes, not all of them are 'production' servers, some are test environments, etc. Not all admins can access production servers, those than can though use their same SSO user/pass|key as they would elsewhere. But automated users are a bit more secure, for instance an automated tool that non-production admins can access has a user & credentials that cannot be used in production. If you want to launch ansible on all nodes, you'd have to do it in two batches, once for non-production and once for production.
We also use puppet though, since it's an enforcing configuration management tool, so most changes to all environments would get pushed out through it.
Obviously, if that feature request you cited gets reopened/completed, what you're looking to do would be entirely supported. Even then though, security is a process of risk assessment and compromise. If you only have a few nodes that you can remember the passwords for without resorting to a post-it note, separate passwords would be slightly more secure. But for most of us, it's not a feasible option.
I don't see any reason this would be a bad idea. Staging implies a non-production service, so you can have all your staging happen on one server without concern for redundancy, performance, etc. One other reason for separating applications and instances of applications is for security, but once again for a non-production service this is less of a concern, unless you host a copy of your production data on your staging server (which is another type of separation to think about). If there are security concerns, and you can't use subdomains, you could still use proxying in nginx to divide the instances up across multiple VMs. But, in the simplest form, you'd just run them all on the same instance, and use 'location' directives to determine what app each path runs.
nginx doesn't care how you divide up your applications, though you'll need to configure a different proxy configuration (either different ports or different locations for configuration files, depending on the app server) to each application server based on the directory. How you do this will depend somewhat on what your app server is, which you have not specified.
For example, with Ruby using the Passenger application server, you would follow these instructions: https://www.phusionpassenger.com/documentation/Users%20guide%20Nginx.html#deploying_rails_to_sub_uri
Notice that Passenger hides some of the implementation details from you, so you don't need to think about ports, only directories. Whereas in the following example, you'd specify the port for proxying and it would need to be different for each application instance.
With Perl Mojolicious, you could follow this example: http://search.cpan.org/~sri/Mojolicious-5.27/lib/Mojolicious/Guides/Cookbook.pod#Nginx
Note that the location
section in either example can be replicated for as many paths as you need independent instances of your app. It's the same as running multiple apps; you just give each one its own directory and configuration files.
I suspect the reason you haven't gotten any answers thus far is that you haven't provided any concrete information about your deployment. I'm finding it hard to offer advice when you've specified you'll be using "Ansible, Chef, Puppet, etc." and "Redis/MongoDB"...are you using all of these things, despite the fact that they do very similar jobs? Most people choose one deployment tool (from Ansible, Chef, Puppet, etc.) and one key/value store (Redis/MongoDB). Configuration examples for all of these, and the caveats for things to look out for will be different for each.
Anyway, the short answer to your question is: Yes, you can do that, and there's no technical reason you shouldn't.
Best Answer
What I would do is to have roles for each system service that your application needs, a play and a role for each application/microservice, and group and/or host variables and role variables and defaults which define what to do.
I deploy a lot of PHP-based applications, so that looks a lot like this:
I'll have a play
app_microservice.yml
:So I'll have a role
roles/app_microservice
which deploys the code. When I run this play, the nginx, mariadb and php-fpm prerequisites will be installed and configured first, if they haven't already been.In addition to calling
roles
, a play can also run arbitrarytasks
. Feel free to mix and match these if something is simple enough that a full role isn't called for.This play also goes into
all.yml
along with every other play, so that I can occasionally doansible-playbook all.yml
. Remember that ansible doesn't guarantee idempotence like puppet tries to, so this is something you have to be careful of.I use group variables to define things which are common to a group (though there are precious few of these that won't fit in the role variables or defaults instead), group
all
variables for global stuff, and host variables for anything that's unique to a host.For instance, I give a unique MySQL root password to every host, but I have SSL ciphers and protocols defined in
group_vars/all/main.yml
so that, if they need to be changed, there is one source of truth for them.