Terminal Server CPU usage at 100%

In cases like this, you need to go beyond task manager and looking at % CPU usage. That does not tell you if something is adversely affecting performance. For a case like this, the next step would be to use Performance Monitor to view System\Processor Queue Length. This tells you if processes are waiting for the CPU to become idle possibly affecting performance. This is similar to what you see in the top or load commands in Unix.

This article has a good description of the performance metrics to look at when troubleshooting these issues. It was originally for NT4, but still applies to newer versions.

Here is a more recent article from the Windows Performance Team talking about how to hunt down performance issues with the CPU.

To accurately plan capacity for this, you need to investigate the characteristics of the application and the load it'll place on the terminal.

First off, if this is a critical line-of-business app that will lose the business money if it's not available for a few hours, you want to be looking at 2+ terminal servers running in parallell. Basic load-balancing of users across multiple terminal servers is actually really easy, you just do a round-robin DNS. The benefit here is that if one TS goes down for any reason, users that urgently require access can continue to access the system. My recommendation would be to look at 2 or 3 servers and ensure you have enough capacity in the environment to withstand the loss of one of the servers.

As for capacity/load, check the amount of memory that a user's session occupies on the TS when they're running the app. Multiply that by the number of users you expect to accomodate, add maybe a Gb for the system's own use, and then add another 20% on top for comfort. That's how much RAM you need, as a starting point, to support that number of users running that app on your TS. You must calculate based on an actual user connected to an actual example TS session, because each user will occupy extra RAM for other user processes in addition to the app itself. Those extra little processes add up.

Next, check your processing load and the characteristics of the app. Do users tell the app to run reports, which can peg the CPU at 100% for a short while? If so, big problem. Scaling that up to 60 users (even on a 16-core machine) means you'll have peak times where several people are trying to run reports and everyone's suffering.

Also consider any extra apps required by the users of the system. It's fairly common for users to want to output from business apps to office applications like Excel. Are they going to be able to shuffle things via shared drives to accomplish this, or is there a requirement to run office on the Terminal Server itself. If so, you need to be aware that a) office is licensed entirely differently in a terminal environment and regular versions will not install. b) A couple of Excel sessions soon eat up all your RAM.

tl;dr scale out across multiple servers rather than up within a single box