Windows Server 2012 R2 kernel threads increase until the system hang

Could you describe how this process is started, and provide us with the output of ulimit -a from a context as close from that as possible?

As pthread_create(3) explains, you are running into:

ERRORS
       EAGAIN Insufficient  resources  to create another thread, or a system-
              imposed limit on the number of threads  was  encountered.   The
              latter  case  may  occur  in  two  ways:  the RLIMIT_NPROC soft
              resource limit (set via setrlimit(2)), which limits the  number
              of  process  for  a  real user ID, was reached; or the kernel's
              system-wide limit on  the  number  of  threads,  /proc/sys/ker‐
              nel/threads-max, was reached.

After enabling TLS 1.2 (also 1.0 and 1.2) on Windows 2012 R2 and 2008 R2 SP1, process lsass.exe began to overload CPU. I used the IISCrypto40, click "Best Practices", and in "Key Exchanges Enabled" left enabled only PKCS. After it, the load on the cpu returned to normal (5 - 10%). p.s. sorry for the poor english IISCrypto preferences