We are trying to transfer large number of files from a network share into Openstack using a custom written Erlang application (using Erlang version R15B).
After about 20k files transferred the Erlang process starts returning enobufs errors resulting in it being un-able to establish HTTP connections to OpenStack. As well, C# code running on the same box can no longer open sockets resulting in errors seen in this question about lacking sufficient buffer space. Closing all the custom applications and restarting does not alleviate the situation.
The number of open connections never exceeds 300 connected but does typically go into the 2k range when including TIME_WAIT. Connections are not being held open but are being opened and closed quickly.
All the articles I could find seem to mention that 'ephemeral' port limit but it doesn't seem like we are reach that, and closing the Erlang application doesn't result in any improvements.
Is there something else that could be holding on to buffer space and not releasing it properly?
Best Answer
One of the functions of the application was to talk to a local service. It appears as if this triggered a kernel socket leak in Server 2008 where it would not release sockets properly. Applying the hotfix mentioned in the support article resolved the issue.