Puppet file transfer slow

file-transferpuppet

I have a puppet master and slaves in different datacenters. The latency between them is ~40ms. When I run "puppet agent –test" on a slave to apply the latest manifest it takes ~360 seconds to finish.

After doing some digging I can see the main cause of the slow down is file transfers. It seems it's taking ~10 seconds to transfer each file. The files are only small (configuration files) so I can't understand why they would take so long.

This is an example of a file in my manifest:

file { "/etc/rsyncd.conf" :
    owner       => "root",
    group       => "root",
    mode        => 644,
    source  => "puppet:///files/rsyncd/rsyncd.conf"
}

Running puppet-profiler I see this:

10.21s - File[/etc/rsyncd.conf]

It also seems I cannot update more than one server at once using puppet. If I run two servers at the same time then puppet takes twice as long.

I have changed the puppet master from using webrick to mongrel, but this doesn't seem to help. This is making deploying changes painful. A simple config change can take an hour to roll out to all servers.

Best Answer

Disclaimer: I am one of the Puppet developers.

When you source files with puppet:// URLs, Puppet will make two SSL connections back to the file server - one for the metadata, including checksum, and another for the content. The second is made, hopefully as expected, only if the content on disk is out of date.

The transfer itself is standard HTTPS, with nothing fancy going on, so beyond the SSL overhead and the latency cost of two connections, there is no reason that Puppet should be so slow.

Puppet will also checksum the file on both sides, the client and the server, when it comes to checking if it is up to date. That may be slow if your backing store is slow, too.

Finally, you should check if your network link is slow because it is saturated, or something like that. Wouldn't be the first time big buffers led to terrible network performance - and someone put a traffic shaper in the middle that made pings super-fast. ;)

Beyond that, I suggest you file a bug report. Including the details, and network traces, would be a big help in our being able to resolve this.

What you describe certainly isn't normal, and it isn't what we would expect to see for small configuration files on a reasonably fast network such as you describe. (Also, it isn't what others see, so clearly something unusual is happening in your setup.)

Related Topic