DNS errors in libcurl/c-ares, but not in command-line curl

curldomain-name-systemhttps

Problem description

I have an executable which repeatedly posts data to an HTTPS endpoint, using libcurl with libcares. This is getting occasional DNS resolution timeouts on some clients (not all). If I run the equivalent command in command-line curl, I never see any timeouts.

What makes this even more confusing is that the host is explicitly specified in /etc/hosts, so there shouldn't be any DNS resolution required.

The error from libcurl (with verbose mode) is:

* Adding handle: conn: 0xcbca20
* Adding handle: send: 0
* Adding handle: recv: 0
* Curl_addHandleToPipeline: length: 1
* - Conn 88 (0xcbca20) send_pipe: 1, recv_pipe: 0
* Resolving timed out after 2002 milliseconds
* Closing connection 88

My libcurl executable is sending 2-3 queries a second, and I see this error about once every 300 requests. Using command-line curl, I have run 10000 queries without a single timeout.

Can anyone suggest anything I can try to resolve these errors from libcurl? Are there any settings which I need to add to my libcurl setup,
or system configuration I might be missing?

I wasn't sure whether to put this in Stack Overflow, Server Fault, or Ask Ubuntu; apologies if it is in the wrong place.

Thanks for your time!

More detailed information

The client is Ubuntu 12.04, 64-bit. The same problem has been observed on several clients, all with the same OS.

Usernames/passwords/urls have been obfuscated in the following snippets.

Command-line curl tester (using v 7.22.0):

while true; do curl -v -u username:password "https://myhost.com/endpoint" -X POST --data "a=x&b=y" >> /tmp/commandLine.log 2>&1; sleep 0.1; done &

Libcurl source code (using curl 7.30.0, with c-ares 1.10.0):

#include <curl/curl.h>
#include <unistd.h>
#include <string>
#include <iostream>

using namespace std;

int main(int argc, char** argv) {

   while (1) {

       // Initialise curl
       CURL *curl = curl_easy_init();

       // Set endpoint
       string urlWithEndpoint = "https://myhost.com/endpoint";
       curl_easy_setopt(curl, CURLOPT_URL, urlWithEndpoint.c_str());

       // Set-up username and password for request
       curl_easy_setopt(curl, CURLOPT_USERPWD, "username:password");

       // Append POST data specific stuff
       string postData = "a=x&b=y";
       long postSize = postData.length();
       curl_easy_setopt(curl, CURLOPT_POSTFIELDSIZE, postSize);
       curl_easy_setopt(curl, CURLOPT_COPYPOSTFIELDS, postData.c_str());

       //set timeouts
       curl_easy_setopt(curl, CURLOPT_TIMEOUT, 10);
       curl_easy_setopt(curl, CURLOPT_CONNECTTIMEOUT, 2);
       curl_easy_setopt(curl, CURLOPT_DNS_CACHE_TIMEOUT, 60);

       cout << endl << endl << "=========================================================" << endl << endl;
       cout << "Making curl request to " << urlWithEndpoint << "(POST size " << postSize << "B)" << endl;

       // Set curl to log verbose information
       curl_easy_setopt(curl, CURLOPT_VERBOSE, 1);

       // Perform the request
       CURLcode curlRes = curl_easy_perform(curl);

       // Handle response
       bool success = false;
       if (curlRes == CURLE_OK) {
           long httpCode;
           curl_easy_getinfo(curl, CURLINFO_RESPONSE_CODE, &httpCode);
           success = (httpCode==200);
           cout << "Received response " << httpCode << endl;
       } else if ( curlRes == CURLE_OPERATION_TIMEDOUT ) {
           cout << "Received timeout" << endl;
       } else {
           cout << "CURL error" << endl;
       }

       curl_easy_cleanup(curl);

       if (success) {
           cout << "SUCCESS! (" << time(0) << ")" << endl;
           usleep(0.1*1e6);
       } else {
           cout << "FAILURE!!!! (" << time(0) << ")" << endl;
           usleep(10*1e6);
       }

   }

}

Best Answer

Answering my own question...

It turned out that the problem was in lib c-ares (which gives thread-safe DNS resolution in libcurl). Having recompiled libcurl with --enable-threaded-resolver instead, the resolution timeouts stop.

I've tried updating to the very latest lib c-ares, and looking on the c-ares forums for similar bugs, but no luck on either front. So I'm ditching c-ares and using the Curl threaded resolver going forward.