Golang HTTP Concurrent Requests POST EOF

gohttpweb

I'm running a small client in Go. We're getting a number of false alarms at the moment, which seem to come down to client.Do() returning an EOF error when num>=1000.

This is the essence of my code:

func DoCreate(js string, cli *http.Client) {

    url2 := "http://xvz.myserver.com:9000/path/create"

    postBytesReader := bytes.NewReader([]byte(js))
    request, _ := http.NewRequest("POST", url2, postBytesReader)

    resp, err := cli.Do(request)
    if err != nil {
        fmt.Println(err) // Post http://xvz.myserver.com:9000/path/create: EOF 
        return
    }
    body, _ := ioutil.ReadAll(resp.Body)
    fmt.Println(string(body))

    defer resp.Body.Close()
}

func main() {
    client := &http.Client{
        CheckRedirect: nil,
    }

    for i := 0; i < num; i++ {
        var pq CreateReqInfo
        pq.Uid = strconv.Itoa(bid + i)
        petstr, _ := json.Marshal(pq)
        go DoCreate(string(petstr), client)
    }
}

Is problem about number of file handles or max connections?

Best Answer

It looks to me that you might be having the issue described in this answer. Basically, the Go http client will try to reuse connections unless you've specifically indicated that it shouldn't either in the Transport you've set your Client instance to use or on the request itself with:

request.Close = true

This becomes a problem when the server on the other end closes the connection without indicating so with a Connection: close header in a response. The net.http code assumes that the connection is still open and so the next request that tries to use the connection next, encounters the EOF from the connection being closed the other time.

You would need to check what is happening on the connection around the 1000th request. Is the receiving server set to limit the number of connections per client? Is your code running into a connection timeout? In either case, if it so happens the connection is closed by the server in a way that the Go Client code can't predict, then you'll run into the EOF.

Related Topic