• notice
  • Congratulations on the launch of the Sought Tech site

Solve the problem of Connection reset by peer or EOF for Golang crawler

The code of an http Client program is as follows:

// create a requestreq, err := http.NewRequest(method, url, body)if err != nil {    return nil, err
}// send JSON to firebaseresp, err := http.DefaultClient.Do(req)if err != nil {    return nil, err
}if resp.StatusCode != http.StatusOK {    return nil, fmt.Errorf("Bad HTTP Response: %v", resp.Status)

defer resp.Body.Close()
b, err := ioutil.ReadAll(resp.Body)if err != nil {    return nil, err

After sending to the server multiple times, the following error occurs almost every time:

ERROR 10108 socket.cpp:985 0x7fffe81426e0 recvmsg(62, 1): (104, "Connection reset by peer")
WARN 10108 server.cpp:467 0x7fffe80abd60-2 Unexpected SocketException


Before solving the problem, you need to understand some background knowledge about how go implements connection: there are two coroutines, one for reading and one for writing (that is, readLoop and writeLoop). In most cases, readLoop will detect whether the socket is closed, and close the connection when appropriate. If a new request arrives before readLoop detects the close, then an EOF error is generated and execution is interrupted instead of closing the previous request. The same is true here. I establish a new connection when I execute it, and exit after this program is executed. When I open it again, the server does not know that I have closed the connection, so it prompts that the connection is reset; if I do not exit the program and use for When the loop is sent multiple times, the old connection is not closed, but the new connection arrives, and EOF will be reported.

Add property settings to req:

req.Close = true

It will prevent the connection from being reused, which can effectively prevent this problem, that is, the short connection of Http


Technical otaku

Sought technology together

Related Topic


Leave a Reply