About the problems I encountered in nginx multi-layer proxy static resources in a certain technology

In the nginx deployment architecture, the back-end interface and the front-end static resources will be proxied by nginx. The deployment architecture diagram is as follows.

In nginx, the following proxy will be done:

• Listen to port 80, which is also the entry point of the endpoint

• Forward request to microservice gateway based on path

• Forward the request to the front-end static resource proxy based on the path

• Listen to port 5000 as the front-end static resource proxy entry

• Match the corresponding static resource directory according to the path

The request for the front-end static resources will be forwarded from the server on port 80 to the server on port 5000. This forwarding is forwarded by the svc name. For svc, you need to know the relevant knowledge of k8s (recommended to read "kubernetes authoritative guide"). Simply think that the svc name is the domain name created by k8s for nginx internally. Applications deployed in k8s can request nginx through the svc name, but this request may be allocated to different nginx nodes because load balancing will be performed.

Problem : Under this architecture, we performed stress testing on the front-end static resources and performed stress testing through jmeter. The performance was unsatisfactory, and after a period of time, errors were reported. After viewing the nginx log, we found a large number of errors, prompting Not enough ports.

The problem at that time was located in the investigation

• Through our analysis of the error report, when the static resource request is a request from port 80 and forwarded to port 5000, it is actually a re-created tcp connection. At this time, the request may be loaded to any nginx, and the initiator of the tcp connection will randomly use the system. A port is used as its own port to establish a connection with 5000. When the concurrency is sufficient, all 65535 ports will be occupied (in fact, there is no 65535, the system occupancy and other applications will occupy some ports, reserved ports, etc.) .

• After checking some information, we learned that when the application calls the close() method, it will not release the connection immediately, but will change the connection from the active state to the TIME_WAIT state. The operating system will release the connection after the TIME_WAIT time. Do you want to reduce this time? The linux2.x kernel does support modifying this system parameter, but the disadvantage of container deployment is that I don’t know how to modify the kernel parameters of the operating system in the base image, even if I enter the container and modify the operation The system parameters will be invalid after restarting, so I gave up this idea.

• The front-end static resources are actually the http protocol. We will start with the http protocol to see if there is a way to solve the problem of the port being occupied. It happens that http provides the keepalive parameter, which allows the client to request several Or multiple http requests are combined into one tcp connection transmission; we try to add keepalive to the jmeter request header and continue the stress test, but the situation has not been alleviated, and we have declared keepalive in the nginx configuration, but it has not been alleviated.

• Later, after checking the information according to the error report, we found that the operating system provides a parameter tw_reuse, which means that the connection in TIME_WAIT can be applied to a new tcp connection. In short, the tcp connection in the TIME_WAIT state can be reused , This is the default state of 0 in the Linux kernel system. We still need to modify the system configuration, and we return to the original problem. After the modification, it only fails after a temporary restart. We did not choose to try.

• We will try to change the version of nginx later to see if this problem can be solved. We changed several versions (continuously upgrading or downgrading versions) and found that 502 will not appear, and there is no indication of insufficient ports in the log, but tps does not Rising, and ushered in new problems. After the nginx version was changed, the back-end login interface was abnormal during the stress test. This is a problem that did not exist in the original version. The specific error has not been reviewed, but we have never found static A version in which both the proxy and the backend login interface are not abnormal at the same time.

• I got to 3:00 the next day during the day, and I had no energy. Fortunately, it was already Saturday. After taking a taxi back, Meimei took a nap, and the first thing I did when I got up was to reproduce this scene on my computer, which is not a virtual machine locally. I built nginx on the environment of the machine, and used the same mode to forward requests from port 80 to port 5000 without any exception, but my upstream is localhost:5000, I don’t know if it is because of localhost and it will not consume the port. . But I found a more important phenomenon, the performance loss of request forwarding to port 5000 through port 80 is very large, directly requesting a static resource tps of 1kp size on port 5000 can reach about 6200 tps, and then forwarding to port 80 and then to The same static resource request tps on port 5000 can only reach about 2500 tps. Below the stress test report, I sorted out the test results and sent them to the relevant colleagues in the architecture group. The relevant colleagues in the subsequent maintenance of nginx will use the static resources as the nginx localhost configuration to exist in 80 in the server block of the port.
  
  

Retrospective :
Now that I have left the company at the time, the issues at the time are now a bit vague, and some statements may be wrong, but prevent us from doing retrospectives.

Reanalysis :

• First of all, it will cause the problem that the ports are occupied, mainly because there are more than 40 static resources in a combined scene during the stress test, and many files are not small, so we first tested the connection between the jmeter cluster and the nginx cluster. Whether there is enough bandwidth, although the maximum bandwidth is not obtained, it far exceeds the traffic when the tps is the highest at that time.

• To look at the problem, you have to look at the essence through the phenomenon. The reason for the insufficient port is because there are too many static resources, and the static resources are too large. If the resources are too many and large, it will take a long time to transmit a resource. , the data of a back-end interface is only tens to hundreds of bytes, and a compressed image is about tens to hundreds of kb, if it is a high-definition image, it will be larger, which will cause the tcp connection to take a long time. Long, high concurrency, there will be no connections available, and a connection needs to be bound to a port.

• Whether the keepalive attribute is useful, I think it is useful, but our deployment of nginx at that time was based on mirroring, the nginx configuration only opened the mapping of path and upstream, other configurations can only be injected through environment variables, keepalive can be configured on the upstream block , environment variables should only be injected into the global configuration, some configurations are not applied to the global scope, so I was skeptical about whether the keepalive took effect at that time.

upstream backend {
     server 10.0.0.100:1234;
     server 10.0.0.101:1234;
       #Use keepalive to achieve connection multiplexing
     keepalive 128;
}

• Back to the operating system level, we have an idea to release the connection in the TIME_WAIT state faster by modifying the TIME_WAIT parameter of the system. Is it useful if we apply this parameter? Later, I checked more information and found that this parameter is not as simple as I imagined. The function of the TIME_WAIT parameter needs to be understood in combination with the 4 waves of tcp. It is not as simple as what I wrote above. For details, please refer to : https://blog.51cto.com/u_13291771/2798453 In short, tcp shutdown requires both parties to know that the active party a initiates the shutdown just to ensure that a will not send data to the other party b, and b flows to the direction of data a It does not necessarily have to be sent, so we need to wave 4 times, then we a send a close command, b also send a close command after the data is sent, a also receives and replies the ack, but because of the network problem, the ack is lost this time, so b It is not known if we received the shutdown command from him, as shown in the picture below. Therefore, a cannot release the connection immediately after receiving the closing command from the other party, it will set the connection to the TIME_WAIT state, and will not immediately recycle the connection. We can check it through the netstat command, as shown below:

• 
  

netstat -aonp
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name Timer
tcp 81 0 0.0.0.0:8080 0.0.0.0:* LISTEN 1/java off (0.00/0/0)
tcp 0 0 127.0.0.1:8100 0.0.0.0:* LISTEN 24/soffice.bin off (0.00/0/0)
tcp 932 0 172.20.5.59:40576 172.20.4.203:8080 TIME_WAIT - off (0.00/0/0)
tcp 947 0 172.20.4.172:57166 172.20.4.203:8080 TIME_WAIT - off (0.00/0/0)

At this time, a needs to remain for a period of time to respond to the closing command retransmitted by b. Network: top-down"), TIME_WAIT time is to guide how long a needs to keep this connection alive, so reducing TIME_WAIT can alleviate the situation that the port is exhausted to a certain extent, but the method of reducing the duration of TIME_WAIT is a reliable way to A way to trade sex for performance.

Note : Connections entering TIME_WAIT still require resources.

• tcp_tw_reuse, the function of this parameter is to reuse the socket in TIME_WAIT when a new connection comes in. The default value is 0. Like TIME_WAIT, reliability will be lost.

So how do we solve the problem of running out of ports?

1. Use keepalive in upstream block

2. In fact, this question is answered in the official nginx, original reference: https://www.nginx.com/blog/overcoming-ephemeral-port-exhaustion-nginx-plus or reference: https://www.codenong.com/cs105164127 /

Explanation of the second plan:

Those who understand the composition and operating system of the computer should understand that the io device network card in the computer can be bound to multiple IPs, and many devices also have IPs of different subnets, such as routers. In fact, the high availability of keepalived is also achieved through the network card. To bind multiple IPs to the nginx, we achieve the purpose of expanding the available ports by assigning multiple IPs of the same subnet to the machine where nginx is located.
1. Assign 2 IPs to the network card 
2. Configuration example

http {
     #Upstream service, here we are the port to static resources
     upstream static_server {
         server http://svc.name:8999
      }

   server {
         # ...
         location / {
             # ...
             proxy_pass http://static_server;
             proxy_bind $split_ip;
             proxy_set_header X-Forwarded-For $remote_addr;
         }
    }
     #Use ip by traffic
    split_clients "$remote_addr$remote_port" $split_ip {
         #10% 10.0.0.210;
         #10% 10.0.0.211;
         #10% 10.0.0.212;
         #10% 10.0.0.213;
         #10% 10.0.0.214;
         #10% 10.0.0.215;
         #10% 10.0.0.216;
         #10% 10.0.0.217;
         50% 192.168.1.53;
         * 192.168.1.54;
    }
}

3. Demonstration results

insufficiency

1. At that time, I did not check the network status of the container. I should check the network of the container with netstat, and check the network of the host with netstat to confirm that all ports are occupied in the container or on the host. Why check the network of the host, because the forwarding passes through svc In progress, nginx scheduling will preferentially schedule to the host without the same pod, so there is nginx forwarding to another nginx, and the k8s network will communicate with the other host through the host's nat mode. There are multiple pods on the host, so the host's ports may be used up first.

2. I don't remember if I looked at the resource utilization of the pod and the utilization of the host's resources

3. The positioning problem is not rigorous enough, and the keepalive application does not take effect

4. At that time, I didn't know enough about tcp, and I also had a shallow understanding of nginx.