• notice
  • Congratulations on the launch of the Sought Tech site

Remember a large area of ​​404 problems in the production environment!

write in front

The interface services published online have always been good. Today, the operation feedback suddenly said that many functions cannot be used normally. After investigation, it was found that when the front-end called the back-end interface, some interfaces appeared 404. Today, I arrived at the company late. It must have been a little friend who got off work last night and didn't pray that the server would not go wrong before walking out of the office. To find this man and hang it on the server - sacrifice to heaven!

Articles have been included in:



Problem recurrence

After learning the feedback from the operation, I quickly logged into the server to troubleshoot the problem. First, check that the startup process of the interface service is normal. Verify that the ip and port of the interface service are normal, and the result is no problem. Next, forward the request through Nginx, at which point there is a problem and the interface cannot be accessed. At the same time, the following log information is output in the access.log file of Nginx. - - [26/Feb/2021:21:34:21 +0800] "GET /third/system/base/thirdapp/get_detail HTTP/1.1" 404 0 "" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:85.0) Gecko/20100101 Firefox/85.0" - - [26/Feb/2021:21:34:22 +0800] "GET /third/system/base/thirdapp/get_detail HTTP/1.1" 404 0 "" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:85.0) Gecko/20100101 Firefox/85.0" - - [26/Feb/2021:21:34:26 +0800] "GET /third/system/base/thirdapp/get_detail HTTP/1.1" 404 0 "" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:85.0) Gecko/20100101 Firefox/85.0"

At this time, it is found from the Nginx log that the output status is 404, and the back-end interface service was not found. curlIn order to further locate the problem, I directly access the interface service through commands in the online environment , and the result is normal.

After this series of operations, we can determine that the problem is with Nginx.

problem analysis

Nginx enables debug module

Now that the problem has been located, let's analyze the specific reasons for the problem. Since it is an Nginx problem, my first thought is to debug Nginx to find the cause of the error. So I entered the following command on the server command line to view the configuration when installing Nginx.

nginx -V

Note: The system environment variables have been configured for Nginx here. If the system environment variables are not configured, you need to enter the full path of the directory where the nginx command is located, for example:

/usr/local/nginx/sbin/nginx -v

The command line outputs the following information.

configure arguments: --prefix=/usr/local/nginx --with-http_stub_status_module --add-module=/usr/local/src/fastdfs/fastdfs-nginx-module-1.22/src --with-openssl=/usr/local/src/openssl-1.0.2s --with-pcre=/usr/local/src/pcre-8.43 --with-zlib=/usr/local/src/zlib-1.2.11 --with-http_ssl_module

It can be seen that the debug module of Nginx is not configured when Nginx is installed.

So I found the Nginx installation file on the server, and entered the following command on the command line to recompile Nginx.

cd /usr/local/src/nginx/ #Enter the root directory of the Nginx installation file
make clean #Clear compilation information
./configuration --prefix=/usr/local/nginx-1.17.8 --with-http_stub_status_module --add-module=/usr/local/src/fastdfs/fastdfs-nginx-module-1.22/src --with- openssl=/usr/local/src/openssl-1.0.2s --with-pcre=/usr/local/src/pcre-8.43 --with-zlib=/usr/local/src/zlib-1.2.11 -- with-http_ssl_module --with-debug #Set the configuration information for compiling Nginx
make #Compile Nginx, remember not to enter make install

In the above command, remember not to enter make installto install.

After executing the makecommand , the nginx command will be generated in the objs directory of the current directory. At this time, we need to stop the Nginx service, backup the nginx command in the /usr/local/nginx/sbin/ directory, and then copy the nginx command in the objs directory. Copy it to the /usr/local/nginx/sbin/ directory, and then start the Nginx service.

nginx_service.sh stop #Stop Nginx service through script
mv /usr/local/nginx/sbin/nginx /usr/local/nginx/sbin/nginx.bak #Back up the original nginx command
cp ./objs/nginx /usr/local/nginx/sbin/nginx #Copy nginx command
nginx_service.sh start #Start Nginx service through script

Note: Here, before stopping the Nginx service, this Nginx has been removed from the access layer gateway, so it will not affect the online environment. In order to avoid the problem of restarting Nginx with the newly compiled nginx command, the Nginx service is stopped first through the script, and then the Nginx service is started after copying the nginx command.

Configure Nginx to output debug logs

Configure the following information in the nginx.conf file of Nginx.

error_log  logs/error.log debug;

At this point, the debug log function of Nginx is enabled, and the debug information is output to the error.log file.

analyse problem

Next, enter the following command on the server command line to monitor the output log of the error.log file.

tail -F /usr/local/nginx/logs/error.log

Then simulate accessing the http interface, and you can see the following information is output in the error.log file.

2021/02/26 21:34:26 [debug] 31486#0: *56 http request line: "GET /third/system/base/thirdapp/get_detail HTTP/1.1"2021/02/26 21:34:26 [debug] 31486#0: *56 http uri: "/third/system/base/thirdapp/get_detail"2021/02/26 21:34:26 [debug] 31486#0: *56 http args: ""2021/02/26 21:34:26 [debug] 31486#0: *56 http exten: ""2021/02/26 21:34:26 [debug] 31486#0: *56 posix_memalign: 0000000000FF6450:4096 @162021/02/26 21:34:26 [debug] 31486#0: *56 http process request header line2021/02/26 21:34:26 [debug] 31486#0: *56 http header: "Host:"2021/02/26 21:34:26 [debug] 31486#0: *56 http header: "User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:85.0) Gecko/20100101 Firefox/85.0"2021/02/26 21:34:26 [debug] 31486#0: *56 http header: "Accept: */*"2021/02/26 21:34:26 [debug] 31486#0: *56 http header: "Accept-Language: zh-CN,zh;q=0.8,zh-TW;q=0.7,zh-HK;q=0.5,en-US;q=0.3,en;q=0.2"2021/02/26 21:34:26 [debug] 31486#0: *56 http header: "Accept-Encoding: gzip, deflate"2021/02/26 21:34:26 [debug] 31486#0: *56 http header: "Referer:"2021/02/26 21:34:26 [debug] 31486#0: *56 http header: "Connection: keep-alive"2021/02/26 21:34:26 [debug] 31486#0: *56 http header done2021/02/26 21:34:26 [debug] 31486#0: *56 rewrite phase: 02021/02/26 21:34:26 [debug] 31486#0: *56 test location: "/"2021/02/26 21:34:26 [debug] 31486#0: *56 test location: "file/"2021/02/26 21:34:26 [debug] 31486#0: *56 test location: ~ "/base"2021/02/26 21:34:26 [debug] 31486#0: *56 using configuration "/base"

From the above output log, we can see that the accessed interface address is "/third/system/base/thirdapp/get_detail", as shown below.

2021/02/26 21:34:26 [debug] 31486#0: *56 http uri: "/third/system/base/thirdapp/get_detail"

When Nginx forwards, it matches "/", "file/", and "~/base" respectively, and finally forwards the request to "/base", as shown below.

2021/02/26 21:34:26 [debug] 31486#0: *56 test location: "/"2021/02/26 21:34:26 [debug] 31486#0: *56 test location: "file/"2021/02/26 21:34:26 [debug] 31486#0: *56 test location: ~ "/base"2021/02/26 21:34:26 [debug] 31486#0: *56 using configuration "/base"

Let's take a look at the configuration of Nginx again, open the nginx.conf file, and find the following configuration.

location ~/base {
 proxy_pass                  http://base;
 proxy_set_header Host $host:$server_port;}location ~/third {
 proxy_pass                  http://third;
 proxy_set_header Host $host:$server_port;}

So here comes the question, the interface to be accessed is obviously "/third/system/base/thirdapp/get_detail", why does it go under "/base"?

Having said that, I believe that the careful friends have found the problem, yes, it is the pot of operation and maintenance! !

Solve the problem

After reading the configuration of Nginx, I believe that many small partners should know how to solve the problem. Yes, that is to configure the following configuration in nginx.conf.

location ~/base {
 proxy_pass                  http://base;
 proxy_set_header Host $host:$server_port;}location ~/third {
 proxy_pass                  http://third;
 proxy_set_header Host $host:$server_port;}

Modified as shown below.

location /base {
 proxy_pass                  http://base;
 proxy_set_header Host $host:$server_port;}location /third {
 proxy_pass                  http://third;
 proxy_set_header Host $host:$server_port;}

Remove the "~" symbol.

Next, simulate accessing the http interface again, and the interface can be accessed normally.

Next, turn off the debug function of Nginx, that is, comment out the error_log logs/error.log debug;configuration , as shown below.

# error_log  logs/error.log debug;

Reload the nginx.conf file.

nginx_service.sh reload

Finally, adding Nginx to the access layer gateway solved the problem.

The forwarding rules of popular science Nginx

Nginx's location syntax

location [=|~|~*|^~] /uri/ { … }
  • = exact match. If the request matches this location, the search will stop and the request will be processed immediately

  • ~ case sensitive matching (regular expression available)

  • ~* case-insensitive match (regular expression available)

  • !~ case-sensitive mismatch

  • !~* case insensitive mismatch

  • ^~ if this prefix is used for a regular string, tells nginx not to test the regular expression if the path matches

Example 1:

location  / { }

match any request

Example 2:

location ~* .(gif|jpg|jpeg)$ {
   rewrite .(gif|jpg|jpeg)$ /logo.png;}

Case-insensitively match any request ending in gif, jpg, jpeg and redirect that request to /logo.png request

Example 3:

location ~ ^.+\.txt$ {
   root /usr/local/nginx/html/;}

Case-sensitively matches requests ending in .txt, and sets the path of this location to /usr/local/nginx/html/. That is, requests ending in .txt will access the txt file under the /usr/local/nginx/html/ path

The difference between alias and root

  • root actually accessing the file path will concatenate the path in the URL

  • alias actually accessing the file path will not concatenate the path in the URL

An example is as follows:

location ^~ /binghe/ {  
  alias /usr/local/nginx/html/binghetic/;  }
  • Request: http://test.com/binghe/binghe1.html

  • Actual access: /usr/local/nginx/html/binghetic/binghe1.html file

location ^~ /binghe/ {  
  root /usr/local/nginx/html/;  }
  • Request: http://test.com/binghe/binghe1.html

  • Actual access: /usr/local/nginx/html/binghe/binghe1.html file

Difference between last and break keywords

(1) When last and break appear outside the location, the roles of the two are the same without any difference

(2) last and break when appearing inside the location:

  • last uses the last instruction, after rewriting, it will jump out of the location scope, and start over and go through the previous behavior again.

  • Break uses the break instruction, which will not jump out of the location scope after rewrite, and its entire life cycle is in the current location.

The difference between permanent and redirect keywords

  • rewrite … permanent Permanent redirect, status code 301 in the request log

  • rewrite … redirect Temporary redirection, the status code in the request log is 302

Comprehensive example

Redirect URLs matching a regular expression to a fixed page

For example: we need to redirect the URL matching the regular expression "/test/(\d+)/[\w-.]+" to a fixed page. Pages matching this regular expression might be: http://test.com/test/12345/abc122.html, http://test.com/test/456/11111cccc.js, etc.

As can be seen from the above introduction, we can use rewrite redirection or alias keyword to achieve our purpose. So here it can be done:

(1) Use the rewrite keyword

location ~ ^.+\.txt$ {
   root /usr/local/nginx/html/;}location ~* ^/test/(\d+)/[\w-\.]+$ {
   rewrite ^/test/(\d+)/[\w-\.]+$ /testpage.txt last;}

Here all qualified URLs (PS: case insensitive) are redirected to /testpage.txt requests, which is /usr/local/nginx/html/testpage.txt file

(2) Use the alias keyword

location ~* ^/test/(\d+)/[\w-\.]+$ {    alias /usr/local/nginx/html/binghetic/binghe1.html;}

This redirects all eligible URLs (case insensitive) to /usr/local/nginx/html/binghetic/binghe1.html file


Technical otaku

Sought technology together

Related Topic


Leave a Reply