Python network programming socket and socketserver
1. Socket socket programming based on TCP protocol
1. Socket workflow
Let's start with the server side. The server first initializes the Socket, then binds it to the port, listens to the port, calls accept to block, and waits for the client to connect. At this time, if a client initializes a Socket, and then connects to the server (connect), if the connection is successful, then the connection between the client and the server is established. The client sends a data request, the server receives the request and processes the request, then sends the response data to the client, the client reads the data, and finally closes the connection, and an interaction ends, using the following Python code:
import socket # socket_family can be AF_UNIX or AF_INET. socket_type can be SOCK_STREAM or SOCK_DGRAM. protocol is generally not filled, the default value is 0 socket.socket(socket_family, socket_type, protocol=0) # get tcp/ip socket tcpSock = socket.socket(socket.AF_INET, socket.SOCK_STREAM) # get udp/ip socket udpSock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
1. Server socket function
s.bind(): bind (host, port number) to the socket
s.listen(): start TCP listening
s.accept(): passively accept connections from TCP clients, (blocking) waiting for connections to arrive
2. Client socket function
s.connect(): Actively initialize the TCP server connection
s.connect_ex(): An extended version of the connect() function that returns an error code on error instead of throwing an exception
3. Socket functions for public use
s.recv(): Receive TCP data
s.send(): Send TCP data (when the amount of data to be sent is greater than the remaining space in the own-end buffer, the data is lost and will not be sent out)
s.sendall(): Send complete TCP data (the essence is to call send in a loop. When the amount of data to be sent is greater than the remaining space in the buffer at its own end, the data is not lost, and send is called in a loop until it is sent out)
s.recvfrom(): Receive UDP data
s.sendto(): send UDP data
s.getpeername(): the address of the remote end connected to the current socket
s.getsockname(): the address of the current socket
s.getsockopt(): returns the parameters of the specified socket
s.setsockopt(): Set the parameters of the specified socket
s.close(): close the socket
4. Lock-oriented socket method
s.setblocking(): Set the blocking and non-blocking mode of the socket
s.settimeout(): Set the timeout for blocking socket operations
s.gettimeout(): Get the timeout for blocking socket operations
5. Functions of file-oriented sockets
s.fileno(): the file descriptor of the socket
s.makefile(): create a file associated with the socket
2. Socket programming based on TCP protocol
You can netstat -an | findstr 8080
check the socket status by
1. Server
import socket # 1. Buy a mobile phone phone = socket.socket(socket.AF_INET, socket.SOCK_STREAM) # tcp is called streaming protocol, udp is called datagram protocol SOCK_DGRAM # print(phone) # 2. Insert/bind mobile phone card # phone.setsockopt(socket.SOL_SOCKET,socket.SO_REUSEADDR,1) phone.bind(("127.0.0.1", 8080)) # 3. Boot phone.listen(5) # Semi-connection pool, limiting the number of requests # 4. Wait for the phone to connect print("start....") while True: # join loop conn, client_addr = phone.accept() # (two-way connection established by three-way handshake, (client's ip, port)) # print(conn) print("A connection has been established successfully", client_addr) # 5. Communication: Send and receive messages while True: # communication loop try: print("The server is receiving data...") data = conn.recv(1024) # The maximum number of bytes received, no data will be waiting to be received in place, that is, the amount of data sent by the sender must be > 0bytes # print("===>") if len(data) == 0: break # Only when the client disconnects unilaterally, the server will receive empty data print("Data from client", data) conn.send(data.upper()) except ConnectionResetError: break # 6. Hang up the phone connection conn.close() # 7. Shut down phone.close() #start.... # A connection has been established successfully ("127.0.0.1", 4065) # The server is receiving data... # data from client b"xad" # The server is receiving data...
2. Client
import socket # 1. Buy a mobile phone phone = socket.socket(socket.AF_INET, socket.SOCK_STREAM) # print(phone) # 2. Dial the phone phone.connect(("127.0.0.1", 8080)) # Specify the server ip and port # 3. Communication: send and receive messages while True: # communication loop msg = input(">>: ").strip() # msg="" if len(msg) == 0: continue phone.send(msg.encode("utf-8")) # print("has send----->") data = phone.recv(1024) # print("has recv----->") print(data) # 4. Close phone.close() # >>: ah # b"a" # >>: ah ah # b"xb0xa1xb0xa1" # >>:
3. Address occupation problem
This is because your server still has the time_wait state of four waved hands occupying the address (if you do not understand, please in-depth study 1.tcp three-way handshake, four waved hands 2.syn flood attack 3. In the case of high server concurrency, there will be a large number of optimization method of the time_wait state)
1. Method 1: Add a socket configuration and reuse ip and port
phone=socket(AF_INET,SOCK_STREAM) phone.setsockopt(SOL_SOCKET,SO_REUSEADDR,1) #That's it, add it before bind phone.bind(("127.0.0.1",8080))
2. Method 2: By adjusting the linux kernel parameters
It is found that there are a large number of connections in the TIME_WAIT state in the system, which can be solved by adjusting the Linux kernel parameters. vi /etc/sysctl.conf Edit the file and add the following: net.ipv4.tcp_syncookies=1 net.ipv4.tcp_tw_reuse=1 net.ipv4.tcp_tw_recycle=1 net.ipv4.tcp_fin_timeout=30 Then execute /sbin/sysctl -p to make the parameters take effect. net.ipv4.tcp_syncookies = 1 means enable SYN Cookies. When the SYN waiting queue overflows, enable cookies to deal with it, which can prevent a small number of SYN attacks. The default value is 0, which means it is closed; net.ipv4.tcp_tw_reuse = 1 means to enable reuse. Allow TIME-WAIT sockets to be reused for new TCP connections, the default is 0, which means close; net.ipv4.tcp_tw_recycle = 1 means to enable fast recycling of TIME-WAIT sockets in TCP connections, the default is 0, which means close. net.ipv4.tcp_fin_timeout Modify the default TIMEOUT time of the system
4. Simulate ssh remote command execution
The server executes the command through subprocess, and then returns the result of the command.
Server:
from socket import * import subprocess server = socket(AF_INET, SOCK_STREAM) server.bind(("127.0.0.1", 8000)) server.listen(5) print("start...") while True: conn, client_addr = server.accept() while True: print("from client:", client_addr) cmd = conn.recv(1024) if len(cmd) == 0: break print("cmd:", cmd) obj = subprocess.Popen(cmd.decode("utf8"), # cmd command entered shell=True, # run through the shell stderr=subprocess.PIPE, # pipe error output for printing stdout=subprocess.PIPE) # pipe the correct output for printing stdout = obj.stdout.read() # print the correct output stderr = obj.stderr.read() # print error output conn.send(stdout) conn.send(stderr) conn.close() server.close()
client
import socketclient = socket.socket(socket.AF_INET, socket.SOCK_STREAM)client.connect(("127.0.0.1", 8000))while True:data = input("please enter your data")client.send(data.encode("utf8"))data = client.recv(1024)print("from server:", data)client.close()
Enter dir
the command. Since the server sends less than 1024 bytes, the client can accept it.
Enter tasklist
the command, because the server sends more than 1024 bytes, the client only accepts part of the data, and when you enter dir
the command again, the client will receive dir
the result of the command, but will print the remaining unsent last time. Data, this is the sticky package problem.
5. Sticky bag
1. The sender needs to wait for the buffer to be full before sending out, resulting in sticky packets
The time interval for sending data is very short, and the amount of data is small, and they are combined together to generate sticky packets.
Server
# _*_coding:utf-8_*_from socket import *ip_port = ("127.0.0.1", 8080)TCP_socket_server = socket(AF_INET, SOCK_STREAM)TCP_socket_server.bind(ip_port)TCP_socket_server.listen(5)conn, addr = TCP_socket_server.accept()data1 = conn.recv(10)data2 = conn.recv(10)print("----->", data1.decode("utf-8"))print("----->", data2.decode("utf-8"))conn.close()
client
# _*_coding:utf-8_*_import socketBUFSIZE = 1024ip_port = ("127.0.0.1", 8080)s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)res = s.connect_ex(ip_port)s.send("hello".encode("utf-8"))s.send("world".encode("utf-8"))# The server receives b"helloworld" together
2. The receiver does not receive the packets in the buffer in time, resulting in multiple packet reception
The client sends a piece of data, but the server only receives a small part. When the server receives the data next time, it still takes the data left over from the last time from the buffer to generate sticky packets.
Server
# _*_coding:utf-8_*_from socket import *ip_port = ("127.0.0.1", 8080)TCP_socket_server = socket(AF_INET, SOCK_STREAM)TCP_socket_server.bind(ip_port)TCP_socket_server.listen(5)conn, addr = TCP_socket_server.accept()data1 = conn.recv(2) # did not receive a completedata2 = conn.recv(10) # When receiving the next time, the old data will be taken first, and then the new one will be takenprint("----->", data1.decode("utf-8"))print("----->", data2.decode("utf-8"))conn.close()
client
# _*_coding:utf-8_*_import socketBUFSIZE = 1024ip_port = ("127.0.0.1", 8080)s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)res = s.connect_ex(ip_port)s.send("hello feng".encode("utf-8"))
6. Solve the sticky package problem
1. The total size of the byte stream sent first (low version)
The root of the problem is that the receiver does not know the length of the byte stream that the sender will transmit, so the solution to the sticky packet is how to let the sender let the receiver send the total size of the byte stream it will send before sending the data. The end knows, and then the receiving end receives all the data in an infinite loop.
Why low: The running speed of the program is much faster than the network transmission speed, so before sending a segment of bytes, use send to send the length of the byte stream, which will amplify the performance loss caused by network delay.
Server:
import socket, subprocessserver = socket.socket (socket.AF_INET, socket.SOCK_STREAM)server.bind(("127.0.0.1", 8000))server.listen(5)while True:conn, addr = server.accept()print("start...")while True:cmd = conn.recv(1024)print("cmd:", cmd)obj = subprocess.Popen(cmd.decode("utf8"),shell=True,stderr=subprocess.PIPE,stdout=subprocess.PIPE)stdout = obj.stdout.read()if stdout:ret = stdoutelse:stderr = obj.stderr.read()ret = stderrret_len = len (ret) conn.send(str(ret_len).encode("utf8"))data = conn.recv(1024).decode("utf8")if data == "recv_ready":conn.sendall(ret)conn.close()server.close()
Client:
import socketclient = socket.socket(socket.AF_INET, socket.SOCK_STREAM)client.connect(("127.0.0.1", 8000))while True:msg = input("please enter your cmd you want>>>").strip()if len(msg) == 0: continueclient.send(msg.encode("utf8"))length = int(client.recv(1024))client.send("recv_ready".encode("utf8"))send_size = 0recv_size = 0data = b""while recv_size < length:data = client.recv(1024)recv_size += len(data)print(data.decode("utf8"))
2. Custom fixed-length header (struct module)
struct module analysis
import structimport json# "i" is the formattry:obj = struct.pack("i", 1222222222223)except Exception as e:print(e)obj = struct.pack("i", 1222)print (obj, len (obj))# "i" format requires -2147483648 <= number <= 2147483647# b"xc6x04x00x00" 4res = struct.unpack("i", obj)print(res[0])# 1222
The core of solving the sticky packet problem is to add a custom fixed-length header to the byte stream, and the header contains the length of the byte stream, and then send it to the peer at one time. When the peer receives it, it first takes out the fixed-length header from the cache. , and then take the real data.
1. Use the struct module to create a header:
import jsonimport structheader_dic = {"filename": "a.txt","total_size":111111111111111111111111111111111222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222223131232,"hash": "asdf123123x123213x"}header_json = json.dumps(header_dic)header_bytes = header_json.encode("utf-8")print(len(header_bytes))# 223# "i" is the formatobj = struct.pack("i", len(header_bytes))print (obj, len (obj))# b"xdfx00x00x00" 4res = struct.unpack("i", obj)print(res[0])# 223
2. Server:
from socket import *import subprocessimport structimport jsonserver = socket(AF_INET, SOCK_STREAM)server.bind(("127.0.0.1", 8000))server.listen(5)print("start...")while True:conn, client_addr = server.accept()print(conn, client_addr)while True:cmd = conn.recv(1024)obj = subprocess.Popen(cmd.decode("utf8"),shell=True,stderr=subprocess.PIPE,stdout=subprocess.PIPE)stderr = obj.stderr.read()stdout = obj.stdout.read()# make headerheader_dict = {"filename": "a.txt","total_size": len(stdout) + len(stderr),"hash": "xasf123213123"}header_json = json.dumps(header_dict)header_bytes = header_json.encode("utf8")# 1. First package the length of the header len(header_bytes) into 4 bytes, and then sendconn.send(struct.pack("i", len(header_bytes)))# 2. Send header conn.send(header_bytes)# 3. Send real data conn.send(stdout)conn.send(stderr)conn.close()server.close()
3. Client:
from socket import *import jsonimport structclient = socket(AF_INET, SOCK_STREAM)client.connect(("127.0.0.1", 8000))while True:cmd = input("please enter your cmd you want>>>")if len(cmd) == 0: continueclient.send(cmd.encode("utf8"))# 1. Receive 4 bytes first, these 4 bytes contain the length of the headerheader_len = struct.unpack("i", client.recv(4))[0]# 2. Receive the header againheader_bytes = client.recv(header_len)# 3. Parse what you want from the headerheader_json = header_bytes.decode("utf8")header_dict = json.loads(header_json)total_size = header_dict["total_size"]# 4. Re-receive real datarecv_size = 0res = b""while recv_size < total_size:data = client.recv(1024)res += datarecv_size += len(data)print(res.decode("utf8"))client.close()
2. Socket socket programming based on UDP protocol
UDP is unconnected, whichever side is started first will not report an error, and multiple clients can communicate with the server at the same time
The UDP protocol is a datagram protocol. When it is empty, it will also bring its own header. Therefore, if the client input is empty, the server can also receive it.
The UPD protocol is generally not used to transmit large data.
UPD socket has no sticky packet problem, but cannot replace TCP socket, because UPD protocol has a defect: if data is lost in the way of data transmission, the data will be lost, while TCP protocol will not have this defect, So generally UPD socket users don't care about data sending, such as qq chat.
Simple example of UDP socket
1. Server
import socketserver = socket.socket(socket.AF_INET, socket.SOCK_DGRAM) # Datagram protocol-"UDPserver.bind(("127.0.0.1", 8080))while True:data, client_addr = server.recvfrom(1024)print("===>", data, client_addr)server.sendto(data.upper(), client_addr)server.close()
2. Client
import socketclient = socket.socket(socket.AF_INET, socket.SOCK_DGRAM) # Datagram protocol-"UDPwhile True:msg = input(">>: ").strip() # msg=""client.sendto(msg.encode("utf-8"), ("127.0.0.1", 8080))data, server_addr = client.recvfrom(1024)print(data)client.close()
3. Implement concurrent socket programming based on socketserver
1. Based on TCP protocol
The key to tcp-based sockets is two loops, a link loop and a communication loop
The socketserver module is divided into two categories: the server class (to solve the link problem) and the request class (to solve the communication problem).
1. server class
2. Request class
Based on tcp socketserver in our own defined class.
self.server is the socket object
self.request is a link
self.client_address is the client address
3. Server
import socketserverclass MyHandler(socketserver.BaseRequestHandler):def handle(self):# communication loopwhile True:# print(self.client_address)# print(self.request) #self.request=conntry:data = self.request.recv(1024)if len(data) == 0: breakself.request.send(data.upper())except ConnectionResetError:breakif __name__ == "__main__":s = socketserver.ThreadingTCPServer(("127.0.0.1", 8080), MyHandler, bind_and_activate=True)s.serve_forever() # represents the connection loop# Loop to establish a connection, each time a connection is established, a thread (waiter) is started + an object is generated by calling the Myhanlder class, and the handle method under the object is called to communicate with the newly established connection.
4. Client
import socketphone = socket.socket(socket.AF_INET, socket.SOCK_STREAM)phone.connect(("127.0.0.1", 8080)) # Specify the server ip and portwhile True:# msg=input(">>: ").strip() #msg=""msg = "client33333" # msg=""if len(msg) == 0: continuephone.send(msg.encode("utf-8"))data = phone.recv(1024)print(data)phone.close()
2. Based on UDP protocol
udp-based socketserver in our own defined class
self.request is a tuple (the first element is the data sent by the client, and the second part is the udp socket object of the server), such as (b'adsf', )
self.client_address is the client address
1. Server
import socketserverclass MyHandler(socketserver.BaseRequestHandler):def handle(self):# communication loopprint(self.client_address)print(self.request)data = self.request[0]print("Customer message", data)self.request[1].sendto(data.upper(), self.client_address)if __name__ == "__main__":s = socketserver.ThreadingUDPServer(("127.0.0.1", 8080), MyHandler)s.serve_forever()
2. Client
import socketclient = socket.socket(socket.AF_INET, socket.SOCK_DGRAM) # Datagram protocol-"udpwhile True:# msg=input(">>: ").strip() #msg=""msg = "client1111"client.sendto(msg.encode("utf-8"), ("127.0.0.1", 8080))data, server_addr = client.recvfrom(1024)print(data)client.close()
4. Python Internet Module
Some important modules of Python network programming are listed below:
protocol | Functional use | The port number | Python modules |
---|---|---|---|
HTTP | web access | 80 | httplib, urllib, xmlrpclib |
NNTP | Read and post news articles, commonly known as "posts" | 119 | nntplib |
FTP | file transfer | 20 | ftplib, urllib |
SMTP | send email | 25 | smtplib |
POP3 | incoming mail | 110 | poplib |
IMAP4 | get mail | 143 | imaplib |
Telnet | Command Line | twenty three | telnetlib |
Gopher | Information lookup | 70 | gopherlib, urllib |
buy atorvastatin 20mg sale & lt;a href="https://lipiws.top/"& gt;order atorvastatin 20mg pills& lt;/a& gt; buy lipitor 20mg sale
Rqlckn
2024-03-09