How to apply limit to get 10 results
I tried to scrapeimages
they will give me 23 images
but I don't want to apply limit
they will only give me 10 pictures can you help me with these?
import requests
from bs4 import BeautifulSoup
import pandas as pd
baseurl='https://twillmkt.com'
headers={
'User-Agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Safari/537.36'
}
r=requests.get('https://twillmkt.com/collections/denim')
soup=BeautifulSoup(r.content, 'html.parser')
tra=soup.find_all('div',class_='ProductItem__Wrapper')
productlinks=[]
for links in tra:
for link in links.find_all('a',href=True):
comp=baseurl link['href']
productlinks.append(comp)
data=[]
for link in set(productlinks):
r=requests.get(link,headers=headers)
soup=BeautifulSoup(r.content, 'html.parser')
up=soup.find('div',class_='Product__SlideshowNavScroller')
for e,pro in enumerate(up):
t=pro.find('img').get('src')
data.append({'id':t.split('=')[-1], 'image':'Image ' str(e) ' UI','link':t})
df=pd.DataFrame(data)
df.image=pd.Categorical(df.image,categories=df.image.unique(),ordered=True)
df=df.pivot(index='id', columns='image', values='link').reset_index().fillna('')
df.to_csv('kj.csv')
uj5u.com enthusiastic netizens replied:
Slice the resulting set of images[:10]
...
up=soup.select('div.Product__SlideshowNavScroller img')[:10]
for e,pro in enumerate(up):
t=pro.get('src')
data.append({'id':t.split('=')[-1], 'image':'Image ' str(e) ' UI','link':t})
...
If you want to name images starting with 1 instead of 0:
...
up=soup.select('div.Product__SlideshowNavScroller img')[:10]
for e,pro in enumerate(up, start=1):
t=pro.get('src')
data.append({'id':t.split('=')[-1], 'image':'Image ' str(e) ' UI','link':t})
...
Edit
Ok got it-the behavior is not based on the number of images, the problem here is that the id is not unique, it Not the id/sku of the product.
How to fix it?
Let's select the sku from the product and use it as the id in the data frame:
sku=soup.select_one('.oos_sku').text.strip().split(' ')[-1]
for e,pro in enumerate(up, start=1):
t=pro.get('src')
data.append({'id':sku, 'image':'Image ' str(e) ' UI','link':t})
Example
import requests
from bs4 import BeautifulSoup
import pandas as pd
baseurl='https://twillmkt.com'
headers={
'User-Agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Safari/537.36'
}
r=requests.get('https://twillmkt.com/collections/denim')
soup=BeautifulSoup(r.content, 'html.parser')
tra=soup.find_all('div',class_='ProductItem__Wrapper')
productlinks=[]
for links in tra:
for link in links.find_all('a',href=True):
comp=baseurl link['href']
productlinks.append(comp)
data=[]
for link in set(productlinks):
r=requests.get(link,headers=headers)
soup=BeautifulSoup(r.content, 'html.parser')
up=soup.select('div.Product__SlideshowNavScroller img')
sku=soup.select_one('.oos_sku').text.strip().split(' ')[-1]
for e,pro in enumerate(up, start=1):
t=pro.get('src')
data.append({'id':sku, 'image':'Image ' str(e) ' UI','link':t})
df=pd.DataFrame(data)
df.image=pd.Categorical(df.image,categories=df.image.unique(),ordered=True)
df=df.pivot(index='id', columns='image', values='link').reset_index().fillna('')
df#.to_excel('test.xlsx')
output
ID th> | Figure 1 User Interface | Figure 2 User Interface | Figure 3 User Interface | Figure 4 User Interface | Figure 5 User Interface | Figure 6 User Interface | Figure 7 User Interface | Figure 8 User Interface | Figure 9 User Interface | Figure 10 User Interface | Figure 11 User Interface | Figure 12 User Interface | Figure 13 User Interface | Figure 14 User Interface | Figure 15 User Interface | Figure 16 User Interface | Figure 17 User Interface | Figure 18 User Interface | Figure 19 User Interface | Figure 20 User Interface | Figure 21 User Interface | Figure 22 User Interface | Figure 23 User Interface | Figure 24 User Interface | |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
0 td> | LOTFEELPJ023-30 | //cdn.shopify.com/s/files/1/0089/7912/0206/products/Blue-Ripped-Knee-Distressed-Skinny-Denim_160x.jpg?v=1631812617 | //cdn.shopify.com/s/files/1/0089/7912/0206/products/Blue-Ripped-Knee-Distressed-Skinny-Denim-2_160x.jpg?v=1631812617 | //cdn.shopify.com/s/files/1/0089/7912/0206/products/Blue-Ripped-Knee-Distressed-Skinny-Denim-3_160x.jpg?v=1631812617 | //cdn.shopify.com/s/files/1/0089/7912/0206/products/Blue-Ripped-Knee-Distressed-Skinny-Denim-4_160x.jpg ?v=1631812617 | //cdn.shopify.com/s/files/1/0089/7912/0206/products/Blue-Ripped-Knee-Distressed-Skinny-Denim-5_160x.jpg ?v=1631812617 | //cdn.shopify.com/s/files/1/0089/7912/0206/products/Blue-Ripped-Knee-Distressed-Skinny-Denim-6_160x.jpg ?v=1631812617 | //cdn.shopify.com/s/files/1/0089/7912/0206/products/Blue-Ripped-Knee-Distressed-Skinny-Denim-7_160x.jpg ?v=1631812617 | //cdn.shopify.com/s/files/1/0089/7912/0206/products/Blue-Ripped-Knee-Distressed-Skinny-Denim-8_160x.jpg ?v=1631812617 | //cdn.shopify.com/s/files/1/0089/7912/0206/products/Blue-Ripped-Knee-Distressed-Skinny-Denim-9_160x.jpg ?v=1631812617 | //cdn.shopify.com/s/files/1/0089/7912/0206/products/Blue-Ripped-Knee-Distressed-Skinny-Denim-10_160x.jpg ?v=1631812617 | //cdn.shopify.com/s/files/1/0089/7912/0206/products/Blue-Ripped-Knee-Distressed-Skinny-Denim-11_160x.jpg?v=1631812617 | |||||||||||||
... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... |
47 td> | LOTFEELPJ564-S-BRN font> | //cdn.shopify.com/s/files/1/0089/7912/0206/products/LOTFEELPJ564_16_160x.jpg?v=1639467815 | //cdn.shopify.com/s/files/1/0089/7912/0206/products/LOTFEELPJ564_17_160x.jpg?v=1639467815 | //cdn.shopify.com/s/files/1/0089/7912/0206/products/LOTFEELPJ564_22_160x.jpg?v=1639467815 | //cdn.shopify.com/s/files/1/0089/7912/0206/products/LOTFEELPJ564_15_160x.jpg?v=1639467815 | //cdn.shopify.com/s/files/1/0089/7912/0206/products/LOTFEELPJ564_6_160x.jpg?v=1639467815 | //cdn.shopify.com/s/files/1/0089/7912/0206/products/LOTFEELPJ564_9_160x.jpg?v=1639467815 | //cdn.shopify.com/s/files/1/0089/7912/0206/products/sizechart-stretch-pants_3_ec7e0b0c-1043-4306-a766-33f7e0b3edc8_160x.png?v=166994677 |
0 Comments