• notice
  • Congratulations on the launch of the Sought Tech site

How to apply limit to get 10 results

I tried to scrapeimagesthey will give me 23 imagesbut I don't want to apply limitthey will only give me 10 pictures can you help me with these?

 import requests
    from bs4 import BeautifulSoup
    import pandas as pd
    baseurl='https://twillmkt.com'
    headers={
        'User-Agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Safari/537.36'
    }
    r=requests.get('https://twillmkt.com/collections/denim')
    soup=BeautifulSoup(r.content, 'html.parser')
    tra=soup.find_all('div',class_='ProductItem__Wrapper')
    productlinks=[]
    for links in tra:
        for link in links.find_all('a',href=True):
            comp=baseurl link['href']
            productlinks.append(comp)
    
    data=[]
    
    for link in set(productlinks):
        r=requests.get(link,headers=headers)
        soup=BeautifulSoup(r.content, 'html.parser')
        up=soup.find('div',class_='Product__SlideshowNavScroller')
        for e,pro in enumerate(up):
            t=pro.find('img').get('src')
            data.append({'id':t.split('=')[-1], 'image':'Image ' str(e) ' UI','link':t}) 
            
    df=pd.DataFrame(data)
    df.image=pd.Categorical(df.image,categories=df.image.unique(),ordered=True)
    df=df.pivot(index='id', columns='image', values='link').reset_index().fillna('')
    df.to_csv('kj.csv')

uj5u.com enthusiastic netizens replied:

Slice the resulting set of images[:10]

...
up=soup.select('div.Product__SlideshowNavScroller img')[:10]
for e,pro in enumerate(up):
    t=pro.get('src')
    data.append({'id':t.split('=')[-1], 'image':'Image ' str(e) ' UI','link':t}) 
...

If you want to name images starting with 1 instead of 0:

...
up=soup.select('div.Product__SlideshowNavScroller img')[:10]
for e,pro in enumerate(up, start=1):
    t=pro.get('src')
    data.append({'id':t.split('=')[-1], 'image':'Image ' str(e) ' UI','link':t}) 
...

Edit

Ok got it-the behavior is not based on the number of images, the problem here is that the id is not unique, it Not the id/sku of the product.

How to fix it?

Let's select the sku from the product and use it as the id in the data frame:

sku=soup.select_one('.oos_sku').text.strip().split(' ')[-1]
for e,pro in enumerate(up, start=1):
    t=pro.get('src')
    data.append({'id':sku, 'image':'Image ' str(e) ' UI','link':t})

Example

import requests
from bs4 import BeautifulSoup
import pandas as pd
baseurl='https://twillmkt.com'
headers={
    'User-Agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Safari/537.36'
}
r=requests.get('https://twillmkt.com/collections/denim')
soup=BeautifulSoup(r.content, 'html.parser')
tra=soup.find_all('div',class_='ProductItem__Wrapper')
productlinks=[]
for links in tra:
    for link in links.find_all('a',href=True):
        comp=baseurl link['href']
        productlinks.append(comp)

data=[]

for link in set(productlinks):
    r=requests.get(link,headers=headers)
    soup=BeautifulSoup(r.content, 'html.parser')
    up=soup.select('div.Product__SlideshowNavScroller img')
    sku=soup.select_one('.oos_sku').text.strip().split(' ')[-1]
    for e,pro in enumerate(up, start=1):
        t=pro.get('src')
        data.append({'id':sku, 'image':'Image ' str(e) ' UI','link':t})

df=pd.DataFrame(data)
df.image=pd.Categorical(df.image,categories=df.image.unique(),ordered=True)
df=df.pivot(index='id', columns='image', values='link').reset_index().fillna('')
df#.to_excel('test.xlsx')

output

ID th> Figure 1 User Interface Figure 2 User Interface Figure 3 User Interface Figure 4 User Interface Figure 5 User Interface Figure 6 User Interface Figure 7 User Interface Figure 8 User Interface Figure 9 User Interface Figure 10 User Interface Figure 11 User Interface Figure 12 User Interface Figure 13 User Interface Figure 14 User Interface Figure 15 User Interface Figure 16 User Interface Figure 17 User Interface Figure 18 User Interface Figure 19 User Interface Figure 20 User Interface Figure 21 User Interface Figure 22 User Interface Figure 23 User Interface Figure 24 User Interface
0 td> LOTFEELPJ023-30 //cdn.shopify.com/s/files/1/0089/7912/0206/products/Blue-Ripped-Knee-Distressed-Skinny-Denim_160x.jpg?v=1631812617 //cdn.shopify.com/s/files/1/0089/7912/0206/products/Blue-Ripped-Knee-Distressed-Skinny-Denim-2_160x.jpg?v=1631812617 //cdn.shopify.com/s/files/1/0089/7912/0206/products/Blue-Ripped-Knee-Distressed-Skinny-Denim-3_160x.jpg?v=1631812617 //cdn.shopify.com/s/files/1/0089/7912/0206/products/Blue-Ripped-Knee-Distressed-Skinny-Denim-4_160x.jpg ?v=1631812617 //cdn.shopify.com/s/files/1/0089/7912/0206/products/Blue-Ripped-Knee-Distressed-Skinny-Denim-5_160x.jpg ?v=1631812617 //cdn.shopify.com/s/files/1/0089/7912/0206/products/Blue-Ripped-Knee-Distressed-Skinny-Denim-6_160x.jpg ?v=1631812617 //cdn.shopify.com/s/files/1/0089/7912/0206/products/Blue-Ripped-Knee-Distressed-Skinny-Denim-7_160x.jpg ?v=1631812617 //cdn.shopify.com/s/files/1/0089/7912/0206/products/Blue-Ripped-Knee-Distressed-Skinny-Denim-8_160x.jpg ?v=1631812617 //cdn.shopify.com/s/files/1/0089/7912/0206/products/Blue-Ripped-Knee-Distressed-Skinny-Denim-9_160x.jpg ?v=1631812617 //cdn.shopify.com/s/files/1/0089/7912/0206/products/Blue-Ripped-Knee-Distressed-Skinny-Denim-10_160x.jpg ?v=1631812617 //cdn.shopify.com/s/files/1/0089/7912/0206/products/Blue-Ripped-Knee-Distressed-Skinny-Denim-11_160x.jpg?v=1631812617
... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ...
47 td> LOTFEELPJ564-S-BRN font> //cdn.shopify.com/s/files/1/0089/7912/0206/products/LOTFEELPJ564_16_160x.jpg?v=1639467815 //cdn.shopify.com/s/files/1/0089/7912/0206/products/LOTFEELPJ564_17_160x.jpg?v=1639467815 //cdn.shopify.com/s/files/1/0089/7912/0206/products/LOTFEELPJ564_22_160x.jpg?v=1639467815 //cdn.shopify.com/s/files/1/0089/7912/0206/products/LOTFEELPJ564_15_160x.jpg?v=1639467815 //cdn.shopify.com/s/files/1/0089/7912/0206/products/LOTFEELPJ564_6_160x.jpg?v=1639467815 //cdn.shopify.com/s/files/1/0089/7912/0206/products/LOTFEELPJ564_9_160x.jpg?v=1639467815 //cdn.shopify.com/s/files/1/0089/7912/0206/products/sizechart-stretch-pants_3_ec7e0b0c-1043-4306-a766-33f7e0b3edc8_160x.png?v=166994677

Tags

Technical otaku

Sought technology together

Related Topic

0 Comments

Leave a Reply

+