Jump to content

Python Webscraping


Recommended Posts

Is anyone using their own Python webscraping to look for online deals? I was thinking of paying someone to build something for me. The paid versions I think are geared towards the developers making the most money possible, and not so much about you finding all the deals - plus they scrape so much so often, that they probably have to scrape less often than an individual would to keep them from getting banned. I would love to hear someone's thoughts or experiences. I'm also open to collaboration. 

Link to comment
Share on other sites

I have some limited experience building things in Python. It's not too bad. However, not getting caught by the websites or security measures is.

 

I tried to write an auto-buy script for Amazon. I could have gotten it working except I tripped the account protection thing and also the website flagged my script as a bot (which it was). There are ways to get around the latter (and probably the first as well). A professional should definitely be able to help with both. Otherwise, it's going to take a lot of time, research, and trial and error. 

 

I also wrote an inventory checker for Walmart which was successful. 

Link to comment
Share on other sites

I would love a better inventory checker than ********. Was that hard to figure out? I can easily explain the price scaper to a python guy. But i have no idea how ******** even searches for walmart inventory.

I am aware of the websites blocking people for pinging to often. I think there are some parameters online that tells you not to ping more than x number of minutes.. etc. 

Link to comment
Share on other sites

14 hours ago, president89 said:

I would love a better inventory checker than ********. Was that hard to figure out? I can easily explain the price scaper to a python guy. But i have no idea how ******** even searches for walmart inventory.

I am aware of the websites blocking people for pinging to often. I think there are some parameters online that tells you not to ping more than x number of minutes.. etc. 

The Walmart inventory checker is not. You just use their publicly available, mobile API and have the program parse through the response and pick out out the price. 

 

For a scaper, the quantity of pings is important. However, even with a few, you are still likely to get caught by the websites. You have to try to mimic a user to try to fool the website into thinking you are just a normal user. This includes using fake headers on the requests sent to the website. This also includes using a proxy and rotating through a set of proxies. Those two things are the hard part. 

Edited by bigboy61
Link to comment
Share on other sites

1 hour ago, bigboy61 said:

The Walmart inventory checker is not. You just use their publicly available, mobile API and have the program parse through the response and pick out out the price. 

 

For a scaper, the quantity of pings is important. However, even with a few, you are still likely to get caught by the websites. You have to try to mimic a user to try to fool the website into thinking you are just a normal user. This includes using fake headers on the requests sent to the website. This also includes using a proxy and rotating through a set of proxies. Those two things are the hard part. 

Do you know if have to rotate machines/operating systems/etc? Like I know eBay can get your windows fingerprint and even if you use a different IP there is a chance they will know it is you.

Link to comment
Share on other sites

2 hours ago, president89 said:

Do you know if have to rotate machines/operating systems/etc? Like I know eBay can get your windows fingerprint and even if you use a different IP there is a chance they will know it is you.

You use proxies so that they can't identity you. You use headers so the requests look like they are coming from an actual user and not a bot. I couldn't get the proxies to work. 

Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

  • Recently Browsing   0 members

    • No registered users viewing this page.


×
×
  • Create New...