president89 Posted May 8, 2020 Posted May 8, 2020 Is anyone using their own Python webscraping to look for online deals? I was thinking of paying someone to build something for me. The paid versions I think are geared towards the developers making the most money possible, and not so much about you finding all the deals - plus they scrape so much so often, that they probably have to scrape less often than an individual would to keep them from getting banned. I would love to hear someone's thoughts or experiences. I'm also open to collaboration. Quote
bigboy61 Posted May 9, 2020 Posted May 9, 2020 I have some limited experience building things in Python. It's not too bad. However, not getting caught by the websites or security measures is. I tried to write an auto-buy script for Amazon. I could have gotten it working except I tripped the account protection thing and also the website flagged my script as a bot (which it was). There are ways to get around the latter (and probably the first as well). A professional should definitely be able to help with both. Otherwise, it's going to take a lot of time, research, and trial and error. I also wrote an inventory checker for Walmart which was successful. Quote
president89 Posted May 9, 2020 Author Posted May 9, 2020 I would love a better inventory checker than ********. Was that hard to figure out? I can easily explain the price scaper to a python guy. But i have no idea how ******** even searches for walmart inventory. I am aware of the websites blocking people for pinging to often. I think there are some parameters online that tells you not to ping more than x number of minutes.. etc. Quote
BrickLegacy Posted May 9, 2020 Posted May 9, 2020 All of these problems have been solved before. Quote
president89 Posted May 9, 2020 Author Posted May 9, 2020 10 minutes ago, BrickLegacy said: All of these problems have been solved before. Elaborate - Quote
Alpinemaps Posted May 9, 2020 Posted May 9, 2020 I’ve tried a few times but ugh. So frustrating for me. Walmart is usually the one I’ve been trying. But I was trying to scrape LEGO Ideas and that was giving me a pain in the ass last week too. Quote
bigboy61 Posted May 9, 2020 Posted May 9, 2020 (edited) 14 hours ago, president89 said: I would love a better inventory checker than ********. Was that hard to figure out? I can easily explain the price scaper to a python guy. But i have no idea how ******** even searches for walmart inventory. I am aware of the websites blocking people for pinging to often. I think there are some parameters online that tells you not to ping more than x number of minutes.. etc. The Walmart inventory checker is not. You just use their publicly available, mobile API and have the program parse through the response and pick out out the price. For a scaper, the quantity of pings is important. However, even with a few, you are still likely to get caught by the websites. You have to try to mimic a user to try to fool the website into thinking you are just a normal user. This includes using fake headers on the requests sent to the website. This also includes using a proxy and rotating through a set of proxies. Those two things are the hard part. Edited May 9, 2020 by bigboy61 Quote
president89 Posted May 9, 2020 Author Posted May 9, 2020 1 hour ago, bigboy61 said: The Walmart inventory checker is not. You just use their publicly available, mobile API and have the program parse through the response and pick out out the price. For a scaper, the quantity of pings is important. However, even with a few, you are still likely to get caught by the websites. You have to try to mimic a user to try to fool the website into thinking you are just a normal user. This includes using fake headers on the requests sent to the website. This also includes using a proxy and rotating through a set of proxies. Those two things are the hard part. Do you know if have to rotate machines/operating systems/etc? Like I know eBay can get your windows fingerprint and even if you use a different IP there is a chance they will know it is you. Quote
bigboy61 Posted May 9, 2020 Posted May 9, 2020 2 hours ago, president89 said: Do you know if have to rotate machines/operating systems/etc? Like I know eBay can get your windows fingerprint and even if you use a different IP there is a chance they will know it is you. You use proxies so that they can't identity you. You use headers so the requests look like they are coming from an actual user and not a bot. I couldn't get the proxies to work. Quote
BrickLegacy Posted May 9, 2020 Posted May 9, 2020 I'd hire a professional to do this if you are serious. You might want to consider a business plan first. Quote
president89 Posted May 9, 2020 Author Posted May 9, 2020 (edited) 1 hour ago, BrickLegacy said: I'd hire a professional to do this if you are serious. You might want to consider a business plan first. i have a business plan. Edited May 9, 2020 by president89 Quote
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.