Roy 10,832 / 0 Report Post Posted February 11, 2022 Description My first project using Python's Scrapy framework. I'm using this project personally for a couple friends of mine and I. Basically, it scrapes a products listing page from BestBuy that lists RTX 3080 TIs. It scans each product and if the c-button-disable class doesn't exist within each entry (indicating it is not sold out and available), it will email a list of users from the settings.py file. It keeps each ID tracked in SQLite to make sure users don't get emailed more than once. Settings Settings are configured in the src/bestbuy_parser/bestbuy_parser/settings.py file. The following are defaults. # General Scrapy settings. BOT_NAME = 'bestbuy_parser' SPIDER_MODULES = ['bestbuy_parser.spiders'] NEWSPIDER_MODULE = 'bestbuy_parser.spiders' TELNETCONSOLE_ENABLED = False LOG_LEVEL = 'ERROR' # The User Agent used to crawl. USER_AGENT = 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36' # Obey robots.txt rules ROBOTSTXT_OBEY = True # Best Buy Parser-specific settings. # The email's subject to send. MAIL_SUBJECT = "RTX 3080 TI In Stock On Best Buy!" # Where the email is coming from. MAIN_FROM = "[email protected]" # The email body. MAIL_BODY = '<html><body><ul><li><a href="https://www.bestbuy.com{link}">{name}</a></li><li>{price}</li></ul></body></html>' # Recipients to send to. MAIL_TO = [ '[email protected]' ] # If any items exceed this price and are labeled as available, users will not be notified on this product. MAX_PRICE = 1500.00 # How often to scan in seconds. SCAN_TIME = 5.0 # Whether to print to `stdout` when a product isn't sold out/valid and a user is being emailed. PRINT_WHEN_NOT_SOLDOUT = True GitHub Repository For More Details Share this post Link to post Share on other sites More sharing options...
Salad 1,132 / 18,942 Report Post Posted February 11, 2022 · Hidden Hidden make this for @Ben so he can get his fuckin PS5 lmao Director of Divisions and Physion Unturned Division Leader xQc on GFL Surf 7/16/22 Share this post Link to post
Roy 10,832 / 0 Report Post Posted February 12, 2022 5 hours ago, Salad said: make this for @Ben so he can get his fuckin PS5 lmao I may make something more generic so it would support other websites. Also: The project above definitely works and I had the opportunity to buy the RTX 3080 TI at just $1119 today! Share this post Link to post Share on other sites More sharing options...
DrakoHD 222 / 4,291 Report Post Posted February 15, 2022 · Hidden Hidden I also want this lol I want a ps5 too Thanks to @Auralanity for the amazing signature Share this post Link to post
Ben 2,378 / 31,861 Report Post Posted February 15, 2022 · Hidden Hidden On 2/11/2022 at 7:56 PM, Roy said: I may make something more generic so it would support other websites. Also: The project above definitely works and I had the opportunity to buy the RTX 3080 TI at just $1119 today! Please do this so I can finally get PS5 - I've been up super early lately in hope for a restock lol Share this post Link to post
