Updates Nov 25

PGrid

• Got generic item searching working for amazon. Includes item indexing, and saving price snapshot for listings found from search results and then going to individual pages for previously classified items that didn’t show up in search.
• Upping how often I scrape all sites for black friday sales. Added 3 temp servers from vultr, once nomad was installed and setup (mostly using ansible), scaling workers on nomad was super easy through a command which was cool.
  • Turns out those vultr servers have very bad reputation IP addresses. I forwarded my local traffic and lots of websites blocked access including getting cloudflare WAF for mwave. The vultr servers ended up only doing ebay scraping which was still useful as ~700 pages are scraped every hour.
• Added ebay scraper for generic items. Both searching and scraping individual pages. This didn’t fit with how the amazon search worked so basically had to have a different way of doing ebay and amazon annoyingly.
• Updated ebay scraper to handle pages with multiple selections for products
• Added a price snapshot deduplication function since prices are scraped on amazon by searching for items, the same item can show up in different searches and result in duplicate snapshots.
• Added method to fail and unfail generic items so they aren’t repeatedly individually searched if the listing is taken down.
• Total revenue from Novemeber: $414 ebay + $116 amazon = $530. From about 5k visitors, 11k visits, 32k page views. Mostly due to black friday deals so will be difficult replicating revenue without new products or more traffic for next month.