Updates March 23

PGrid

• integrated Best Buy. using API instead of scraping.
• updated descriptions of more GPUs, this time using Perplexity and updating the Gemini validation prompt when finding validation issues
• updated classification across all categories to ignore failed items to save tokens
• updated ebay scraper on more issues:
  • new specific filtering for USA listings based on the search page. the search scraper will ignore data for search result listings that don’t have the text ‘located in united states’
  • Best Buy for some reason has hidden prices in the search result. this resulted in undefined prices getting saved which led to in stock no price data. updated search scraper to ignore these listings. the single page scraper still can parse the correct prices for these listings
  • the biggest issue was search results showing /itm/ links that redirected to /p/ pages. the /p/ pages in ebay are supposed to represent a single product which means different listings can appear for the same /p/ url. this broke the fundamental assumption of the app that a single url points to a single product. fixed this to follow through the /p/ urls and find the /itm/ page that the /p/ page is showing. this change got rid of a lot of issue ebay listings that caused the price history for a lot of items to look incorrect.
• work in progress on overall price history charts that cover lots of models of specific capacities in RAM table
• bhphoto enabled laptop scraping
• newegg enabled laptop scraping
• pccasegear enabled laptop scraping
• updated newegg scraper to be able to parse coupon codes. they add coupon code discounts in a ‘worded’ format like ‘twenty off’ so needed to create a parser for this
• more detailed metrics added for ebay scraping to try to debug why coupon codes aren’t always getting parsed
• added cli commands to do duckdb based full sync between postgres and clickhouse. this is a much faster way to do full sync. needed it after deleting 180k snapshot rows related to ebay USA.
• updated ram specs search to make it use more shared utils that were already used across other 3 categories
• posted one slideshow about SSDs on tiktok. got ~700 views and mostly from USA. don’t think it resulted in any traffic to the app though.