Updates Mar 17

PGrid

• Ebay introduced some new bot checking and needed to update ebay scrapers to handle the bot check where it would check the title and wait for the page to be redirected
• Progress on clickhouse db working syncing with postgres db
  • gpu snapshot data gets copied over to generic snapshot data
  • new snapshot data gets added to clickhouse
  • some queries for generic items redone to use clickhouse
    • Had issues with updating queries getting price history. There was inconsistencies between old gpu queries, a new generic snapshot based query and then a clickhouse based query. AI both caused and solved the problem through iteration.
    • Updated the clickhouse time series schema to be able to handle multiple categories with slightly different structures and have a shared query interface for price history
    • Had issues converting getCheapestItem to use clickhouse. Showing no results when there should have been results. Then showing mostly correct values except for a few which needed further debugging.
    • Performance issues getting cheapest item for all gpu and all cpu products. Queries were taking 10 seconds or more. They were done by querying for cheapest item for each gpu or cpu in a loop. Potential fix to do loop within sql but not done yet.
• Fixed issues with duplicate snapshot deletion where it now batch loads relevant snapshot information into memory before processing then deleting in batches. Also upped the batch size for querying gpu snapshots.