Updates Mar 31
PGrid
- Updated the old gpu table to new generic item migration functions to handle updates to existing gpu tables
- Added scraping from jw.com.au for gpus
- Added feature flag to enable/disable crawling gap at night time in australia
- Added cron task to migrate gpu data from old tables to generic tables once a day
- Updated db related tasks to run only on the server that also hosts the db due to performance issues (i think bandwidth of lots of data during inefficient queries)
- Added feature to sync deletes across old gpu tables and new generic tables for gpus. Corresponding sync not implemented for clickhouse as full sync from scratch can be done for now. These aren’t ran on cron yet.
- Did a little plot of daily lowest price vs number of listings in stock for 5090s as requested by someone on discord.
