Updates July 7

PGrid

• Working on a combination materialized view + aggregating merge tree table to keep track of daily cheapest item for each category + item variant combination. Required a bit of iterating to get the schema definition working.
• Fixed incorrect regions for ebay/amazon intl sources in postgres db (amazon eu and us were set as AU region). Claude found this while investigating another issue
• Added a backfill script for the daily price history table which will allow dealing with deleted/edit price history for specific dates. Got claude to generate tests for this script.
• Generated a test suite for both the backfill script as well as the daily price history table and materialized view. This resulted in a day of fixing tests, the schema, and query functions with little bugs to get both the daily price history functions to return the exact same data
• Started replacing price history graph queries with the newly created materialized view queries starting with cpus. Noticed there was a small difference between prices on the graph between the old clickhouse query and the new one. This resulted in a large amount of debugging across both cpus and gpus and across both clickhouse queries and the original postgres daily price history query. Fixed a few things from this investigation:
  • Updated the materialized view schema to be more specific with its if conditions regarding out of stock
  • Fixed an incorrect conversion that was happening when postgres snapshots were being synced to clickhouse
  • The main issue: A clickhouse toDate type missing in the materialized view schema causing rare shifting of prices based on timezone conversions
• After finding price history issue, moved cpu details page to use the new materialized daily price history. This both sped up the nextjs builds as well as reducing load on clickhouse server.
• Fixed minor bug with internal dashboard clickhouse database reset command. Can now reset database even if it doesn’t exist in clickhouse.
• Shipped ssd table + ssd details pages. Initial version is basic but happy with new table format. Performance of builds didn’t change thanks to materialized view queries. Had to fix timeout error with nextjs builds doing too many pages in parallel.
• Cleaned up some unfinished code related to camera version of pgrid.
• Added ssd description generation functionality but haven’t ran to generate descriptions for all ssds yet. The prompt is much simpler to generate a very short description for each ssd.