Updates March 2

PGrid

• Working on DuckDB-backed full sync between Postgres and ClickHouse. Dealing with timeouts and memory issues trying to do the rebuild step locally. Needing to implement this quickly as I’m running out of disk space on the ClickHouse server and need a fast method of recreating the database as required.
• Added x set to filters button. Helps when URLs are shared to know there are filters set.
• Wanted to update GPU description generation to use simplified functions and BAML prompts used for CPU/SSDs, but after a little testing found Gemini Flash Search Grounded hallucinated a GPU and didn’t follow full instructions in the prompt. Quite frustrating. Ended up updating my local AI browser project to be able to use my Gemini Pro queries and route BAML prompts to the Gemini web app to generate descriptions.
  • Found that not even Gemini 3 Pro running on a Pro account gave accurate information. Ended up adding a verification step that used Search Grounded Gemini Flash to check contents for errors and raise an exception if it found one. This happened multiple times.
• Made filter sliders look cooler with a glow effect and the ticks changing colour when active.
• Setting up DuckDB-backed export of snapshots from Postgres into Parquet files to reduce snapshot table size and have backups of snapshots across cloud storage and locally. This is the other part required to get DuckDB-backed full sync working with ClickHouse. Also some random bugs here that needed fixing and good tests.
• Added TUI to run specs search manually while having configurable params.
• Email alerts were sent out for a GPU; found a bug where coupon codes were incorrectly added from a different listing. Fixed bug.
• Created and posted a new video on TikTok that does the line graph video animation of a price history chart. This one being DDR5 32GB and 64GB.
• Investigated the ClickHouse disk issue; it was due to excessive logs. Updated config to log less and ran commands to delete logs.
• Started working on BHPhoto crawler.
• Enabled auto USA region redirect based on user IP on root pgrid.app navigation.
• Got Kimi agent to audit the live site; fixed some SEO issues.
• Ran specs search on over 100 SSD variants to add new SSD variants to the database. Found some normalisation/data issues on some new SSD variants created. Fixed SSD specs search to use shared normalisation function and prompt.
• Fixed issue where number of pages for frontend build was exceeding 20k limit for Cloudflare Workers upload, causing frontend build failures. Reduced pages created for RAM/SSD to only variants that had listings available for the region.
• Spent a bunch of time trying to get AI to create an Astro.js version of the frontend to migrate away from Next.js. The reasoning behind this being that Astro appears to allow rebuilding specific pages instead of rebuilding the entire site. this could be useful in the future to only update pages where information has. Astro.js builds also appear to be much faster than Next.js. Cursor browser feature has been useful to get AI to do most work autonomously but it still makes bad decisions sometimes.