Updates Dec 23

PGrid

• Migrated all references in code from just ‘amazon’ and ‘ebay’ to country specific sites to clean up the code and support international sites
• Added shipping costs column to db for price data. Updated ebay scraper to save shipping costs
• Refactoring codebase in the generic item search parts to support searching international ebay/amazon sites.
• Started working on a more generic ai classification pipeline derived from the cpu classification pipeline
• Had a partial hard to debug failure of workers. Initially caused by one of the servers going down causing otel collector and influxdb to go down. Once it was back up workers resumed fine but kept going down after a while. Was due to two simultaneous reasons: I added otel traces and there seemed to be some failure sending traces after a while causing the whole worker to go down. Was unable to debug where this was being caused and removed traces for now. The other failure was that for 2 of the servers, docker kept trying and failing to bind to ipv6 ip addresses even after putting settings in to not advertise on ipv6. Was hard to debug and the next day it just went away by itself. Still a potential issue.
• Rest of the week was doing more easy and fun stuff using ai. Updated the internal app to have an index page
• Added a prometheus style metrics server that runs on each worker and currently only shares current job and number of seconds on that job. Updated internal app to scrape each worker and display stats.