Updates Dec 9
PGrid
- My entire nomad admin ui was visible to the internet which includes job configs containing secrets and ability to exec into containers. In my nomad config for all clients I had
network_interface = "tailscale0" and then I checked one or two servers manually and the admin ui wasn’t available. So I assumed nomad web ui was running on the tailscale ip for each of the servers. This was incorrect and the servers i manually checked had default iptables blocking that ports. But given I have nomad now on 9 servers, some of them didn’t have any default blocking and port 4646 was available for the web to access. This has been the case for at least 1 month since I have migrated all servers to use nomad.
- Setup systemd based nomad client service on each server that specifically binds to the tailscale ip.
- While I didn’t notice any suspicious usage from any api keys, rotated keys for:
- AWS (SES)
- Github PAT (triggering github actions)
- Self hosted docker registry password
- Cloudflare keys (deploying to pages)
- Updated private docker registry to save contents onto disk. before if it restarted I lost all images heh
- Added a http server on top of the http proxy that uses nordvpn. This allows control to set the country using a http request and view current status of the vpn.
- Using proxy can connect to amazon.com using a US connection, issues connecting to amazon.de using a german connection.
- Updated amazon scraper to handle:
- different sites: amazon.com, amazon.com.au, amazon.de
- handle variants from single page
- Worked on performance of PGrid with 2m rows in gpu price snapshots table.
- added a duplicate cleanup function for gpu price snapshots similar to generic item price snapshots
- explored using clickhouse/duckdb but both not in use yet