Gigatoken can tokenize the entire internet in <7 hours on one machine.
Specifically, this refers to tokenizing Common Crawl on a dual socket AMD Epyc server, reaching 24GB/s of throughput, as seen here: https://t.co/jroCWjxKxi — Marcel Rød
https://x.com/marcelroed/status/2079642156374093876?s=12