map.name

Gigatoken data processing benchmarks

T · Marcelroed · 22 July 2026

Gigatoken can tokenize the entire internet in <7 hours on one machine. Specifically, this refers to tokenizing Common Crawl on a dual socket AMD Epyc server, reaching 24GB/s of throughput, as seen here: — Marcel Rød

Open on T