Google targets AI inference bottlenecks with TurboQuant
Google says its new TurboQuant method could improve how efficiently AI models run by compressing the key-value cache used in LLM inference and supporting more efficient vector search. In tests on Gemma and Mistral models, the company reported significant memory savings and faster runtime with no measurable accuracy loss, including a 6x reduction in memory…