Business Wire

Red Hat Launches the llm-d Community, Powering Distributed Gen AI Inference at Scale

Forged in collaboration with founding contributors CoreWeave, Google Cloud, IBM Research and NVIDIA and joined by industry leaders AMD, Cisco, Hugging Face, Intel, Lambda and Mistral AI and university ...
NextBigFuture

Defeating Nondeterminism in LLM Inference by Thinking Machines

A research article by Horace He and the Thinking Machines Lab (X-OpenAI CTO Mira Murati founded) addresses a long-standing issue in large language models (LLMs). Even with greedy decoding bu setting ...
Forbes

Cerebras Now The Fastest LLM Inference Processor; Its Not Even Close

The company tackled inferencing the Llama-3.1 405B foundation model and just crushed it. And for the crowds at SC24 this week in Atlanta, the company also announced it is 700 times faster than ...
Semiconductor Engineering

Four Architectural Opportunities for LLM Inference Hardware (Google)

“Large Language Model (LLM) inference is hard. The autoregressive Decode phase of the underlying Transformer model makes LLM inference fundamentally different from training. Exacerbated by recent AI ...
Forbes

Nvidia Dynamo — Next-Gen AI Inference Server For Enterprises

At the GTC 2025 conference, Nvidia introduced Dynamo, a new open-source AI inference server designed to serve the latest generation of large AI models at scale. Dynamo is the successor to Nvidia’s ...
Semiconductor Engineering

The On-Device LLM Revolution

Users running a quantized 7B model on a laptop expect 40+ tokens per second. A 30B MoE model on a high-end mobile device ...
InfoQ

GPULlama3.java Brings GPU-Accelerated LLM Inference to Pure Java

A monthly overview of things you need to know as an architect or aspiring architect. Unlock the full InfoQ experience by logging in! Stay updated with your favorite authors and topics, engage with ...
SiliconANGLE

Startup FuriosaAI debuts RNGD chip for LLM and multimodal AI inference

FuriosaAI Inc., a semiconductor startup that’s laser-focused on artificial intelligence, has unveiled a new accelerator chip it says is geared for large language models and multimodal AI. Its new chip ...
InfoQ

Cactus v1: Cross-Platform LLM Inference on Mobile with Zero Latency and Full Privacy

A monthly overview of things you need to know as an architect or aspiring architect. Unlock the full InfoQ experience by logging in! Stay updated with your favorite authors and topics, engage with ...
XDA Developers on MSN

I served a 200 billion parameter LLM from a Lenovo workstation the size of a Mac Mini

This mini PC is small and ridiculously powerful.
Business Wire

Enfabrica Unveils Industry’s First Ethernet-Based AI Memory Fabric System for Efficient Superscaling of LLM Inference

MOUNTAIN VIEW, Calif.--(BUSINESS WIRE)--Enfabrica Corporation, an industry leader in high-performance networking silicon for artificial intelligence (AI) and accelerated computing, today announced the ...