CloudRift

Last month we benchmarked libvirt and QEMU configurations across six GPU systems: H200, MI350X, RTX PRO 6000 Blackwell, 4090, 4080, and 5090. We wanted to see how much performance the defaults are leaving on the table. On the H200 host, default libvirt gave us 77 GB/s of memory bandwidth. The same hardware with vCPU pinning and guest NUMA topology exposure delivered 561 GB/s, a 7.3x gain from configuration alone. MI350X and RTX PRO 6000 showed the same shape with smaller magnitudes (2.8x and 3.2x). That kind of result tempts you to enable NUMA exposure on every guest. We'd have done the same a year ago. But when we ran NCCL benchmarks on the H200 with NUMA exposed and GPUs spanning multiple nodes, cross-node collectives dropped 57% in bandwidth. The setting that helps host memory access can quietly damage multi-GPU communication when GPU-to-node placement isn't deliberate. If you want one safe starting point, use vCPU pinning with a single guest NUMA node. That captures most of the latency improvement and sidesteps the NCCL penalty. Layer in NUMA exposure only after you've verified GPU placement and confirmed the workload is actually bandwidth-bound. Separately, SMT siblings (default at the BIOS level on most systems) gained us 2-3% of total CPU throughput while halving per-thread performance. Slawek's recommendation is to skip SMT for GPU workloads unless you're optimizing for high-density MIG or vGPU consolidation. Full walkthrough on our blog: https://lnkd.in/g_kHRifw #GPUcloud #NUMA #NCCL AMD

Running Llama 3 70B in FP16 on NVIDIAs H100s is a coordination problem before it's a compute problem. 140 GB of weights, 80 GB per GPU -> so you're already in tensor parallelism, NVLink topology, sharded KV cache, and inter-GPU traffic on the critical path. Two GPUs minimum, and the failure domain grows with every one you add. The #MI350X has 288 GB of HBM3e and 8 TB/s of bandwidth on a single accelerator. The same model fits on one GPU with 148 GB left for KV cache and long context. One machine, one process, one thing that can break. We have MI350X VMs, hosted by Grafica Veneta S.p.A., available on demand at $3.65/hour, with discounts for longer commitments. Hourly billing, no minimums, from a single accelerator up to 8-GPU configurations with 2.3 TB of HBM3e. If sharding is what's slowing your inference stack down, this is the GPU to try first: https://lnkd.in/gw2hK-c9 #AMDInstinct #ROCm #LLMinference

V100 32GB VMs are now on CloudRift at $0.29 per GPU/hour, in partnership with Cato Digital.                                                                                    A few things worth knowing:              The V100 is older silicon (Volta, launched 2017) but still capable. A single 32GB V100 fits a LoRA fine-tune of Llama 3 8B, Whisper Large inference, or a batch embedding pipeline. For workloads that do not require Hopper-class hardware, it is more than enough. The same GPU on AWS or Azure runs above $3 per GPU/hour, and the 32GB variant is usually only sold in 8-GPU bundles. CloudRift offers it as a single-GPU VM, billed by the second, with no minimums.                                                                                   The capacity comes from Cato Digital. Rather than commissioning new servers, they redeploy enterprise GPU systems retired from Meta and NVIDIA fleets, which gives the hardware a longer operational life and a smaller carbon footprint than buying new.                                                                                    If you are running fine-tuning, batch inference, rendering, or scientific compute on a budget, this is roughly a tenfold reduction in compute cost for workloads where it fits.                                                                                    Thanks to the team of Cato Digital and Colin Murcray for the partnership!  → https://lnkd.in/gPChGrQM  #GPU #AI #SustainableAI #MLOps

Dmitry built a complete LLM compiler from scratch to document how a modern ML compiler stack works end to end. 5,000 lines, two weeks, no library use, just pure Python and raw CUDA. The pipeline takes a PyTorch graph through six intermediate representations: Torch IR, Tensor IR, Loop IR, Tile IR, Kernel IR, CUDA. Each lowering moves closer to the hardware: decompose Torch ops, convert to loops and fuse, schedule kernels, render CUDA. GELU at seq=32 runs 31 µs in eager PyTorch and 6 µs in our stack, a 4.87x speedup. Softmax sits at parity with eager. Matmul lands at 50% to slightly above NVIDIA cuBLAS depending on shape. Vendor kernels are still hard to beat at full prefill on the FFN-width matmuls, which is why every production stack falls back to cuBLAS, cuDNN, and CUTLASS on the heavy hitters.   https://lnkd.in/g6qbVdFv  #CUDA #MLCompilers #PyTorch 

Check out our new deep dive comparing open-source VFIO, NVIDIA AI Enterprise, and AMD GPU virtualization.

Exciting collaboration with dstack: rent and orchestrate on-demand clusters of AMD MI350X! Go Team Red!

Want to optimize your LLM model for your hardware? Check out our latest article to see how you can run your own benchmark for free in March.

AMD Instinct MI350X are available for rental! Built for the most demanding AI Workloads: - Up to 256 compute units with CDNA 4 architecture - 288 GB HBM3E memory with up to 8 TB/s bandwidth! - 128 GB/s inter-GPU connectivity via Infinity Fabric - SR-IOV virtualization support via MxGPU — enabling secure, near-native GPU sharing across multiple VMs with hardware-level isolation - Full ROCm ecosystem support for PyTorch, JAX, and HPC frameworks Whether you're running massive LLM training, multi-tenant inference, or HPC simulations, the MI350X delivers the memory capacity and bandwidth to handle models that simply don't fit elsewhere. Rent here: https://www.cloudrift.ai/

Check out our latest LLM Inference benchmark for B200 / H200 / H100 / RTX PRO 6000.

B200 vs H200 vs H100 vs RTX PRO 6000. Who is the king of cost-efficient inference?