Can a Mini PC Run a 70B Model? Strix Halo and Unified Memory Explained
A 70B model needs about 40GB of memory, more than any consumer graphics card has. AMD's Strix Halo mini PCs solve that with up to 128GB of unified memory. Here is what they run, how fast, and why a big MoE model...