EVGA

Hot!LLM Inference Rig

Author
Brad_Hawthorne
Insert Custom Title Here
  • Total Posts : 18001
  • Reward points : 0
  • Joined: 6/6/2004
  • Location: Dazed & Confused
  • Status: offline
  • Ribbons : 39
Saturday, January 11, 2025 9:00 PM (permalink)
I’m currently building up a compute rig for LLM experiments using a Supermicro X9DRG-QF, two E5-2697v2, 1TB of memory and a rack of old Tesla M40 24GB cards. Two EVGA 1600 P2 for power and two EK AIO 280 for CPU cooling. A decade old example of parts machine CUDA compute power at work. Got some serious deals on the hardware, so it’s been a cheap build so far.
#1

5 Replies Related Threads

    Brad_Hawthorne
    Insert Custom Title Here
    • Total Posts : 18001
    • Reward points : 0
    • Joined: 6/6/2004
    • Location: Dazed & Confused
    • Status: offline
    • Ribbons : 39
    Re: LLM Inference Rig Saturday, January 11, 2025 9:05 PM (permalink)

    #2
    Brad_Hawthorne
    Insert Custom Title Here
    • Total Posts : 18001
    • Reward points : 0
    • Joined: 6/6/2004
    • Location: Dazed & Confused
    • Status: offline
    • Ribbons : 39
    Re: LLM Inference Rig Saturday, January 11, 2025 9:07 PM (permalink)
    Currently set up for Ollama and Open-WebUI use. Running Llama 3.3 70B FP16, QwQ 32B FP16 and Llama Vision 3.2 90B Q8 models.
    post edited by Brad_Hawthorne - Saturday, January 11, 2025 9:11 PM
    #3
    Brad_Hawthorne
    Insert Custom Title Here
    • Total Posts : 18001
    • Reward points : 0
    • Joined: 6/6/2004
    • Location: Dazed & Confused
    • Status: offline
    • Ribbons : 39
    Re: LLM Inference Rig Saturday, January 11, 2025 9:29 PM (permalink)
    If anyone has a SR-X sitting around I'd be interested in swapping one in for the mobo.
    #4
    cotter32
    iCX Member
    • Total Posts : 275
    • Reward points : 0
    • Joined: 7/19/2018
    • Status: offline
    • Ribbons : 7
    Re: LLM Inference Rig Tuesday, January 21, 2025 1:54 AM (permalink)
    I like it .  I don't pretend to understand it, but I like it .
    Is it a Linux OS ?
     
    Too The Moon !
     
    #5
    Brad_Hawthorne
    Insert Custom Title Here
    • Total Posts : 18001
    • Reward points : 0
    • Joined: 6/6/2004
    • Location: Dazed & Confused
    • Status: offline
    • Ribbons : 39
    Re: LLM Inference Rig Thursday, January 23, 2025 9:34 AM (permalink)
    cotter32
    I like it .  I don't pretend to understand it, but I like it .
    Is it a Linux OS ?

    It's running Ubuntu 24.04 LTS at the moment.
    #6
    Jump to: