How to run LLama 405B model on a desktop PC

https://huggingface.co/bartowski/Hermes-3-Llama-3.1-405B-GGUF

Download LLama 405B model from Huggingface.co to C:\AIModels

Open Huggingface page
Figure 1 - Open Huggingface.co page

Open LLama 405B model page
Figure 2 - Open LLama 405B model page

Download model files
Figure 3 - Download model files

Paste model files to AIModels folder
Figure 4 - Paste model files to C:\AIModels folder

Create Ozeki AI server model and Chatbot

PC system specification
Figure 5 - PC system specification

Multi GPU system
Figure 6 - Multi GPU system

Open Ozeki 10
Figure 7 - Open Ozeki 10

Open AI studio
Figure 8 - Open AI studio

Create new GGUF model
Figure 9 - Create new GGUF model

Select model file
Figure 10 - Select model file

Open model details
Figure 11 - Open model details

Set GPU layer options
Figure 12 - Set GPU layer options

Create new AI chatbot
Figure 13 - Create new AI chatbot

Select model
Figure 14 - Select model

Enable chatbot
Figure 15 - Enable chatbot

Conversation with the model
Figure 16 - Conversation with the model

Conversation Log
Figure 17 - Conversation Log

More information