Local AI Stack on RTX 3090: LM Studio, ComfyUI and Hybrid Fallback
In the modern world of artificial intelligence (AI), many users strive for maximum autonomy and security of their data. One effective solution to achieve these goals is using a local AI stack on a powerful graphics card like the RTX 3090 with 24 GB VRAM. In this article, we will explore why it’s worth keeping your GPU node at home, how to set up a local stack and what problems might arise during its use, as well as propose solutions to these problems.
Why Keep Your GPU Node at Home
Using a local GPU node has several key advantages:
- Privacy: All data is processed on your own hardware, eliminating the risk of information leaks through cloud services.
- Zero token cost: You do not pay for using API tokens since all calculations are performed locally.
- Independence from limits: You are not limited by the number of requests or data volume that can be processed, unlike cloud services.
These advantages make a local GPU node an ideal solution for users who value confidentiality and flexibility.
Stack on Stationary
To create a powerful local AI stack, we use the following components:
- Processor: Xeon LGA2011-v3
- Mainboard: X99
- RAM: 32GB quad-channel
- Graphics Card: RTX 3090 with 24 GB VRAM
On this hardware, we deploy two key components: LM Studio and ComfyUI.
LM Studio on Port 1234
LM Studio provides an OpenAI-compatible API, making it easy to integrate with various applications. We use the model mistral-small-24b-instruct-2501 for processing textual data and nomic-embed-text for text embedding.
Example LM Studio configuration:
{
"model": "mistral-small-24b-instruct-2501",
"embedding_model": "nomic-embed-text",
"port": 1234
}
This configuration file specifies that LM Studio will run on port 1234 and use the specified models for data processing.
ComfyUI on Port 8188
ComfyUI is a powerful tool for creating custom user interfaces and interacting with AI models. It runs on port 8188 and allows easy integration of various models and algorithms.
Practical Problem: Hybrid Fallback
One potential problem when using a local GPU node is the possibility of it being turned off or put into hibernation during a power outage. To avoid this issue, we propose using a hybrid LLM adapter.
Solution: health-check on 192.168.1.63:1234
The hybrid adapter first checks the availability of the local LM Studio server at 192.168.1.63:1234. If the server is unavailable, the request is redirected to a cloud service such as Gemini or DeepSeek.
Example health-check:
curl -X GET http://192.168.1.63:1234/health
If no response is received within a certain timeframe, the request is redirected to a cloud service.
Example curl/JSON Request to Local LM Studio API
To send a request to the local LM Studio API, you can use the curl command. Example request:
curl -X POST http://192.168.1.63:1234/v1/chat/completions
-H "Content-Type: application/json"
-d '{
"model": "mistral-small-24b-instruct-2501",
"messages": [{"role": "user", "content": "Hello, world!"}]
}'
This request sends a text message for processing by the mistral-small-24b-instruct-2501 model and returns the response.
How Local Model Fills Blog and Translates It into 3 Languages
The local model can be used not only for processing textual data but also for generating content. For example, this blog is filled using the local LM Studio model.
The process of content generation includes several stages:
- Data Collection: The local model analyzes existing content and generates new ideas for articles.
- Text Generation: Based on the collected data, the model creates full-fledged articles.
- Translation: The generated text is translated into three languages using the
nomic-embed-textmodel.
Thus, the local model allows not only processing data but also creating new content, making it an indispensable tool for bloggers and content creators.
Conclusion
Using a local GPU node based on RTX 3090 with 24 GB VRAM provides numerous benefits, including privacy, zero token cost, and independence from limits. The stack consisting of LM Studio and ComfyUI allows easy integration of various AI models and creation of powerful applications. Hybrid fallback ensures system reliability even in the event of temporary local server downtime.
This approach opens up new possibilities for using AI in various fields, making it an accessible and flexible tool for all users.
«Artificial intelligence is not just a technology of the future; it’s a technology of the present that is changing our world today.»
Добавить комментарий