Definition
An Inference Server is a computing server configured to run trained artificial intelligence or machine learning models and generate predictions, classifications, recommendations, or other outputs from new input data. It typically uses GPUs, CPUs, or other AI accelerators to execute model inference efficiently.
Inference servers are optimized for serving AI models in production or other real-time and batch inference environments. They commonly include model-serving software, compute accelerators, high-speed networking, memory, and storage required to load and execute AI models.
Unlike a Training Server, which is primarily used to train or fine-tune models, an inference server focuses on executing trained models and serving their outputs to applications or users. Multiple inference servers can also be deployed together as an inference cluster to provide greater capacity, scalability, and availability.
Related Terms
- AI Inference — The process of using a trained AI model to generate an output from new input data.
- Training Server — A server optimized for training or fine-tuning AI models.
- Inference Cluster — Multiple interconnected inference servers working together to serve AI workloads at scale.
- AI Cluster — A broader cluster architecture that can support training, inference, or other AI workloads.
- GPU Server — A server equipped with one or more GPUs for accelerated computing, including AI inference.
Related ATTOM Solutions
High-performance inference servers can create significant requirements for power density, thermal management, networking, and rack infrastructure, particularly when deployed at scale. ATTOM provides modular and prefabricated data center solutions designed to support high-density AI computing environments.
