An enterprise AI GPU server is sized based on the model to be run, its precision, concurrent users, request length, and expected availability level. In 2026, loading a model is not enough: it must absorb its working memory, maintain an acceptable response time, and account for failures. The right choice therefore comes from a representative load test, not a simple specification sheet.
How do you size an enterprise AI GPU server?
Sizing an enterprise AI GPU server starts with six data points: the application, the model, numerical precision, input and output length, peak concurrent users, and target response time. In 2026, NVIDIA recommends this business-oriented approach rather than choosing based solely on the advertised performance of the GPUs.
A GPU, or graphics processing unit, is a chip capable of performing in parallel the many calculations required for artificial intelligence. For inference—that is, using an already-trained model—its video memory, or VRAM, stores the model weights, conversation cache, and temporary computational workspaces.
Before requesting quotes, formalize the assumptions that actually determine the required capacity:
- the exact model and the minimum acceptable quality level;
- the number of daily users and the peak number of concurrent requests;
- the usual and maximum length of submitted documents;
- the expected response length and desired display speed;
- the time to the first word, called TTFT for “time to first token”;
- load hours, availability rate, and commitment duration.
This method avoids oversizing infrastructure for a rare peak or, conversely, discovering that an assistant that runs smoothly in a demonstration slows down as soon as it is made available to employees. The choice between on-premises AI and cloud infrastructure should come after this assessment, because the location does not correct an inaccurate load estimate.
How much GPU memory does an AI model need?
A model’s minimum memory requirement approximately corresponds to the number of parameters multiplied by the number of bytes used for each parameter. In 2026, a model with 70 billion parameters requires around 140 GB in FP16 or BF16, 70 GB in INT8 or FP8, and 35 GB with 4-bit quantization, before allowing for any overhead.
Quantization reduces the precision with which the model’s numbers are stored. It decreases memory usage and can increase the number of requests processed in parallel. According to NVIDIA recommendations published in 2026, however, its effect on quality and latency must be tested on real-world tasks rather than inferred from the numerical format alone.
The less visible factor is the KV cache, temporary memory that retains the context already processed. The longer the documents and the more numerous the conversations, the larger this cache becomes. Two applications using the same model may therefore require very different configurations.
In 2026, each NVIDIA RTX PRO 6000 Blackwell Server Edition has 96 GB of GDDR7 memory and bandwidth of up to 1.6 TB/s, according to NVIDIA. A machine equipped with two cards provides 192 GB raw; eight cards total 768 GB raw. Some of this remains necessary for the inference engine, computational buffers, and safety headroom.
In the projects we undertake, we often see model size become an objective in itself. That is rarely a good compass. For a targeted task that is sensitive to response time, a smaller, low-latency model may cost less while delivering a better experience.
How many GPUs are needed for concurrent users?
The number of GPUs cannot be calculated from the model weights alone. In 2026, concurrency, context length, per-user throughput, and time to first token determine useful capacity. Total throughput generally reaches a ceiling while latency continues to increase under load.
Batching groups multiple requests so that the GPU calculates them together. This technique often improves overall throughput, but it can worsen initial wait time and the speed perceived by each user. A high tokens-per-second figure therefore does not necessarily indicate a pleasant application experience.
As a reference point, an NVIDIA reference architecture published in 2026 reports that an NVIDIA H100 NVL supported 137 concurrent users of Llama 3.1 8B in FP8, with 2,977 output tokens per second and a TTFT of approximately 987 ms. The test used exactly 2,000 input tokens and 200 output tokens: this result cannot be generalized without reproducing those conditions.
A serious load test gradually varies concurrency using representative requests. It records output tokens per second, tokens per second per user, requests per second, TTFT, and the time between tokens. Honestly, buying before this test is like budgeting for a conference room without knowing the number of participants.
How do the Lenovo AI Express Small, Medium, and Large configurations compare?
Since September 2026, Lenovo AI Express has offered three prevalidated levels for enterprise inference: Small, Medium, and Large. Lenovo associates them respectively with models ranging from 7 to 70 billion, from 70 to 400 billion, and up to one trillion parameters. These ranges remain starting points, not capacity guarantees.
| Level | Server and GPU | Targeted models | Indicative Lenovo workload | Announced shipping time |
|---|---|---|---|---|
| Small | ThinkSystem SR650a V4, 2 × NVIDIA RTX PRO 6000 | 7B to 70B | Dozens of users, 30 output tokens/s or more | Starting at 15 business days |
| Medium | ThinkSystem SR675 V3, 8 × NVIDIA RTX PRO 6000 | 70B to 400B | Hundreds of users, 30 output tokens/s or more | Starting at 20 business days |
| Large | ThinkSystem SR680a V4 with NVIDIA HGX B300 | Up to 1 trillion parameters | Thousands of users | Starting at 25 business days |
In 2026, Lenovo specifies that actual capacity varies according to the model, workload, software, and SLA, meaning the contractual service-level commitment. The figures come from its internal sizing tool. No reliable public purchase price or comparable independent measurement of the three configurations was available at the time of the September 30, 2026 announcement.
The Lenovo shipping times of 15, 20, and 25 business days announced in 2026 concern the shipment of eligible configurations. They do not guarantee an operational platform by that date. Electrical installation, networking, security, inference engine, monitoring, and recovery testing extend the actual timeline.
Power quickly becomes a project in its own right. According to NVIDIA’s 2026 guide, an eight-GPU DGX B300 reference system contains 2.3 TB of GPU memory and consumes up to approximately 14.5 to 15 kW. Available power, cooling, and hosting must be checked; data center selection criteria go far beyond the floor space occupied.
Should you buy, rent GPUs, or use an AI API?
Buying is primarily suitable for a stable, sensitive, or long-term workload; GPU rental meets variable needs and experimentation requirements; an API mainly bills for the tokens processed and reduces technical operations. In 2026, the relevant comparison is based on the cost per useful token at the required service level, not solely on the hourly rate.
A purchased server combines hardware, financing, electricity, cooling, networking, licenses, support, and operating time costs. Unused capacity must also be taken into account. A server whose GPUs are all needed to load the model cannot absorb the loss of that machine without interruption; resilient production requires replicas or standby capacity.
Rental transforms the initial investment into variable expenditure, but unused hours are still billed when the instance remains reserved. In 2026, AWS Capacity Blocks listed, for example, $41.528 per hour before taxes and any applicable fees for a p5.48xlarge instance with eight NVIDIA H100s in the US East Atlanta local zone, compared with $98.84 per hour before taxes and any applicable fees for a p6-b200.48xlarge with eight NVIDIA B200s in several regions. Availability, regions, and commercial terms vary.
An API is often the rational choice for validating a use case, especially when demand is low or unpredictable. However, it creates dependence on the provider and requires examining data location, retention periods, and contractual commitments. For a French SME, these issues relate to the sovereignty and GDPR considerations presented in the guide dedicated to Mistral and enterprise data.
NVIDIA’s recommended compromise in 2026 is to reserve baseline capacity for predictable demand, then use the cloud for peaks. From an agency perspective, the instinct should be to model several workload scenarios rather than rely on a misleading monthly average. With a constrained budget, a modest baseline combined with cloud overflow may sometimes be preferable to an oversized, underused server.
What risks should be anticipated before going into production?
An enterprise AI GPU server should be designed around service continuity, data security, operations, and model evolution. In 2026, successfully loading a model does not prove resilience to spikes or recovery after a failure. Testing must include saturation, the loss of a node, and updates.
Plan for memory and compute headroom, then document its behavior when that headroom disappears. Maintenance matters as much as the hardware: NVIDIA drivers, inference engines, containers, and operating systems evolve at different rates. A policy for updating without interrupting production reduces the risk that a fix will cause unexpected downtime.
Security also covers requests, documents injected into the model, logs, and generated responses. Encryption, access rights, retention periods, and traceability must be defined before opening the system to users. For personal data, GDPR remains applicable even when processing takes place on an internal server.
Framing this type of project upstream avoids most unpleasant surprises. An outside perspective is especially helpful for connecting business assumptions, performance measurements, hosting architecture, and budget before making any hardware commitment.
FAQ about GPU servers for artificial intelligence
Can a 70-billion-parameter model fit on two 96 GB GPUs?
A 70-billion-parameter model takes up around 140 GB in FP16 or BF16 in 2026, before accounting for the KV cache and buffers. Two 96 GB GPUs provide 192 GB of raw capacity, but only a test with the intended context and concurrency can confirm usable capacity.
Does 4-bit quantization always degrade quality?
4-bit quantization significantly reduces weight memory, but its impact depends on the model and the task. In 2026, response quality and latency should be evaluated using a representative set of business use cases.
How long does it take to put an AI server into production?
The timeline is not limited to shipping the equipment. In September 2026, Lenovo announced a lead time starting at 15 business days for certain Small configurations, but installation, security, software, and load testing come afterward.
Which metric should you track to compare two GPU servers?
The cost per useful token at the expected service level is the most meaningful economic metric in 2026. The comparison should also retain TTFT, tokens per second per user, total throughput, and the error rate under load.