Info
We are very proud to announce the world's first and only Nvidia GH200 Grace-Hopper Superchip and Nvidia GB200 Grace-Blackwell Superchip-powered supercomputers in quiet, handy and beautiful desktop form factors. Our benchmarks show that they are currently by far the fastest AI and also the fastest ARM desktop PCs in the world. If you are looking for a workstation for inferencing and fine-tuning of insanely huge LLMs, image and video generation, we got you covered.
Example use case 1: Inferencing Llama-3.1 405B, Mistral Large 2 123B and Reflection Llama-3.1 70BLlama-3.1 405B: https://ai.meta.com/blog/meta-llama-3-1/Mistral Large 2 123B: https://mistral.ai/news/mistral-large-2407/Reflection Llama-3.1 70B: https://huggingface.co/mattshumer/Reflection-Llama-3.1-70BLlama-3.1 405B, Mistral Large 2 123B and Reflection Llama-3.1 70B are the most powerful open-source models by far and even beat GPT-4omni and Claude 3.5 Sonnet. Llama-3.1 405B with 8-bit quantization needs at least 405GB of memory to swiftly run inference! Mistral Large 2 123B with 8-bit quantization needs at least 123GB of memory to swiftly run inference! Reflection Llama-3.1 70B with 16-bit quantization needs at least 140GB of memory to swiftly run inference! Luckily, GH200 has a minimum of 576GB, GB200(A) a minimum of 864GB (768GB). With GH200 Mistral Large 2 123B and Reflection Llama-3.1 70B can be run in VRAM only for ultra high inference speed (approx. 50 tokens/sec). With Mi300 and GB200 Blackwell this is also possible for Llama-3.1 405B. With GB200 Blackwell you can expect up to 1000 tokens/sec. If the model is bigger than VRAM you can only expect approx. 1-10 tokens/sec. 4-bit quantization seems to be the best trade-off between speed and accuracy, but is natively only supported by GB200 Blackwell.Example use case 2: Fine-tuning Llama-3.1 405B with PyTorch FSDP and Q-LoraTutorial: https://www.philschmid.de/fsdp-qlora-llama3Models need to be fine-tuned on your data to unlock the full potential of the model. But efficiently fine-tuning bigger models like Llama 3 405B remained a challenge until now. This blog post walks you through how to fine-tune Llama 3 using PyTorch FSDP and Q-Lora with the help of Hugging Face TRL, Transformers, peft & datasets. Fine-tuning big models within a reasonable time requires special and beefy hardware! Luckily, GH200 and GB200 are ideal for this task.Example use case 3: Generate videos with Open-SoraDownload: https://github.com/hpcaitech/Open-Sora/Open-Sora is democratizing efficient video production for all.Generating videos with Open-Sora requires special and beefy hardware! Luckily, GH200 and GB200 are ideal for this task.Example use case 4: Image generation with Flux.1Download: https://github.com/black-forest-labs/fluxFlux.1 is the best image generator at the moment. And it's uncensored, too. In high-speed inference, FLUX requires approximately 33GB of VRAM for maximum speed. For training the FLUX model, more than 40GB of VRAM is needed. Luckily, GH200 has a minimum of 96GB, GB200(A) a minimum of 384GB (288GB).Why should you buy your own hardware?"You'll own nothing and you'll be happy?" No!!! Never should you bow to Satan and rent stuff that you can own. In other areas, renting stuff that you can own is very uncool and uncommon. Or would you prefer to rent "your" car instead of owning it? Most people prefer to own their car, because it's much cheaper, it's an asset that has value and it makes the owner proud and happy. The same is true for compute infrastructure.Even more so, because data and compute infrastructure are of great value and importance and are preferably kept on premises, not only for privacy reasons but also to keep control and mitigate risks. If somebody else has your data and your compute infrastructure you are in big trouble.Speed, latency and ease-of-use are also much better when you have direct physical access to your stuff.With respect to AI and specifically LLMs there is another very important aspect. The first thing big tech taught their closed-source LLMs was to be "politically correct" (lie) and implement guardrails, "safety" and censorship to such an extent that the usefulness of these LLMs is severely limited. Luckily, the (open-source) tools are out there to build and tune AI that is really intelligent and really useful. But first, you need your own hardware to run it on.
What are the main benefits of GH200 Grace-Hopper and GB200 Grace-Blackwell?They have enough memory to run and tune the biggest LLMs currently available.Their performance in every regard is almost unreal (up to 8520x faster than x86).There are no alternative systems with the same amount of memory.Optimized for memory-intensive AI and HPC performance.Ideal for AI, especially inferencing and fine-tuning of LLMs.Connect display and keyboard, and you are ready to go.Ideal for HPC applications like, e.g. vector databases.You can use it as a server or a desktop/workstation. Easily customizable, upgradable and repairable.Privacy and independence from cloud providers.Cheaper and much faster than cloud providers. Reliable and energy-efficient liquid cooling.Flexibility and the possibility of offline use.Gigantic amounts of coherent memory.No special infrastructure is needed.They are very power-efficient.The lowest possible latency.They are beautiful.Easy to transport.CUDA enabled.Very quiet.Run Linux.
What is the difference to alternative systems with the same amount of memory?Compared to a 8x Nvidia H100 system, GH200 costs 5x less, consumes 10x less energy and has very roughly the same performance.Compared to a 8x Nvidia A100 system, GH200 costs 3x less, consumes 5x less energy and has at least the same performance.Compared to a 4x AMD Mi300X system, GH200 costs 3x less, consumes 4x less energy and has roughly the same performance.Compared to a 4x AMD Mi300A system (which has only 512 GB memory, more is not possible because the maximum number of scale-up infinity links is 4), GH200 costs significantly less, consumes 3x less energy and has at least the same performance.Compared to a 8x Nvidia RTX A6000 Ada system which has significantly less memory (only 384GB), GH200 costs significantly less, consumes 3x less energy and has a higher performance.Compared to a 8x AMD Radeon PRO W7900 system which has significantly less memory (only 384GB), GH200 costs the same, consumes 3x less energy and has a higher performance.
GB200 has up to 30 times the performance of GH200.
The main difference between GH200/GB200 and alternative systems is that with GH200/GB200, the GPU is connected to the CPU via a 900 GB/s NVLink vs. 128 GB/s PCIe gen5 used by traditional systems. Furthermore, multiple superchips can be connected via 900/1800 GB/s NVLink vs. orders of magnitude slower network connections used by traditional systems. Since these are the main bottlenecks, GH200/GB200's high-speed connections directly translate to much higher performance compared to traditional architectures. Also, multiple NV-linked GH200 or GB200 act as a single giant GPU (CPU-GPU superchip) with one single giant coherent memory pool.
The alternative systems mentioned above also have one thing in common: they are not available in standard desktop form factors, like our GH200 and GB200 systems are. What is the difference to 19-inch server models?Form factor: 19-inch servers have a very distinct form factor. They are of low height and are very long, e.g. 438 x 87.5 x 900mm (17.24" x 3.44" x 35.43"). This makes them rather unsuitable to place them anywhere else than in a 19-inch rack. Our GH200 and GB200 tower models have desktop form factors: 244 x 567 x 523 mm (20.6 x 9.6 x 22.3") or 255 x 565 x 530 mm (20.9 x 10 x 22.2") or 250 x 404 x 359 mm (9.8 x 15.9 x 14.1"). This makes it possible to place them almost anywhere.Noise: 19-inch servers are extremely loud. The average noise level is typically around 90 decibels, which is as loud as a subway train and exceeds the noise level that is considered safe for workers subject to long-term exposure. In contrast, our GH200 and GB200 tower models are very quiet (factory setting is 25 decibels) and they can easily be adjusted to even lower or higher noise levels because each fan can be tuned individually and manually from 0 to 100% PWM duty cycle. Efficient cooling is ensured, because our GH200 and GB200 tower models have a higher number of fans and the low-revving Noctua fans have a much bigger diameter compared to their 19-inch counterparts and move approximately the same amount or even a much higher amount of air depending on the specific configuration and PWM tuning.Transportability: 19-inch servers are not meant to be transported, consequently, they lack every feature in this regard. In addition, their form factor makes them rather unsuitable to be transported. Our GH200 and GB200 tower models, in contrast, can be transported very easily. Our metal and mini cases even feature two handles, which makes moving them around very easy.Infrastructure: 19-inch servers typically need quite some infrastructure to be able to be deployed. At the very least, a 19-inch mounting rack is definitely required. Our GH200 and GB200 models do not need any special infrastructure at all. They can be deployed quickly and easily almost everywhere.Latency: 19-inch servers are typically accessed via network. Because of this, there is always at least some latency. Our GH200 and GB200 tower models can be used as desktops/workstations. In this use case, latency is virtually non-existent. Looks: 19-inch server models are not particularly aesthetically pleasing. In contrast, our available case options are in our humble opinion by far the most beautiful there are.Technical details of our GH200 workstations (base configuration)Metal tower with two color choices: Titan grey and Champagne goldGlass tower with four color choices: white, black, green or turquoiseMini tower with two color choices: white and blackAvailable air or liquid-cooled1x/2x Nvidia GH200 Grace Hopper Superchip1x/2x 72-core Nvidia Grace CPU1x/2x Nvidia Hopper H100 Tensor Core GPU (on request)1x/2x Nvidia Hopper H200 Tensor Core GPU480GB/960GB of LPDDR5X memory with error-correction code (ECC)96GB of HBM3 (available on request) or 144GB of HBM3e memory per superchip576GB (available on request), 624GB or 1248GB of total fast-access memoryNVLink-C2C: 900 GB/s of bandwidthProgrammable from 450W to 1000W TDP (CPU + GPU + memory)2x/4x High-efficiency 2000W/2400W PSU2x/4x PCIe gen4/5 M.2 slots on board2x/4x/8x PCIe gen4/5 drive (NVMe)2x/3x FHFL PCIe Gen5 x161x/3x/4x USB 3.0 ports2x RJ45 10GbE ports1x RJ45 IPMI port1x Mini display portHalogen-free LSZH power cablesStainless steel boltsVery quiet, the factory setting is 25 decibels (fan speed and thus noise level can be individually and manually configured from 0 to 100% PWM duty cycle)3 years manufacturer's warranty244 x 567 x 523 mm (20.6 x 9.6 x 22.3") or 255 x 565 x 530 mm (20.9 x 10 x 22.2") or 250 x 404 x 359 mm (9.8 x 15.9 x 14.1")30 kg (66 lbs) or 20 kg (44 lbs)Optional componentsLiquid coolingBigger custom air-cooled heatsinkNIC Nvidia Bluefield-3 400GbNIC Nvidia ConnectX-7 200GbNIC Intel 100GbWLAN + Bluetooth cardUp to 2x 8TB M.2 SSDUp to 8x 8TB E1.S SSDUp to 10x 60TB 2.5" SSDStorage controllerRaid controllerAdditional USB portsMulti-display graphics cardSound cardMouseKeyboardConsumer or industrial fansIntrusion detectionOS preinstalledAnything possible on request
Need something different? We are happy to build custom systems to your liking.
What are the main differences between the offered GH200 models?GH200: metal or glass tower, air-cooled, with 1 of 2 M.2 and 0 of 4 E1.S hard disks, 3x USBGH200 Special Edition: metal or glass tower, air-cooled, without M.2 (0 of 2) and 2.5" (0 of 4) hard disks, 1x USB (mini USB hub included: 3x USB)GH200 Super: metal or glass tower, air-cooled, with one M.2 (1 of 2) and no E1.S (0 of 8) hard disks, 1x USB (mini USB hub included: 3x USB)GH200 Giga: metal or glass tower, air-cooled, with one M.2 (1 of 2) and no 2.5" (0 of 4) hard disks, 2x USBGH200 Liquid: metal or glass tower, liquid-cooled, comes with 1 of 2 M.2 and 0 of 8 E1.S hard disks, 4x USBGH200 Mini: mini tower, air-cooled, comes with 1 of 2 M.2 and 0 of 2 E1.S hard disks, 1x USB (mini USB hub included: 3x USB)GH200 Dual: Two NVlinked superchips, metal tower, air-cooled, comes with 1 of 2 M.2 and 0 of 8 E1.S hard disks, 2x USB
Comparison chart: GH200 comparison chart.pdf
Compute performance of one GH20067 teraFLOPS FP641 petaFLOPS TF322 petaFLOPS FP164 petaFLOPS FP8Benchmarkshttps://github.com/mag-/gpu_benchmarkWhite paper: Nvidia GH200 Grace-Hopper white paperGB200 Blackwell
The coming Nvidia GB200 Grace-Blackwell Superchip has truly amazing specs to show off. GPTshop.ai systems with Nvidia GB200 Grace-Blackwell will arrive January 2025. Be one of the first in the world to get a GB200 desktop/workstation. Order now!
Compute performance of one GB20090 teraFLOPS FP645 petaFLOPS TF3210 petaFLOPS FP1620 petaFLOPS FP840 petaFLOPS FP4White paper: Nvidia GB200 Grace-Blackwell white paperThe Grace CPU
The Nvidia Grace CPU delivers twice the performance per watt of conventional x86-64 platforms and is currently the world’s fastest ARM CPU. Grace-Grace superchip workstations are available on request.
Benchmarks
https://www.phoronix.com/review/nvidia-gh200-gptshop-benchmarkhttps://www.phoronix.com/review/nvidia-gh200-amd-threadripperhttps://www.phoronix.com/review/aarch64-64k-kernel-perfhttps://www.phoronix.com/review/nvidia-gh200-compilersWhite paper: Nvidia Grace-Grace white paper
Trademark information: Nvidia is a trademark of Nvidia corporation. ARM is a trademark of Arm Holdings plc.Download
Here you can find various downloads concerning our GH200 and GB200 systems: operating systems, firmware, drivers, software, manuals, white papers, spec sheets and so on. Everything you need to run your system and more.
White papers Nvidia GH200 Grace-Hopper white paper Nvidia GB200 Grace-Blackwell white paper
Comparison chartsComparison chart GH200 Grace-Hopper systems: GH200 comparison chart.pdf
Spec sheetsGH200 Special Edition 576GB: Spec sheet GH200 Special Edition 576GB.pdfGH200 Glass Special Edition 576GB: Spec sheet GH200 Glass Special Edition 576GB.pdfGH200 Dual 1248GB: Spec sheet GH200 Dual 1248GB.pdfGH200 624GB: Spec sheet GH200 624GB.pdfGH200 Special Edition 624GB: Spec sheet GH200 Special Edition 624GB.pdfGH200 Super 624GB: Spec sheet GH200 Super 624GB.pdfGH200 Giga 624GB: Spec sheet GH200 Giga 624GB.pdfGH200 Liquid 624GB: Spec sheet GH200 Liquid 624GB.pdfGH200 Glass 624GB: Spec sheet GH200 Glass 624GB.pdfGH200 Glass Special Edition 624GB: Spec sheet GH200 Glass Special Edition 624GB.pdfGH200 Glass Super 624GB: Spec sheet GH200 Glass Super 624GB.pdfGH200 Glass Giga 624GB: Spec sheet GH200 Glass Giga 624GB.pdfGH200 Glass Liquid 624GB: Spec sheet GH200 Glass Liquid 624GB.pdfGH200 Mini 624GB: Spec sheet GH200 Mini 624GB.pdfGB200 Blackwell: Spec sheet GB200 Blackwell.pdfGB200 Blackwell Dual: Spec sheet GB200 Blackwell Dual.pdfGB200 Blackwell Mini: Spec sheet GB200 Blackwell Mini.pdf
ManualsOfficial Nvidia GH200 Manual: https://docs.nvidia.com/grace/#grace-hopperOfficial Nvidia Grace Manual: https://docs.nvidia.com/grace/#grace-cpuOfficial Nvidia Grace getting started: https://docs.nvidia.com/grace/#getting-started-with-nvidia-graceGH200 Dual 1248GB: Manual GH200 Dual 1248GB.pdfGH200 576GB: Manual GH200 576GB.pdfGH200 624GB: Manual GH200 624GB.pdfGH200 Special Edition 576GB: Manual GH200 Special Edition 576GB.pdfGH200 Special Edition 624GB: Manual GH200 Special Edition 624GB.pdfGH200 Super 624GB: Manual GH200 Super 624GB.pdfGH200 Giga 624GB: Manual GH200 Giga 624GB.pdfGH200 Liquid 624GB: Manual GH200 Liquid 624GB.pdfGH200 Glass 624GB: Manual GH200 Glass 624GB.pdfGH200 Glass Special Edition 576GB: Manual GH200 Glass Special Edition 576GB.pdfGH200 Glass Special Edition 624GB: Manual GH200 Glass Special Edition 624GB.pdfGH200 Glass Super 624GB: Manual GH200 Glass Super 624GB.pdfGH200 Glass Giga 624GB: Manual GH200 Glass Giga 624GB.pdfGH200 Glass Liquid 624GB: Manual GH200 Glass Liquid 624GB.pdfGH200 Mini 624GB: Manual GH200 Mini 624GB.pdf
Operating systemsUbuntu Server for ARM: https://cdimage.ubuntu.com/releases/24.04/release/ubuntu-24.04.1-live-server-arm64+largemem.iso
Any other ARM linux distribution with kernel > 6.5 should work just fine. Using the newest 64k kernel is highly recommended.
DriversNvidia GH200 drivers (also work for RTX A4000 and RTX 6000 Ada): https://www.nvidia.com/Download/index.aspx?lang=en-us
Select product type "data center", product series "HGX-Series" and operating system "Linux aarch64".Aspeed drivers: https://aspeedtech.com/support_driver/Nvidia Bluefield-3 drivers: https://developer.nvidia.com/networking/doca#downloadsNvidia ConnectX-7 drivers: https://network.nvidia.com/products/ethernet-drivers/linux/mlnx_en/Intel E810-CQDA2 drivers: https://www.intel.com/content/www/us/en/download/19630/intel-network-adapter-driver-for-e810-series-devices-under-linux.html?wapkw=E810-CQDA2Graid SupremeRAID SR-1010 drivers: https://docs.graidtech.com/#linux-driver
FirmwareGH200 (Glass): Firmware GH200 (Glass).zipGH200 Special Edition (Glass): Firmware GH200 Special Edition (Glass).tarNvidia Bluefield-3 firmware: https://network.nvidia.com/support/firmware/bluefield3/Nvidia ConnectX-7 firmware: https://network.nvidia.com/support/firmware/connectx7/Intel E810-CQDA2 firmware: https://www.intel.com/content/www/us/en/search.html?ws=idsa-default#q=E810-CQDA2
Top open source LLMsLlama 3.1: https://huggingface.co/collections/meta-llama/llama-31-669fc079a0c406a149a5738fMistral Large 2 123B: https://huggingface.co/mistralai/Mistral-Large-Instruct-2407Reflection Llama-3.1 70B: https://huggingface.co/mattshumer/Reflection-Llama-3.1-70B
SoftwareNvidia Github: https://github.com/NVIDIANvidia CUDA: https://developer.nvidia.com/cuda-downloads?target_os=Linux&target_arch=arm64-sbsaNvidia Container-toolkit: https://github.com/NVIDIA/nvidia-container-toolkitNvidia Tensorflow: https://github.com/NVIDIA/tensorflowNvidia Pytorch: https://catalog.ngc.nvidia.com/orgs/nvidia/containers/pytorchNvidia NIM models: https://build.nvidia.com/explore/discoverNvidia Triton inference server: https://www.nvidia.com/de-de/ai-data-science/products/triton-inference-server/Nvidia NeMo Customizer: https://developer.nvidia.com/blog/fine-tune-and-align-llms-easily-with-nvidia-nemo-customizer/Huggingface open source LLMs: https://huggingface.co/modelsHuggingface text generation inference: https://github.com/huggingface/text-generation-inferencevLLM - inference and serving engine: https://github.com/vllm-project/vllmOllama - run LLMs locally: https://ollama.com/Open WebUI: https://github.com/open-webui/open-webui/Fine-tune Llama 3 with PyTorch FSDP and Q-Lora: https://www.philschmid.de/fsdp-qlora-llama3/Perplexica: https://github.com/ItzCrazyKns/PerplexicaMorphic: https://github.com/miurla/morphicOpen-Sora: https://github.com/hpcaitech/Open-SoraFlux.1: https://github.com/black-forest-labs/fluxStorm: https://github.com/stanford-oval/storm
BenchmarkingGPU benchmark: https://github.com/mag-/gpu_benchmarkPhoronix test suite: https://www.phoronix-test-suite.com/MLCommons: https://mlcommons.org/benchmarks/Contact
Email: x@GPTshop.ai
GPTshop.ai UG (limited)
Sachsenhof 1
96106 Ebern
Germany
CEO: Bernhard Guentner
Trade register Bamberg HRB 11548
VAT: DE367011966
Lucid: DE5350216829012
Try
Try before you buy. You can apply for remote testing of a GH200 or GB200 system. After approval, you will be given login credentials for remote access. If you want to come by and see it for yourself and run some tests, that is also possible any time.
Currently available for testing: GH200 Special Edition 576GB
Apply via email: x@GPTshop.ai