Cerebras Systems in Sunnyvale, California, a pioneer of complex computing systems with artificial intelligence, presented its AI supercomputer Andromeda with 13.5 million cores at the SC22 (International Conference for High Performance Computing) in Dallas, Texas, in November. The special feature is the setup as a cluster of 16 of Cerebras' CS-2 systems based on Cerebras MemoryX and SwarmX technologies. Andromeda delivers more than 1 exaflops in the AI area and 120 petaflops for extremely high computing power (dense computing) with half precision (16 bit).
Andromeda's most important application to date is as an AI supercomputer for near-perfect linear scaling of large GTP-class language models (GPT-3, GPT-J and GPT-NeoX). It uses simple data paralellisms. Powered by 18,176 AMD EPYC processors, Andromeda contains more cores than 1,953 Nvidia A100 GPUs (Graphic Processor Unit) and 1.6 times the world's largest supercomputer Frontier with 8.7 million cores.
Linear scaling means that the training time is reduced in almost perfect proportion with each additional CS-2 wafer. This applies to the processing of large language models with 25 billion parameters and very long sequences. This cannot be achieved with traditional GPUs, such as Polaris systems with 2,000 Nvidia A100 GPUs, due to their limited memory bandwidth.
Andromeda is already being used in academic research, for example at the American Argonne National Laboratory and at Cambridge University, with free access via notebook for students. It is also used in commercial environments such as AstraZeneca and GlaxoSmithKline. Andromeda is currently installed in the form of 16 racks at Colovore, a large data center in Santa Clara, California. The machine can be used simultaneously, either as a supercomputer cluster with its 16 CS-2 systems for one user, or as 16 individual CS-2 systems for 16 different jobs in any practicable combination.
Cerebras, founded in 2015, has distinguished itself with its wafer-scale processor systems (Wafer-Scale Engine) for solving AI problems. In 2019, WSC-1 with an edge length of 8.5 inches (216 mm) was introduced. 2021 saw the launch of the CS-2 system based on WSE-2 with 850,000 cores. Each WSE-2 wafer has three physical layers for arithmetic with 850,000 cores and 3.4 million FPUs (floating point units), memory for 40 GB onboard SRAM and communication. The Leibniz Supercomputing Center (LRZ) in Garching near Munich is the first European research institution to install a supercomputer with CS-2.
