This pretraining phase is one of the main reasons AIs are so expensive to build. The need for fast computers, with very expensive chips, to run for months in pretraining is largely responsible for the fact that more advanced LLMs cost over $100 million to train, using large amounts of energy in the process.
