Mixture Of Experts - Is GPT-4 a mixture of experts?

A collection or ensemble of eight smaller models, GPT-4 brings together the unique strengths of its creators rather than being a single massive model. It is estimated that there are 220 billion parameters in each of these models. The 'Mixture of Experts' model paradigm, which is also called the hydra of models, is a well-established methodology that uses this strategy (see below).