▸We are members from the Improbable AI group at MIT led by Professor Pulkit Agrawal.
▸Our research motto is to tackle the improbable problems in Machine Learning and Robotics. We hope to share our insights through our articles below.Improbable AI
Mixture of Experts (MoE)
MoE are rumored to be a critical components in scaling up to a trillion parameter model. By routing tokens to specialized modular functions, it enables models to have representational power of a much larger model than what is used for prediction. We will be discussing how MoE works and its recent advances in this literature...