A foundation model is a machine learning or deep learning model that is trained on broad data such that it can be applied across a wide range of use cases.[1] Foundation models have transformed artificial intelligence (AI), powering prominent generative AI applications like ChatGPT.[1] The Stanford Institute for Human-Centered Artificial Intelligence's (HAI) Center for Research on Foundation Models (CRFM) created and popularized the term.[2]
Foundation models are general-purpose technologies that can support a diverse range of use cases. Building foundation models is often highly resource-intensive, with the most expensive models costing hundreds of millions of dollars to pay for the underlying data and compute required.[3] In contrast, adapting an existing foundation model for a specific use case or using it directly is much less expensive.
Early examples of foundation models are language models (LMs) like Google's BERT[4] and OpenAI's "GPT-n" series. Beyond text, foundation models have been developed across a range of modalities—including DALL-E and Flamingo[5] for images, MusicGen[6] for music, and RT-2[7] for robotic control. Foundation models constitute a broad shift in AI development: foundation models are being built for astronomy,[8] radiology,[9] genomics,[10] music,[11] coding,[12] times-series forecasting,[13] and mathematics.[14]
^ abCompetition and Markets Authority (2023). AI Foundation Models: Initial Report. Available at: https://assets.publishing.service.gov.uk/media/65081d3aa41cc300145612c0/Full_report_.pdf
^"Introducing the Center for Research on Foundation Models (CRFM)". Stanford HAI. 18 August 2021. Retrieved 11 June 2022.
^Nestor Maslej, Loredana Fattorini, Erik Brynjolfsson, John Etchemendy, Katrina Ligett, Terah Lyons, James Manyika, Helen Ngo, Juan Carlos Niebles, Vanessa Parli, Yoav Shoham, Russell Wald, Jack Clark, and Raymond Perrault, “The AI Index 2023 Annual Report,” AI Index Steering Committee, Institute for Human-Centered AI, Stanford University, Stanford, CA, April 2023.
^Rogers, Anna; Kovaleva, Olga; Rumshisky, Anna (2020). "A Primer in BERTology: What we know about how BERT works". arXiv:2002.12327 [cs.CL].
^Tackling multiple tasks with a single visual language model, 28 April 2022, retrieved 13 June 2022
^Copet, Jade; Kreuk, Felix; Gat, Itai; Remez, Tal; Kant, David; Synnaeve, Gabriel; Adi, Yossi; Défossez, Alexandre (7 November 2023). "Simple and Controllable Music Generation". arXiv:2306.05284 [cs.SD].
^"Speaking robot: Our new AI model translates vision and language into robotic actions". Google. 28 July 2023. Retrieved 11 December 2023.
^Nguyen, Tuan Dung; Ting, Yuan-Sen; Ciucă, Ioana; O'Neill, Charlie; Sun, Ze-Chang; Jabłońska, Maja; Kruk, Sandor; Perkowski, Ernest; Miller, Jack (12 September 2023). "AstroLLaMA: Towards Specialized Foundation Models in Astronomy". arXiv:2309.06126 [astro-ph.IM].
^Engineering, Spotify (13 October 2023). "LLark: A Multimodal Foundation Model for Music". Spotify Research. Retrieved 11 December 2023.
^Li, Raymond; Allal, Loubna Ben; Zi, Yangtian; Muennighoff, Niklas; Kocetkov, Denis; Mou, Chenghao; Marone, Marc; Akiki, Christopher; Li, Jia (9 May 2023). "StarCoder: may the source be with you!". arXiv:2305.06161 [cs.CL].
^Se, Ksenia; Spektor, Ian (5 April 2024). "Revolutionizing Time Series Forecasting: Interview with TimeGPT's creators". Turing Post. Retrieved 11 April 2024.
^Azerbayev, Zhangir; Schoelkopf, Hailey; Paster, Keiran; Santos, Marco Dos; McAleer, Stephen; Jiang, Albert Q.; Deng, Jia; Biderman, Stella; Welleck, Sean (30 November 2023). "Llemma: An Open Language Model For Mathematics". arXiv:2310.10631 [cs.CL].
A foundationmodel is a machine learning or deep learning model that is trained on broad data such that it can be applied across a wide range of use cases...
The FoundationalModel of Anatomy Ontology (FMA) is a reference ontology for the domain of human anatomy. It is a symbolic representation of the canonical...
Llama-2 includes foundationalmodels and models fine-tuned for dialog, called Llama-2 Chat. In a further departure from LLaMA-1, all models are released with...
A large language model (LLM) is a computational model notable for its ability to achieve general-purpose language generation and other natural language...
IBM Granite is a series of decoder-only foundationmodels created by IBM. It was announced on September 7, 2023, and an initial paper was published 4...
released model comes in either a base foundationmodel version or an instruct-tuned variant. DRBX outperforms other prominent open-source models such as...
In 2012, the Foundation approved, finalized and adopted the thematic organization and user group recognition models. An additional model for movement...
generally, statistical models are part of the foundation of statistical inference. A statistical model is usually specified as a mathematical relationship...
eight, in response to economic conservatives complaining that the 5 foundationmodel didn't caption their notion of fairness correctly, which focused on...
contrast to DirectShow's "push" model where a pipeline component pushes data to the next component. Media Foundation allows content protection by hosting...
European Foundation. The 14 CEOs were: EFQM provides training services and award schemes via their management framework, the EFQM Model. The EFQM Model (known...
own LLMs. In March 2024, Databricks released DBRX, an open source foundationmodel. It relies on a mixture-of-experts architecture and is built on the...
a model of networking developed contemporarily to the OSI model, and was funded primarily by the U.S. Department of Defense. It was the foundation for...
model. She is the widow of American professional basketball player Kobe Bryant. With her husband, she founded the Kobe and Vanessa Bryant Foundation in...
portfolios. It is foundational to Modern portfolio theory. Markowitz made the following assumptions while developing the HM model: Risk of a portfolio...
the model of Thucydides' work The History of the Peloponnesian War, as he once acknowledged. Asimov tried to end the series with Second Foundation. However...
Bluetooth Mesh specification, the following standard models and model groups have been defined: Foundationmodels have been defined in the core specification....
organizations. The World Health Organization is modeled on the International Health Division of the foundation, which sent doctors abroad to study and treat...
ISRCs which are identifiers only. The DOI system uses the indecs Content Model for representing metadata. The DOI for a document remains fixed over the...