v0.12.3
/ November 10, 2023; 6 months ago (2023-11-10)
Repository
github.com/microsoft/DeepSpeed
Written in
Python, CUDA, C++
Type
Software library
License
Apache License 2.0
Website
deepspeed.ai
DeepSpeed is an open source deep learning optimization library for PyTorch.[1] The library is designed to reduce computing power and memory use and to train large distributed models with better parallelism on existing computer hardware.[2][3] DeepSpeed is optimized for low latency, high throughput training. It includes the Zero Redundancy Optimizer (ZeRO) for training models with 1 trillion or more parameters.[4] Features include mixed precision training, single-GPU, multi-GPU, and multi-node training as well as custom model parallelism. The DeepSpeed source code is licensed under MIT License and available on GitHub.[5]
The team claimed to achieve up to a 6.2x throughput improvement, 2.8x faster convergence, and 4.6x less communication.[6]
^"Microsoft Updates Windows, Azure Tools with an Eye on The Future". PCMag UK. May 22, 2020.
^Yegulalp, Serdar (February 10, 2020). "Microsoft speeds up PyTorch with DeepSpeed". InfoWorld.
^"Microsoft unveils "fifth most powerful" supercomputer in the world". Neowin. 18 June 2023.
^"Microsoft trains world's largest Transformer language model". February 10, 2020.
^"microsoft/DeepSpeed". July 10, 2020 – via GitHub.
^"DeepSpeed: Accelerating large-scale model inference and training via system optimizations and compression". Microsoft Research. 2021-05-24. Retrieved 2021-06-19.
DeepSpeed is an open source deep learning optimization library for PyTorch. The library is designed to reduce computing power and memory use and to train...
and open-source software portal Comparison of deep learning software Differentiable programming DeepSpeed Chintala, Soumith (1 September 2016). "PyTorch...
Deep Zoom is a technology developed by Microsoft for efficiently transmitting and viewing images. It allows users to pan around and zoom in on a large...
machine learning algorithms include the following: Caffe Deeplearning4j DeepSpeed ELKI Google JAX Infer.NET Keras Kubeflow LightGBM Mahout Mallet Microsoft...
Deep Purple in Rock is the fourth studio album by English rock band Deep Purple, released on 5 June 1970. It was the first studio album recorded by the...
The speed of light in vacuum, commonly denoted c, is a universal physical constant that is exactly equal to 299,792,458 metres per second (approximately...
moment, which improves load times and responsiveness. Despite the increased speed in Deepfish, it's quite bandwidth heavy and can render pages slowly if it...
Special electronic circuits called deep learning processors were designed to speed up deep learning algorithms. Deep learning processors include neural...
The speed of sound is the distance travelled per unit of time by a sound wave as it propagates through an elastic medium. At 20 °C (68 °F), the speed of...
natural language processing (specifically machine reading comprehension), deep learning and reinforcement learning. Gray Systems Lab, in Madison, Wisconsin...
converted into a number of Deep Zoom Image (DZI) format files. These can be combined to make up a Deep Zoom Collection (DZC). These Deep Zoom Images create a...
staples "Speed King", "Into The Fire" and "Child in Time". The non-album single "Black Night", released around the same time, finally put Deep Purple into...
certain Deep Purple songs such as "Speed King", "Fireball" and "Highway Star". The latter was called "early speed metal" by Robb Reiner of speed metal band...
Deep Jandu is a Canadian record producer, rapper, and singer associated with Punjabi music. He is founder of record label Royal Music Gang, along with...
August 2014 "GPT-J 6B". EleutherAI. Retrieved 29 August 2023. "Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative...
Jason Watkins Jr. (born January 21, 2005), known online as IShowSpeed (or simply Speed), is an American YouTuber and rapper. He is best known for his variety...
WWE Speed is an American professional wrestling streaming television program produced by WWE. It airs on the social media platform X and premiered with...
Deep frying (also referred to as deep fat frying) is a cooking method in which food is submerged in hot fat, traditionally lard but today most commonly...
in the West Deep (11°20.34' N, 142°13.20 E)." The depth was "obtained during swath mapping ... confirmed in both N–S and E-W swaths." Speed of sound corrections...
DeepMind Technologies Limited, doing business as Google DeepMind, is a British-American artificial intelligence research laboratory which serves as a subsidiary...
at previously unimaginable speed and scale. The time required to move from basic science to applicable technology in deep tech exceed the development...
garage and speed garage tracks and consists of a tight yet deep bass that sweeps in pitch and/or frequencies" History of Speed garage: "Speed garage can...
speed or displacement speed is the speed at which the wavelength of a vessel's bow wave is equal to the waterline length of the vessel. As boat speed...
bests of 10.35 and 20.79 seconds. The Vikings needed a receiver with deepspeed after trading Randy Moss to Oakland, drafting Williamson with the seventh...
Global Information