Byte pair encoding[1][2] (also known as digram coding)[3] is an algorithm, first described in 1994 by Philip Gage for encoding strings of text into tabular form for use in downstream modeling.[4] Its modification is notable as the large language model tokenizer with an ability to combine both tokens that encode single characters (including single digits or single punctuation marks) and those that encode whole words (even the longest compound words).[5][6][7] This modification, in the first step, assumes all unique characters to be an initial set of 1-character long n-grams (i.e. initial "tokens"). Then, successively the most frequent pair of adjacent characters is merged into a new, 2-character long n-gram and all instances of the pair are replaced by this new token. This is repeated until a vocabulary of prescribed size is obtained. Note that new words can always be constructed from final vocabulary tokens and initial-set characters.[8]
All the unique tokens found in a corpus are listed in a token vocabulary, the size of which, in the case of GPT-3.5 and GPT-4, is 100256.
The difference between the modified and the original algorithm is that the original algorithm does not merge the most frequent pair of bytes of data, but replaces them by a new byte that was not contained in the initial dataset. A lookup table of the replacements is required to rebuild the initial dataset. The algorithm is effective for tokenization because it has low computational overhead and remains consistent and reliable.
^Gage, Philip (1994). "A New Algorithm for Data Compression". The C User Journal.
^"A New Algorithm for Data Compression". Dr. Dobb's Journal. 1 February 1994. Retrieved 10 August 2020.
^Witten, Ian H.; Moffat, Alistair; Bell, Timothy C. (1994). Managing Gigabytes. New York: Van Nostrand Reinhold. ISBN 978-0-442-01863-4.
^"Byte Pair Encoding". Archived from the original on 2016-03-26.
^Sennrich, Rico; Birch, Alexandra; Haddow, Barry (2015-08-31). "Neural Machine Translation of Rare Words with Subword Units". arXiv:1508.07909 [cs.CL].
^Paaß, Gerhard; Giesselbach, Sven (2022). "Pre-trained Language Models". Foundation Models for Natural Language Processing. Artificial Intelligence: Foundations, Theory, and Algorithms. pp. 19–78. doi:10.1007/978-3-031-23190-2_2. ISBN 9783031231902. Retrieved 3 August 2023.
and 17 Related for: Byte pair encoding information
Bytepairencoding (also known as digram coding) is an algorithm, first described in 1994 by Philip Gage for encoding strings of text into tabular form...
embedding is associated to the integer index. Algorithms include byte-pairencoding and WordPiece. Probabilistic tokenization also compresses the datasets...
Explorer, a children's animated television show. Dual-Tile encoding, another name for bytepairencoding Directorate of Technical Education, Maharashtra, an...
the attachment. Base64 encoding causes an overhead of 33–37% relative to the size of the original binary data (33% by the encoding itself; up to 4% more...
differential encoding algorithms include: Delta modulation quantizes and encodes differences between consecutive audio samples by encoding the derivative...
Unicode encodings. Two situations are considered: 8-bit-clean environments (which can be assumed), and environments that forbid use of byte values that...
certain issues encoding vocabulary with word tokens by using bytepairencoding. This permits representing any string of characters by encoding both individual...
Windows: Encoding.Convert – .NET API MultiByteToWideChar/WideCharToMultiByte – to convert from ANSI to Unicode & Unicode to ANSI Percent-encoding Alt code...
indefinite encoding, the parser must pair the break markers with the corresponding indefinite-length header bytes. Type 5 is similar but encodes a map (also...
the list of symbol pairs. Context-free grammar Data compression Lossless data compression Straight-line grammar Bytepairencoding Nevill-Manning, C.G...
protocol). PDU max size is 253 bytes. ADU max size on RS232/RS485 network is 256 bytes, and with TCP is 260 bytes. For data encoding, Modbus uses a big-endian...
The byte is a unit of digital information that most commonly consists of eight bits. Historically, the byte was the number of bits used to encode a single...
is a device or computer program capable of encoding or decoding a digital data stream (a codec) that encodes or decodes audio. In software, an audio codec...
(such as bytepairencoding, also called dual tile encoding or DTE, in which certain combinations of two or more letters are encoded as one byte) which...
sequence. Of the 16 bits that make up these two bytes, 11 bits go to encoding the distance, 3 go to encoding the length, and the remaining two are used to...
Express (.500 BPE) Borated polyethylene, a lightweight neutron absorber Bytepairencoding ASME BPE, a standard published by the American Society of Mechanical...
be percent-encoded in HTML forms to "%7E". The encoding of SPACE as '+' and the selection of "as-is" characters distinguishes this encoding from RFC 3986...
Global Information