Towards the end of September 2022, Georgi Gerganov started work on the GGML library, a C library implementing tensor algebra. Gerganov developed the library with the intention of strict memory management and multi-threading. The creation of GGML was inspired by Fabrice Bellard's work on LibNC.[8]
Before llama.cpp, Gerganov worked on a similar library called whisper.cpp which implemented Whisper, a speech to text model by OpenAI.[9]
Gerganov has a background in medical physics, and was part of the Faculty of Physics in Sofia University.[10] In 2006 he won a silver medal in the International Physics Olympiad.[11][12] In 2008 he won a programming competition organized by the Bulgarian Association of Software Companies, PC Magazine and Musala Soft, a Bulgarian software services company.[13]
Development
llama.cpp began development in March 2023 by Georgi Gerganov as an implementation of the Llama inference code in pure C/C++ with no dependencies. This improved performance on computers without GPU or other dedicated hardware, which was a goal of the project.[3][14][15] llama.cpp gained traction with users who lacked specialized hardware as it could run on just a CPU including on Android devices.[14][16][17] While initially designed for CPUs, GPU inference support was later added.[18] As of November 2024 it has more than 67,000 stars on GitHub.[19]
In March 2024 Justine Tunney introduced new optimized matrix multiplication kernels for x86 and ARM CPUs, improving prompt evaluation performance for FP16 and 8-bit quantized data types.[20] These improvements were committed upstream to llama.cpp.[20] Tunney also created a tool called llamafile that bundles models and llama.cpp into a single file that runs on multiple operating systems via the Cosmopolitan Libc library also created by Tunney which allows C/C++ to be more portable across operating systems.[20]
Architecture
llama.cpp supports multiple hardware targets including x86, ARM, CUDA, Metal, Vulkan and SYCL.[21][22][23][24] These back-ends make up the GGML tensor library which is used by the front-end model-specific llama.cpp code.[25] llama.cpp supports ahead of time model quantization as opposed to on-the-fly quantization.[26] llama.cpp makes use of several CPU extensions for optimization: AVX, AVX2 and AVX-512 for X86-64, and Neon on ARM. Apple silicon is an important target for the project.[19][27] It supports grammar-based output formatting as JSON.[15] It also supports speculative decoding.[7]
The GGUF (GGML Universal File)[30] file format is a binary format that stores both tensors and metadata in a single file, and is designed for fast saving, and loading of model data.[31] It was introduced in August 2023 by the llama.cpp project to better maintain backwards compatibility as support was added for other model architectures.[18][32] It succeeded previous formats used by the project such as GGML.
GGUF files are typically created by converting models developed with a different machine learning library such as PyTorch.[31]
Design
The format focuses on quantization, the act of reducing precision in the model weights. This can lead to reduced memory usage, and increased speed at the expense of lower model accuracy.[33][32]
GGUF supports 2-bit to 8-bit quantized integer types;[34] common floating-point data formats such as float32, float16, and bfloat16; and 1.56 bit quantization.[5]
This file format contains information necessary for running a GPT-like language model such as the tokenizer vocabulary, context length, tensor info and other attributes.[35]
^ abRajput, Saurabhsingh; Sharma, Tushar (4 June 2024). "Benchmarking Emerging Deep Learning Quantization Methods for Energy Efficiency". 2024 IEEE 21st International Conference on Software Architecture Companion (ICSA-C). pp. 238–242. doi:10.1109/ICSA-C63560.2024.00049. ISBN979-8-3503-6625-9.
^Gerganov, Georgi; Nguyen, Xuan Son; Slaren (August 13, 2024). "Introduction to ggml". Huggingface.
^Kluska, Piotr; Castell´o, Adri´an; Scheidegger, Florian; I. Malossi, A. Cristiano; Quintana-Ort´ı, Enrique (June 2024). "QAttn: Efficient GPU Kernels for mixed-precision Vision Transformers"(PDF). Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops.
^Cabezas, Darío; Fonseca-Delgado, Rigoberto; Reyes-Chacón, Iván; Vizcaino-Imacaña, Paulina; Morocho-Cayamcela, Manuel (2024). "Integrating a LLaMa-based Chatbot with Augmented Retrieval Generation as a Complementary Educational Tool for High School and College Students". Proceedings of the 19th International Conference on Software Technologies. pp. 395–402. doi:10.5220/0012763000003753. ISBN978-989-758-706-1.
^Dong, Bo; Lin, Jun; Yu, Zhentao; Xu, Zhenzhong; Luo, Yu; Chang, Hanwen; Shen, Haihao (July 2024). "Accelerating GGUF Models with Transformers". The Parallel Universe. No. 57. Intel. pp. 28–33.
