We have previously released our large language model (LLM), 42dot LLM. The released 1.3 billion parameter models were trained using our own carefully collected internal data and training infrastructure. 42dot LLM stands out as a Korean-English bilingual model, being the first of its kind to be accessible to the public in Korea. Our model demonstrates superior performance on various quantitative and qualitative benchmarks compared to other competitive models of similar size. The release comprised two types of LLMs: 42dot LLM-PLM, a Pre-trained Language Model, and 42dot LLM-SFT, a Supervised Fine-Tuned language model, which was fine-tuned from the PLM to better align with human usability.
[Demo of 42dot LLM]
1. SFT Model
Supervised Fine-Tuning is a technique that uses a small, carefully labeled dataset to adjust the weights and biases of a pre-trained model to make it more aligned for a specific task. Incorporating a diverse range of instruction-following data in the fine-tuning dataset can enhance the usability of a language model (LLM), making it more human-friendly for communication and interaction. Large language models that are aligned through this technique are termed SFTs, in order to differentiate from the models aligned using a different alignment technique called reinforcement learning with human feedback (RLHF). One of our released models, 42dot LLM-SFT was aligned in this way. And you can interact and communicate easily with this model similar to the well-known ChatGPT.
2. LLaMA Architecture
42dot LLM consists of transformer decoders like the other LLMs, and is constructed with an architecture compatible with LLaMA, one of the most popular LLMs open-sourced by Facebook (Meta). We opted to align our architecture with LLaMA primarily to ensure compatibility with well-established open-source frameworks. Consequently, our models seamlessly run on popular frameworks, enabling their application in Korean contexts. This stands in contrast to LLaMA models, which lack support for the language. It's essential to note that our models were developed from scratch and are qualitatively different and independent of the available LLaMA models.
3. Model Size
The 42dot LLM consists of 24 layers, resulting in around 1.3 billion parameters in total. It was trained with over 1.4 trillion tokens, which is of similar scale to the other open-source LLMs such as the above-mentioned LLaMA. We emphasize again, unlike the other LLMs, our data includes a large amount of high-quality Korean texts as well as English texts, resulting in a Korean-English bilingual model that can understand both languages fluently.
4. Tokenizer
The process of parsing a sentence into smaller units, such as words or syllables, is called tokenization. These units form a vocabulary utilized by a tokenizer to represent texts in a machine-readable format, presented as a sequence of tokens for processing by Language Models (LLMs). The Byte-Pair Encoding (BPE) algorithm, among various methods, is employed to build such a vocabulary. BPE dissects text into bytes or characters, retaining the most frequently occurring pairs. This method proves particularly advantageous for the Korean language, a highly agglutinative language rich in morphology with a unique writing system. Representing the Korean language poses challenges due to varied ways words and phrases can be tokenized depending on context. BPE adeptly captures these nuances by representing the language as a sequence of sub-word tokens, which can be combined to form larger linguistic units such as words or phrases. Given these advantages, 42dot LLM utilizes BPE tokenization, a choice also embraced by other prominent models like OpenAI's GPT. A custom dictionary, comprising approximately 50,000 BPE sub-word tokens in both Korean and English, was crafted by sampling from a vast pool of over 10 million documents.
5. GPU Training Cluster
42dot has garnered a large amount of compute in order to train powerful LLMs by establishing its own high-performance GPU training cluster, with a capacity to run 320 GPUs simultaneously. Utilizing this infrastructure, we trained 42dot LLM-PLM on A100 80G GPUs with a total of 49,152 GPU hours, which is equivalent to about one week of simultaneous utilization of 320 GPUs. An additional 112 GPU hours of compute was used for fine-tuning the PLM for 42dot LLM-SFT.
6. Evaluation Result
The resulting 42dot LLMs outperform other similarly sized large language models in various zero-shot evaluation metrics, in both Korean and English. In a comprehensive dialogue performance evaluation benchmark featuring 121 prompts across ten different categories, our model outperforms similarly sized LLMs by a significant margin. Notably, it is noteworthy that our model's performance is closely competitive with GPT-3.5, a model over 100 times larger, boasting an estimated 175 billion parameters.
7. Public LLM
We’ve made the models freely available, along with a generation script so that anyone can easily test and run our LLMs. The models are governed by the CC BY-NC 4.0 license, allowing individuals to utilize them for non-commercial purposes, provided appropriate credit is given.
We believe in the power of the open source ecosystem.
Like any LLM, 42dot LLM is not free of limitations and can show signs of hallucinations by generating inaccurate, misleading or inconsistent responses. However, empowered by the potential of open sourcing, we believe that these challenges can be addressed in the foreseeable future.
With the introduction of 42dot's Korean-English bilingual language models, we aim to take a positive stride towards enhancing the Korean LLM ecosystem.
LLM Group
We are developing a large language model to provide new mobility experiences.