LexGPT 0.1: pre-trained GPT-J models with Pile of Law
| Autor: | Lee, Jieh-Sheng |
|---|---|
| Rok vydání: | 2023 |
| Předmět: | |
| Druh dokumentu: | Working Paper |
| Popis: | This research aims to build generative language models specialized for the legal domain. The manuscript presents the development of LexGPT models based on GPT-J models and pre-trained with Pile of Law. The foundation model built in this manuscript is the initial step for the development of future applications in the legal domain, such as further training with reinforcement learning from human feedback. Another objective of this manuscript is to assist legal professionals in utilizing language models through the ``No Code'' approach. By fine-tuning models with specialized data and without modifying any source code, legal professionals can create custom language models for downstream tasks with minimum effort and technical knowledge. The downstream task in this manuscript is to turn a LexGPT model into a classifier, although the performance is notably lower than the state-of-the-art result. How to enhance downstream task performance without modifying the model or its source code is a research topic for future exploration. Comment: 10 pages and 2 figures. To be published in the Proceedings of the Seventeenth International Workshop on Juris-informatics (JURISIN 2023), hosted by JSAI International Symposia on AI 2023 |
| Databáze: | arXiv |
| Externí odkaz: |
|