The release of the Dolly Large Language Model (DLL) by the business software company Databricks marked another advancement for Open Source GPT Chat.
The new ChatGPT clone has been given the name Dolly in honor of the eponymous sheep, which was the first mammal to be cloned.
Open Source Large Language Models
The Dolly LLM is the most recent case of the expanding open source AI movement, which aims to increase access to technology and prevent it from being monopolized and controlled by big businesses.
Business reluctance to provide sensitive data to a third party that manages the AI technology is one of the concerns fueling the open source AI movement.
Based on Open Source
Dolly was developed using open source models developed by the non-profit EleutherAI research institute and Stanford University’s Alpaca model, which in turn was developed using Meta’s 65 billion-parameter LLaMA model.
LLaMA, which stands for Large Language Model Meta AI, is a language model that is trained on publicly available data.
Creating a Better Dataset
An academic research paper (SELF-INSTRUCT: Aligning Language Model with Self Generated Instructions PDF) that described how to produce high quality autogenerated question and response training data that is superior to the sparse public data served as another source of inspiration.
The Self-Instruct research paper explains:
“…we curate a set of expert-written instructions for novel tasks, and show through human evaluation that tuning GPT3 with SELF-INSTRUCT outperforms using existing public instruction datasets by a large margin, leaving only a 5% absolute gap behind InstructGPT…
…Applying our method to vanilla GPT3, we demonstrate a 33% absolute improvement over the original model on SUPERNATURALINSTRUCTIONS, on par with the performance of InstructGPT… which is trained with private user data and human annotations.”
Dolly is significant because it shows how a small but high-quality dataset can be used to generate an effective large language model.
Databricks observes:
“Dolly works by taking an existing open source 6 billion parameter model from EleutherAI and modifying it ever so slightly to elicit instruction following capabilities such as brainstorming and text generation not present in the original model, using data from Alpaca.
…We show that anyone can take a dated off-the-shelf open source large language model (LLM) and give it magical ChatGPT-like instruction following ability by training it in 30 minutes on one machine, using high-quality training data.
Surprisingly, instruction-following does not seem to require the latest or largest models: our model is only 6 billion parameters, compared to 175 billion for GPT-3.”
Databricks Open Source AI
Dolly is said to democratize artificial intelligence. With the launch of Mozilla.ai, the non-profit Mozilla organization has lately joined a gowning movement. The Firefox web browser and other open source programs are published by Mozilla.
Read the full announcement by Databricks:
Hello Dolly: Democratizing the magic of ChatGPT with open models
For more such tips, updates and learning resources, stay tuned to Insitebuild Blog.
