Google has unveiled Gemma, an open large language model inspired by Gemini’s technology. Gemma is a potent yet lightweight model, designed for efficient use in resource-constrained environments, such as laptops or cloud infrastructure. It serves a versatile role, enabling the creation of chatbots, content generation tools, and various other language model applications—an eagerly anticipated tool for SEO professionals.
Available in two versions, Gemma boasts two billion parameters (2B) and a more robust seven billion parameters (7B). The parameter count reflects the model’s complexity and potential capabilities; higher parameter models possess a deeper understanding of language and generate more intricate responses. However, they also demand greater resources for training and execution.
Google’s release of Gemma aims to democratize access to cutting-edge Artificial Intelligence, emphasizing its default safety and responsibility. Additionally, it provides a toolkit for further optimization, allowing users to fine-tune the model for enhanced safety measures.
Released As An Open Model (Variant Of Open Source)
Gemma is accessible to everyone for both commercial and non-commercial purposes under an open license. Unlike a traditional open-source license, this open license includes usage terms, specifically designed to prevent malicious utilization.
Google discussed this on their Open Source Blog, emphasizing that while conventional open-source licenses offer unrestricted usage of technologies, they advocate for a more cautious approach with AI technology. They propose an Open License variant, which permits free use while imposing restrictions to prevent harmful applications, thereby granting users the freedom to innovate responsibly with the technology.
The open source explainer about Gemma explains:
The Gemma models’ terms of use make them freely available for individual developers, researchers, and commercial users for access and redistribution. Users are also free to create and publish model variants. In using Gemma models, developers agree to avoid harmful uses, reflecting our commitment to developing AI responsibly while increasing access to this technology.
Gemma By DeepMind
The model is developed to be lightweight and efficient which makes it ideal for getting it into the hands of more end users.
Google’s official announcement noted the following key points:
- “We’re releasing model weights in two sizes: Gemma 2B and Gemma 7B. Each size is released with pre-trained and instruction-tuned variants.
- A new Responsible Generative AI Toolkit provides guidance and essential tools for creating safer AI applications with Gemma.
- We’re providing toolchains for inference and supervised fine-tuning (SFT) across all major frameworks: JAX, PyTorch, and TensorFlow through native Keras 3.0.
- Ready-to-use Colab and Kaggle notebooks, alongside integration with popular tools such as Hugging Face, MaxText, NVIDIA NeMo and TensorRT-LLM, make it easy to get started with Gemma.
- Pre-trained and instruction-tuned Gemma models can run on your laptop, workstation, or Google Cloud with easy deployment on Vertex AI and Google Kubernetes Engine (GKE).
- Optimization across multiple AI hardware platforms ensures industry-leading performance, including NVIDIA GPUs and Google Cloud TPUs.
- Terms of use permit responsible commercial usage and distribution for all organizations, regardless of size.”
Analysis Of Gemma
In a study conducted by Awni Hannun, a machine learning research scientist at Apple, it was found that Gemma has been finely tuned for exceptional efficiency, making it particularly suitable for deployment in environments with limited resources.
Hannun highlighted Gemma’s remarkable vocabulary, boasting 250,000 tokens compared to the 32,000 tokens found in similar models. This expanded vocabulary equips Gemma with the ability to recognize and process a diverse range of words, enhancing its capability to handle tasks involving complex language. The extensive vocabulary, as per Hannun’s analysis, contributes to Gemma’s adaptability across various content types, including mathematics, code, and other modalities.
Additionally, the analysis noted the substantial size of Gemma’s “embedding weights,” amounting to 750 million. These embedding weights play a crucial role in mapping words to their meanings and relationships. Hannun emphasized that these weights are not only utilized in processing input but are also integral to generating the model’s output. This shared usage enhances the model’s efficiency by leveraging its understanding of language for both input processing and text generation.
For end users, this translates into Gemma providing more precise, relevant, and contextually appropriate responses, improving its effectiveness in content generation, chatbot interactions, and translation tasks.
He tweeted:
“The vocab is massive compared to other open source models: 250K vs 32k for Mistral 7B
Maybe helps a lot with math / code / other modalities with a heavy tail of symbols.
Also the embedding weights are big (~750M params), so they get shared with the output head.”
He followed up that there are more optimizations in data and training but that those two factors are what especially stood out.
Designed To Be Safe And Responsible
A crucial aspect of its design is its inherent focus on safety, making it well-suited for practical deployment. The training data underwent thorough filtration to eliminate any personal or sensitive information. Google implemented reinforcement learning from human feedback (RLHF) to instill responsible behavior in the model.
Extensive debugging ensued, involving manual re-evaluation, automated testing, and scrutiny for potential capabilities linked to undesirable or hazardous activities.
Google also released a toolkit for helping end-users further improve safety:
“We’re also releasing a new Responsible Generative AI Toolkit together with Gemma to help developers and researchers prioritize building safe and responsible AI applications. The toolkit includes:
- Safety classification: We provide a novel methodology for building robust safety classifiers with minimal examples.
- Debugging: A model debugging tool helps you investigate Gemma’s behavior and address potential issues.
- Guidance: You can access best practices for model builders based on Google’s experience in developing and deploying large language models.”
Read Google’s official announcement here.
For more such updates, tips and learning resources, stay tuned to Insitebuild Blog.
