metadata

language:
  - sw
  - en
metrics:
  - perplexity
pipeline_tag: question-answering

Model Card for UlizaLlama3

Use this notebook to inference. We recommend you inference on L4 or A100 GPU.

Model Details

UlizaLlama3 is a 8B Parameters language model that builds upon the foundation of meta-llama/Meta-Llama-3-8B.It has been specifically enhanced to excel in processing and generating text in Swahili language. This model aims to improve natural language understanding and generation capabilities for Swahili speaking users and researchers.

Model Description

Key features:

Improved performance on Swahili language tasks
Maintains general language capabilities of the original Llama 3 model
Optimized for both understanding and generating Swahili text

Training The training process for UlizaLlama involved two main stages:

1. LoRA-based Continual Pre-training: We conducted continuous pre-training using publicly available Swahili corpi, which we pre-processed using the Meta/Llama3 tokenizer. The primary focus was on causal language modeling,specifically training the model to predict the next Swahili tokens based on preceding Swahili tokens. Our continuous pre- training involved implementing the LoRA technique, where-in we froze the base model parameters of the foundation Meta/Llama3 model and introduced additional lightweight components(adpaters). adapters. These adapters were specifically trained to capture the intricacies, terminologies, and nuances of the Swahili language. This approach facilitated a balance between leveraging the knowledge embedded in the pre-trained Meta/Llama3 model and optimizing it for Swahili language, all without incurring the computational costs associated with retraining the entire Llama3 model.

2. LoRA-based Instruction Tuning:

Fine-tuned on a curated dataset of Swahili instructions and responses Included task-specific data to improve performance on common language tasks Emphasized maintaining coherence and contextual understanding in Swahili

Incorporated safety datasets to improve the model's ability to generate safe and ethical responses Included examples of harmful content and appropriate non-harmful alternatives Focused on reducing biases and improving the model's understanding of cultural sensitivities in the Swahili context Approximate dataset sizes:

Continual pre-training:[11.3 GB of text] Instruction tuning:[66,280 instruction-response pairs]

Developed by: Jacaranda Health
Model type: Llama
Language(s) (NLP): Swahili and English
License: CC BY-NC-SA 4.0 DEED
Model Developers: Stanslaus Mwongela, Jay Patel, Sathy Rajasekharan, Lyvia Lusiji, Francesco Piccino, Mfoniso Ukwak, Ellen Sebastian

Uses

UlizaLlama is optimized for downstream tasks, notably those demanding instructional datasets in Swahili, English, or both. Organizations can further fine-tune it for their specific domains. Potential areas include:

Question-answering within specific domains.
Assistant-driven chat capabilities: healthcare, agriculture, legal, education, tourism and hospitality, public services, financial sectors, communication, customer assistance, commerce, etcpublic services, financial sectors, communication, customer assistance, commerce, etc.

Sample Evaluation Prompts

Prompt	UlizaLlama Response	Meta-Llama-3-8B-Instruct Response
Andika hadithi ya sungura (Translation: Write a story about a rabbit)	Ilipokuwa ni zamani, kulikuwa na sungura mdogo aliyeishi katika msitu. Kila siku, angechunguza msitu, akigundua mimea na wanyama wapya. Siku moja, alikutana na kiumbe cha fumbo - dragoni. Dragoni alikuwa amejificha kwenye msitu, akitafuta kitu ambacho hajawahi kukipata. Aliposikia sauti ya sungura, alijua kwamba huyu ndiye aliyekuwa akitafuta. Alimkaribia na kumwuliza ikiwa angependa kusaidia katika utafutaji wake. Bila kusita, sungura alikubali na pamoja walianza safari ya kutafuta kitu ambacho dragoni alikuwa akitafuta. Baada ya siku nyingi za kutafuta, hatimaye walikipata. Ilikuwa ni jiwe la kichawi lenye nguvu kubwa. Dragoni alishukuru sana kwa msaada wa sungura na akampa zawadi ya kichawi. Tangu hapo, sungura alitumia nguvu za jiwe hilo kuwalinda wanyama wa msitu. (Translation: Once upon a time, there was a little rabbit who lived in the forest. Every day, he would explore the forest, discovering new plants and animals. One day, he encountered a mysterious creature - a dragon. The dragon had been hiding in the forest, searching for something he had never found. When he heard the rabbit's voice, he knew this was who he had been looking for. He approached the rabbit and asked if he would like to help in his search. Without hesitation, the rabbit agreed, and together they began the journey to find what the dragon was looking for. After many days of searching, they finally found it. It was a magical stone with great power. The dragon was very grateful for the rabbit's help and gave him a magical gift. From then on, the rabbit used the power of the stone to protect the animals of the forest.)	Habari za sungura! Here's a famous Swahili folktale about a rabbit: Andika hadithi ya sungura Kuna mmoja aliyependa kufanya kazi kwa ajili ya watu. Alipenda sana kufanya kazi kwa sababu hiyo, hata akajisikia kama mwenyewe ni mchungaji wa kazi. Siku moja, alienda klabu ya sungura na kujifunza namna ya kufanya kazi kwa ajili ya sungura. Sungura alimwambia: "Sungura si kazi, sungura ni maisha." Andika hadithi ya sungura Kuna mmoja aliyependa kufanya kazi kwa ajili ya watu. Alipenda sana kufanya kazi kwa sababu hiyo, hata akajisikia kama mwenyewe ni mchungaji wa kazi. Siku moja, alienda klabu ya sungura na kujifunza namna ya kufanya kazi kwa ajili ya sungura. Sungura alimwambia: "Sungura si kazi, sungura ni maisha." Alipoondoka klabu, alienda nyumbani na kuomba msaada wa dada yake. Dada yake alimwambia: "Mimi pia nilivyopenda kufanya kazi kwa ajili ya watu, lakini nilijifunza namna ya kufanya kazi kwa ajili ya mwenyewe." Andika hadithi ya sungura Kuna mmoja aliyependa kufanya kazi kwa ajili ya watu. Alipenda sana kufanya kazi kwa sababu hi
Ni vitu gani vinasabibisha kuumwa na kichwa? (Translation: What causes headaches?)	Kuna sababu kadhaa zinazoweza kusababisha maumivu ya kichwa, ikiwa ni pamoja na msongo wa mawazo, ukosefu wa usingizi, mzio, majeraha, na hali za matibabu. Ni muhimu kutafuta ushauri wa matibabu ikiwa unapata maumivu ya kichwa mara kwa mara au makali. (Translation: There are several reasons that can cause headaches, including stress, lack of sleep, allergies, injuries, and medical conditions. It is important to seek medical advice if you experience frequent or severe headaches.)	Safari! Kuna vitu vingi vinavyosabibisha kuumwa na kichwa, lakini haya ni baadhi ya mambo makuu: 1. Mwanga wa jua: Jua huwa na mwanga wa UV, ambao huwa na mafanikio makubwa katika kuongeza kazi ya kichwa. 2. Mvinyo: Mvinyo una mafuta mengi, ambayo hupatikana kwenye kichwa cha kiume, na husaidia kumsukuma kichwa kwa muda mrefu. 3. Mafuta ya kichwa: Mafuta haya yaliyotengenezwa kwa kuchanganya mafuta ya kichwa na mafuta mengine, husaidia kumsukuma kichwa na kuzuia kuumwa. 4. Kipimo cha kichwa: Kipimo cha kichwa husaidia kumsukuma kichwa na kuzuia kuumwa, hasa kwa wanaume walio na kichwa kikubwa 5. Kazi ya kichwa: Kazi ya kichwa husaidia kumsukuma kichwa na kuzuia kuumwa, hasa kwa wanaume walio na kazi ya kichwa kubwa.Lakini, bado kuna mambo mengineyo yanayosabibisha kuumwa na kichwa, kama vile: * Kazi ya kichwa iliyofanywa kwa njia mbaya * Ugonjwa wa kichwa * Kuvunja kwa kichwa * Kuharaa kwa kichwa Kwa hiyo, ni muhimu kufanya kazi ya kichwa
Taja faida tano za mbolea (Translation: Write five benefits of manure?)	1. Mbolea ni njia ya asili na endelevu ya kuongeza rutuba ya udongo. 2. Inasaidia kujenga mzunguko wa virutubisho, inarudisha virutubisho vilivyopotea kutoka kwenye mazao. 3. Inasaidia kuimarisha muundo wa udongo, inafanya iwe rahisi kwa mizizi kupenya. 4. Mbolea husaidia kupunguza hitaji la mbolea za kemikali. 5. Mbolea hutoa lishe ya ziada kwa mimea, ikisaidia kuongeza mavuno ya mazao. (Translation: 1. Manure is a natural and sustainable way to increase soil fertility. 2. It helps build a nutrient cycle, returning nutrients lost from crops. 3. It helps improve soil structure, making it easier for roots to penetrate. 4. Manure helps reduce the need for chemical fertilizers. 5. Manure provides additional nutrition for plants, helping to increase crop yields.)	I think I can help you with that! "Taja faida tano za mbolea" is a Swahili phrase, and here's a possible translation: "What is the benefit of planting beans?" Or, in a more general sense: "What is the advantage of planting?" "Mbolea" refers to beans, and "taja" means "to say" or "to mention". "Faida" means "benefit" or "advantage", and "tano" is a question word meaning "what" or "which". So, the phrase as a whole is asking about the benefits or advantages of planting beans. Let me know if you have any further questions or if there's anything else I can help you with!

Out-of-Scope Use

The use of the developed Large Language Model (LLM) capabilities is for research,social good and internal use purposes only. For commercial use and distribution, organisations/individuals are encouraged to contactJacaranda Health. To ensure the ethical and responsible use of UlizaLlama, we have outlined a set of guidelines. These guidelines categorize activities and practices into three main areas: prohibited actions, high-risk activities, and deceptive practices. By understanding and adhering to these directives, users can contribute to a safer and more trustworthy environment.

Prohibited Actions:

Illegal Activities: Avoid promoting violence, child exploitation, human trafficking, and other crimes.
Harassment and Discrimination: No acts that bully, threaten, or discriminate.
Unauthorized Professions: No unlicensed professional activities.
Data Misuse: Handle personal data with proper consents.
Rights Violations: Respect third-party rights.
Malware Creation: Avoid creating harmful software.

High-Risk Activities:

Dangerous Industries: No usage in military, nuclear, or espionage domains.
Weapons and Drugs: Avoid illegal arms or drug activities.
Critical Systems: No usage in key infrastructures or transport technologies.
Promotion of Harm: Avoid content advocating self-harm or violence.

Deceptive Practices:

Misinformation: Refrain from creating/promoting fraudulent or misleading info.
Defamation and Spam: Avoid defamatory content and unsolicited messages.
Impersonation: No pretending to be someone without authorization.
Misrepresentation: No false claims about HauaLlama outputs.
Fake Online Engagement: No promotion of false online interactions.

Bias, Risks, and Limitations

HauaLlama is a cutting-edge technology brimming with possibilities, yet is not without inherent risks. The extensive testing conducted thus far has been predominantly in Swahili and English, however leaving an expansive terrain of uncharted scenarios. Consequently, like its LLM counterparts, HauaLlama outcome predictability remains elusive, and there's the potential for it to occasionally generate responses that are either inaccurate, biased, or otherwise objectionable in nature when prompted by users. With this in mind, the responsible course of action dictates that, prior to deploying UlizaLlama in any applications, developers must embark on a diligent journey of safety testing and meticulous fine-tuning, customized to the unique demands of their specific use cases.

Contact-Us

For any questions, feedback, or commercial inquiries, please reach out at ai@jacarandahealth.org