Across Africa, millions of women communicate primarily in their local languages, which feature diverse dialects that vary widely between urban and rural areas. Through fieldwork, surveys, and interviews with communities, our social and behavioral researchers found that rural communities have distinct ways of communicating that can be difficult for urban professionals to fully grasp. This language barrier makes it challenging for doctors and healthcare workers to accurately interpret women’s symptoms and experiences, ultimately hindering the delivery of effective care and support.
These challenges reveal two gaps gaps:
- The Medical Research Gap : Women’s health concerns, particularly in maternal healthcare, are underrepresented in both research and policy due to language barriers. In many African contexts, important healthcare conversations take place in local languages and often go undocumented, contributing to the underrepresentation of women’s lived experiences in medical research.
- The Technology Gap: Beyond Technical Benchmarks: Traditional ASR systems often struggle with low-resource languages; the challenge becomes more pronounced in specialized domains such as healthcare. Even when trained on domain-specific data, they still struggle to capture local accents, code-switching, and traditional expressions.
Automatic Speech Recognition (ASR) has significant potential to overcome literacy and accessibility barriers. By enabling women to report symptoms, access health information, and interact with digital platforms in their native languages, voice AI could transform maternal healthcare for millions of Pulaar speakers across Senegal and West Africa. However, the challenge goes beyond making speech recognition technology work. It also involves ensuring that people can communicate using their own languages, accents, and dialects and that the technology can accurately understand them.
From Understanding the Challenge to Taking Action
To address these challenges, we worked with partners bringing expertise in research, technology, gender, and African communities, including IPAR, AI4D Africa, and Sunbird AI to develop and apply a GEDI framework to assess whether the ASR system is gender-responsive, equitable, diverse, and inclusive.
The GEDI matrix was developed and tested throughout the project, covering key stages of the AI lifecycle, including data collection, model development, evaluation, deployment and user testing.
Building a GEDI Matrix for Pulaar ASR
The framework assesses where innovation might unintentionally exclude or harm the populations it aims to serve. It helped us identify gaps across women, communities, dialects, and regions, using these insights to improve the system and embed the GRAIN GEDI “Design-by-Inclusion” Standard throughout the ASR engine lifecycle.
Stage 1 & 2 : Community Engagement & Data collection
The initial phase focused on deeply understanding the target community, assessing local access to technology, and aligning our research objectives with real-world needs. A central question guided this stage: Are we addressing a genuine community priority, a technical research gap, or both?
- Avoiding the Tororo Trap : The GRAIN GEDI framework describes it as a "Tororo Trap" as the tendency to prioritize research objectives and technical performance (such as Word Error Rate) over community needs.
- Epistemic Power & Decision-Making Power : To ensure the technology was built with the community rather than for them, we conducted an internal Epistemic Power Audit to evaluate who holds decision-making authority over the product's core logic.
- Multidisciplinary & Inclusive Team: One key learning from the GEDI was the need to involve a diverse team, including people with rural experience and community workers who understand how people currently communicate their health experiences.
This community-led approach directly informed the development of our dataset. We collected recordings from 40 speakers from diverse backgrounds, including women with disabilities, representing two key Pulaar dialects: Fouta Toro and Fouta Djalion. We worked with two translators and an expert linguist to ensure linguistic accuracy. We also engaged two maternal health experts to validate healthcare terminology across both dialects. This approach helped ensure that marginalized voices and regional language variations were represented in the dataset from the beginning.
Stage 3 & 4 : Model Training & Evaluation
Fine-Tuning Beyond the Baseline
We benchmarked leading open-source foundation models on our Pulaar audio dataset. Generic baselines struggled, with a ~43% Word Error Rate (WER). By fine-tuning on our dual-dialect dataset covering Fouta Tooro and Fouta Djallon, we reduced WER to 28%, a 15-point improvement. However, average performance can hide bias. A model may perform well overall while still failing on specific groups, such as rural female speakers.
- The GEDI Lens: Auditing Hidden Bias & Equity in speech technology
To ensure high accuracy did not come at the expense of equity, we passed our fine-tuned checkpoints through a rigorous GEDI Evaluation Matrix:
- Dialectal erasure, where minority dialects or regional variants are treated as noise rather than linguistic diversity.
- Acoustic gender gaps, where models perform worse on female speech patterns or differences in pitch and vocal characteristics.
- Accent disparities, where certain regional or rural accents are recognized less accurately than more represented accents.
- Code-switching bias, where switching between Pulaar, French, or other languages reduces transcription accuracy.
Stage 5: Deployment, Accessibility & Responsible AI
One GEDI strategy stands out “designing for the most marginalized women””, we deployed our models to account for limited connectivity, low device power, and varying digital literacy.
- Inclusive System Design: To support women with limited digital literacy, low-spec devices, and unstable connectivity, we built Nakala.ai as a lightweight, mobile-friendly platform. It integrates our Pulaar, Wolof, and Hausa ASR models for low-resource environments.
- Overcoming the "Black Box" & Trust Deficit: AI can make mistakes, especially in healthcare. To build trust, we clearly informed users that transcriptions may contain errors and should be verified by a human before making health decisions.
- Ethical Data Ownership: We prioritized user consent and data dignity by explicitly informing users how their audio data is processed, stored, and utilized on the platform.
Stage 6: Testing Beyond the Laboratory
We tested the system not only for speech-to-text accuracy, but also for how well it serves underserved Pulaar-speaking women in real-world healthcare contexts.
Testing approach under GEDI Lens
- Real-world conditions: Tested in both quiet and noisy environments.
- Speech robustness: Spontaneous speech participants created their own narratives and spoke naturally to test real-time transcription in realistic situations and predefined scenarios, participants followed structured scenarios designed to assess how accuracy varied across accent, region, gender, and dialect.
- Usability & accessibility: Evaluated ease of use, comprehension, and overall user experience using likert scale
- Cultural & linguistic fit: Examined dialects, local expressions, and lived contexts.
Scoring Principle: Each aspect is rated from 1 to 5 using clear criteria to ensure consistent evaluation across participants and testing sessions. (1 = Very Poor → 5 = Excellent)
Participants
We worked with ~10 users from diverse speaker groups and dialects across Senegal. Most participants were women, with a focus on underserved communities. The testing also included 2 community health workers and 2 linguists to assess health terminology, local expressions, and dialectal accuracy.
Key Results
- ASR Quality - 4/5: Good spelling and readability. Minor errors generally did not affect meaning or comprehension.
- Dialect & Linguistic Fidelity - 3/5: The system struggled with some dialect-specific expressions, accents, and spelling conventions, highlighting the need for more dialect-specific data and evaluation.
- Platform Usability - 4/5: Users completed the main tasks without major blockers and reported good satisfaction, confidence, and accessibility.
Conclusion: Framework Evaluation and Next Steps
Building inclusive speech recognition technology requires going beyond accuracy to understand who the technology serves and who may be left behind. Through the GEDI framework, we demonstrated how community participation, gender and dialect considerations, and responsible design practices can be integrated throughout the ASR development lifecycle. Our findings show that fine-tuning can significantly improve Pulaar speech recognition, while real-world testing helps identify gaps that technical benchmarks alone may not capture. The next step is to expand this approach to include more dialects and underserved communities, ensuring that African women can communicate in their own languages and be accurately understood by the technology.
Authors: YUX Research Team
Partner: IPAR, DRC-CRDI, sunbird, CSEA Africa, AI4D Africa
Project : GRAIN EmpowerherIA initiative