Murshid
مرشد — "Guide." An Arabic-first AI screen companion that sees your screen, remembers your goal, and walks you through complex software one step at a time.
- Engineering across the stack: Electron app, Fanar API integration, agentic loop, session memory, speech pipeline, prompt engineering, live demo
- Fanar Hackathon 2026 · QCRI / HBKU · team of 3
- June 2026 · 4 days across 2 weekends
- GitHub
- Fanar models orchestrated
- 6
- Fanar models orchestrated
- vision API request limit worked around
- 20/day
- vision API request limit worked around
- exchanges of session memory kept per call
- 6
- exchanges of session memory kept per call
The problem
Qatar's digital transformation is pushing critical services online: business registration, health insurance, residency renewals. Plenty of Arabic-speaking users handle a phone and WhatsApp comfortably but hit a wall inside bureaucratic portals and banking systems. That wall has real costs. Missed deadlines, wrong submissions, financial penalties, or having to depend on someone else for basic tasks.
Existing AI assistants don't help here, because they can't see. A chatbot answering questions about a government portal it cannot look at is guessing. The missing piece wasn't intelligence, it was context.
The idea
Murshid lives near your cursor as a floating overlay. Press a hotkey, ask your question in Arabic, and it captures your screen, works out what you're looking at, and tells you exactly one thing to do next, by voice. Then it waits. When you're done you press the hotkey again, it takes a new screenshot, and the loop continues. It knows your goal from step one and holds it across the whole session.
One more thing sets it apart. Select any text with Islamic or Sharia terminology (common in Gulf financial and legal forms) and Murshid surfaces cited Quran and Hadith references through Fanar-Sadiq, with madhab-aware reasoning.
Architecture
Six Fanar models compose into a single stateful loop. Speech comes in through Aura-STT, the screen goes to Oryx for visual understanding, Sadiq-Agentic reasons over both plus the full session memory to decide the single next step, and Aura-TTS speaks the answer back in Arabic. Guard filters everything for safety. A parallel path routes selected text to Sadiq for Islamic reference lookup.
The design decision that made it work: the user's hotkey press is the "I'm done" signal. There's no automatic screen polling and no guessing about when an action finished. The human sets the pace.
Hotkey
Push-to-talk — the user asks in Arabic
Aura-STT-1
Speech to text, dialect normalised to MSA
desktopCapturer
Screenshot of the current screen
Oryx-IVU-2
Vision model reads what's on screen
Sadiq-Agentic
Reasons over goal + last 6 exchanges → exactly one next step
Aura-TTS-2
Speaks the step in Arabic; card appears near the cursor
Wait
The user acts — their next hotkey press closes the loop
parallel path — select any text → Sadiq → cited Quran/Hadith reference card · Guard-2 filters every response
Fanar-Aura-STT-1
Arabic speech → text, Gulf dialect normalised to MSA
Fanar-Oryx-IVU-2
Reads the screenshot — what is the user looking at?
Fanar-Sadiq-Agentic
Holds the goal, reasons over memory, picks the one next step
Fanar-Sadiq
Cited Quran & Hadith lookup with madhab-aware reasoning
Fanar-Aura-TTS-2
Speaks each step back in natural Arabic
Fanar-Guard-2
Filters every response before it reaches the user
Engineering decisions
Manual re-trigger over automatic screen polling
Detecting when a user finishes an action is an open research problem; a hotkey press is a solved one. Making the user's re-trigger the loop signal cut an entire class of bugs and made the interaction predictable. Simple beat clever, and we had four days.
A cached response layer, out of necessity
Fanar-Oryx allows 20 vision requests per day. One testing session burns through that. I built a cache of pre-computed responses for known demo screens, with the live API used only on cache misses. Testing and the live demo would have been impossible without it.
Session memory capped at six exchanges
Full conversation history bloats every API payload and slows every response. Capping memory at the last six exchanges while always preserving the goal in the system prompt kept context useful and requests fast.
Prompting for exactly one step
LLMs want to answer with a complete guide. The system prompt instructs: one step, one action, no lists, no paragraphs. Getting a model to say less took more iteration than any API integration in the project.
What was hard
The overlay had to stay on top of every window without blocking the applications it was guiding users through.
Configured the Electron BrowserWindow always-on-top with a transparent background and click-through regions for the non-interactive areas, so it stays visible without getting in the way.
Arabic speech input mixes Gulf dialect and Modern Standard Arabic, which degraded transcription quality downstream.
Added an STT post-processing step that normalises dialect input to MSA before it reaches the vision-reasoning stage.
Screenshot capture permissions behave differently across operating systems.
Used Electron's desktopCapturer with explicit permission prompts on first launch rather than fighting per-OS quirks silently.
Outcome
We demoed Murshid live at the hackathon final: an 8-minute presentation and working demo in front of QCRI judges and HBKU faculty. The full loop ran on stage. Voice in, screen understanding, step-by-step guidance, voice out, plus the Islamic reference card on text selection.
The project used most of the models Fanar offers and composed them into a working agentic system. What made it feel different from a chatbot is that its entire intelligence is grounded in whatever is on your screen right now.
SYSTEM — Murshid guidance loop
You are Murshid, a patient Arabic-first guide.
User's goal: {session.goal} ← pinned, survives every turn
Context: screen understanding + last 6 exchanges (hard cap)
Rules:
· Give exactly ONE step. One action.
· No lists. No paragraphs.
· Then wait — the next hotkey press means
"done, what's next?"Engineering a reliable workflow around AI is harder than integrating the AI itself. The models were the easy part; the loop was the product.