How I Built a Local AI Task Agent with Ollama + Apple Shortcuts + Voice Control (Hey Mac! 🚀)

Jayasurya Mailsamy

— 2 min read
siri vs ollama

Sure! Here's a fun, friendly, and easy-to-understand blog post just for you 😄✍️.

🧠 How I Built a Local AI Task Agent with Ollama + Apple Shortcuts + Voice Control (Hey Mac! 🚀)

Hey there, fellow tech explorer! 👋 Ever wished your Mac could just listen and do what you ask? Like:

"Hey Mac, generate an AppleScript for me!"

Well guess what? You can make that happen — without any cloud magic. It's all local, private, and totally in your control 😎

Let me walk you through how I created a simple AI agent using:

  • 🦙 Ollama (running AI models locally)
  • 🍏 Apple Shortcuts (automation awesomeness)
  • 🗣️ Voice Control (so I can just speak commands)
  • 🎩 Custom phrase: “Hey Mac!”

🎯 What We’re Gonna Build

"Hey Mac, write an AppleScript to mute the volume."

This command will:

  1. Use your voice to trigger a shortcut
  2. Send the task to Ollama
  3. Return just the AppleScript
  4. Run it automatically!

Boom. Magic. 🔮✨

🛠️ Step-by-Step Guide

1️⃣ Install Ollama & Run a Model Locally

Go to ollama.com and install it.

Then, open Terminal and run:

ollama run gemma3:latest

(Or any model you like — Gemma 3,Mistral, Llama 2, etc.)

It’ll start a local API at:
📍 http://localhost:11434/api/generate

2️⃣ Create the Shortcut in Apple Shortcuts

Here’s what the shortcut does (you saw this in the screenshot):

  • 🎙️ Dictate Text – You speak the task
  • 📝 Sends: "Return only valid AppleScript code. Do not explain. The task is: <your task>"
  • 🌐 GET request to: http://localhost:11434/api/generate
  • 🪄 Extracts the pure AppleScript (removes backticks and tags)
  • 🖥️ Runs it using osascript
  • ✅ Done!

3️⃣ Connect Voice Control with Custom Command

Here’s the cherry on top 🍒

  1. Go to System Settings > Accessibility > Voice Control
  2. Turn it ON
  3. Click Commands…
  4. Click ➕ to create a new command:
    • When I say: Hey Mac
    • While using: Any app
    • Perform: Run Shortcut → Select your shortcut (e.g., "AutoLama")

Here is the screenshot of the shortcut i made

{
  "model": "gemma3:latest",

  "messages": [
    {
      "role": "system",
      "content": "you are a apple script writer you can write for mac terminal with osascript -e and code only no more extra text"
    },
    {
      "role": "user", 
      "content": "Dictated Text"
    }
  ],
  "stream": false }

POST request data for ollama api/chat

Replace gemma3:latest into your model
Goto Shortcut

Now say:

"Hey Mac"
It triggers your shortcut 🎤🤖

😍 Example in Action

Let’s say I want to create a new folder on my Desktop. I say:

“Hey Mac, create a folder named LlamaMagic on desktop.”

Your shortcut will:

  • Send that to the local LLM
  • Run it instantly!

Receive:

tell application "Finder"
    make new folder at desktop with properties {name:"LlamaMagic"}
end tell

You just spoke an idea and your Mac obeyed like a friendly genie 🧞‍♂️

🧠 Bonus Tips

  • You can change the prompt to ask for shell scripts, Python, or whatever!
  • Add more automation, like asking for confirmation before running
  • Use clipboard instead of voice if you're shy 😅

🥳 Final Thoughts

This was super fun to build, and it makes your Mac feel like a personal AI assistant. Everything is local — no internet, no tracking, no middleman!

All you need is:

  • Ollama 🦙
  • A Shortcut ⚡
  • Voice Control 🎙️
  • A little creativity 💡

Now go impress your friends and make your Mac say "Your wish is my command." 😄

If you liked this guide, share it with your geeky friends 🧙‍♀️ and tell them:

“You don’t need Siri. You need Shortcuts + Ollama!”

Need help setting it up? Drop your questions below! 💬👇

Read more