add local LLM with Ollama llama3.2:3b

OLLAMA_SETUP.md

@@ -0,0 +1,153 @@
+# Ollama 設置指南
+
+本指南說明如何在 Deep Agentic AI Tool 中設置和使用 Ollama，特別是 Llama 3.2 3B 模型。
+
+## 📋 前置需求
+
+- macOS 或 Linux 系統
+- 至少 16GB 記憶體（推薦）
+- Python >= 3.13
+
+## 🚀 安裝步驟
+
+### 1. 安裝 Ollama
+
+**macOS:**
+```bash
+brew install ollama
+```
+
+或從官網下載：https://ollama.com
+
+**Linux:**
+```bash
+curl -fsSL https://ollama.com/install.sh | sh
+```
+
+### 2. 下載 Llama 3.2 模型
+
+```bash
+ollama pull llama3.2:3b
+```
+
+這會下載約 2GB 的模型文件。
+
+### 3. 啟動 Ollama 服務
+
+Ollama 通常會自動啟動，如果需要手動啟動：
+
+```bash
+ollama serve
+```
+
+服務預設運行在 `http://localhost:11434`
+
+### 4. 驗證安裝
+
+測試模型是否可用：
+
+```bash
+ollama run llama3.2:3b "Hello, how are you?"
+```
+
+## ⚙️ 配置專案
+
+### 1. 更新環境變數
+
+在專案根目錄的 `.env` 文件中添加：
+
+```env
+# 啟用 Ollama
+USE_OLLAMA=true
+OLLAMA_BASE_URL=http://localhost:11434
+OLLAMA_MODEL=llama3.2:3b
+```
+
+### 2. 可選配置
+
+如果需要使用其他 Ollama 模型，可以修改：
+
+```env
+OLLAMA_MODEL=qwen2.5:7b        # 使用 Qwen2.5
+OLLAMA_MODEL=llama3.1:8b        # 使用 Llama 3.1
+OLLAMA_MODEL=deepseek-r1:7b     # 使用 DeepSeek-R1
+OLLAMA_MODEL=mistral:7b         # 使用 Mistral
+```
+
+## 🎯 使用方式
+
+系統會按照以下優先順序自動選擇 LLM：
+
+1. **Groq API**（如果配置了 `GROQ_API_KEY`）
+2. **Ollama**（如果 `USE_OLLAMA=true` 且服務可用）
+3. **MLX 模型**（備援選項）
+
+當 Groq API 額度用完時，系統會自動切換到 Ollama（如果啟用），否則使用 MLX 模型。
+
+## 🔍 檢查當前使用的模型
+
+啟動應用後，查看控制台輸出：
+
+- `✅ 使用 Groq API (優先)` - 使用 Groq API
+- `✅ 使用 Ollama 模型 (llama3.2:3b)` - 使用 Ollama
+- `ℹ️ 使用本地 MLX 模型` - 使用 MLX 模型
+
+## 🐛 故障排除
+
+### Ollama 服務無法連接
+
+**問題：** `⚠️ Ollama 初始化失敗: Connection refused`
+
+**解決方案：**
+1. 確認 Ollama 服務正在運行：`ollama serve`
+2. 檢查端口是否被占用：`lsof -i :11434`
+3. 確認 `OLLAMA_BASE_URL` 配置正確
+
+### 模型找不到
+
+**問題：** `⚠️ Ollama 初始化失敗: model not found`
+
+**解決方案：**
+```bash
+# 下載模型
+ollama pull llama3.2:3b
+
+# 列出已安裝的模型
+ollama list
+```
+
+### 記憶體不足
+
+**問題：** 系統運行緩慢或崩潰
+
+**解決方案：**
+- Llama 3.2:3B 需要約 2GB RAM
+- 確保系統有足夠的可用記憶體（推薦至少 8GB）
+- 這個模型已經很輕量，適合 16GB 記憶體的系統
+
+## 📊 模型比較
+
+| 模型 | 大小 | 記憶體需求 | 特點 |
+|------|------|-----------|------|
+| llama3.2:3b | ~2GB | ~4GB | 輕量高效，適合 16GB 記憶體系統，Meta 開源 |
+| deepseek-r1:7b | ~4.7GB | ~8GB | 優秀的推理能力，適合數學、編程 |
+| qwen2.5:7b | ~4.5GB | ~8GB | 通用能力強，中英文支援好 |
+| llama3.1:8b | ~4.6GB | ~8GB | Meta 開源，性能穩定 |
+| mistral:7b | ~4.1GB | ~7GB | 速度快，效率高 |
+
+## 💡 性能優化建議
+
+1. **優先使用 Groq API**：如果可用，Groq API 速度最快
+2. **Ollama 作為備援**：當 Groq 不可用時，Ollama 提供良好的本地推理
+3. **MLX 作為最後備援**：在 Apple Silicon 上，MLX 模型有硬體優化
+
+## 📚 相關資源
+
+- [Ollama 官方文檔](https://ollama.com/docs)
+- [Llama 3.2 模型資訊](https://ollama.com/library/llama3.2)
+- [LangChain Ollama 整合](https://python.langchain.com/docs/integrations/llms/ollama)
+
+---
+
+**注意**：首次使用時，Ollama 會下載模型文件，這可能需要一些時間，請耐心等待。
+

README.md

@@ -69,6 +69,11 @@ A comprehensive deep research agent system with RAG (Retrieval-Augmented Generat
    # Optional: Groq API (for faster inference)
    GROQ_API_KEY=your_groq_api_key_here
    
+   # Optional: Ollama (for local inference with Llama 3.2 or other models)
+   USE_OLLAMA=true
+   OLLAMA_BASE_URL=http://localhost:11434
+   OLLAMA_MODEL=llama3.2:3b
+   
    # Optional: Tavily API (for web search)
    TAVILY_API_KEY=your_tavily_api_key_here
    
@@ -204,18 +209,38 @@ The system uses a multi-agent workflow orchestrated by LangGraph:
 
 ### LLM Configuration
 
-The system supports multiple LLM backends with automatic fallback:
+The system supports multiple LLM backends with automatic fallback (priority order):
 
-1. **Primary**: Groq API (fast, requires API key)
+1. **Primary**: Groq API (fastest, requires API key)
    - Model: `llama-3.3-70b-versatile`
    - Automatically used if `GROQ_API_KEY` is set
 
-2. **Fallback**: Local MLX Model (privacy-preserving, no API key needed)
-   - Model: `mlx-community/Qwen2.5-Coder-7B-Instruct-4bit`
+2. **Secondary**: Ollama (local inference, excellent reasoning capabilities)
+   - Default Model: `llama3.2:3b` (Llama 3.2 3B)
+   - Requires Ollama installed and model downloaded
+   - Enable with `USE_OLLAMA=true` in `.env`
+   - Lightweight and efficient, suitable for 16GB memory systems
    - Automatically used when Groq API is unavailable or quota exhausted
 
+3. **Fallback**: Local MLX Model (privacy-preserving, no API key needed)
+   - Model: `mlx-community/Qwen2.5-Coder-7B-Instruct-4bit`
+   - Automatically used when both Groq API and Ollama are unavailable
+
 The system automatically switches between backends based on availability.
 
+**Setting up Ollama:**
+```bash
+# Install Ollama (if not already installed)
+# macOS: brew install ollama
+# Or download from https://ollama.com
+
+# Download Llama 3.2 model
+ollama pull llama3.2:3b
+
+# Start Ollama service (usually runs automatically)
+ollama serve
+```
+
 ## ⚙️ Configuration
 
 Key configuration options in `deep_agent_rag/config.py`:
@@ -283,6 +308,7 @@ Key dependencies (see `pyproject.toml` for complete list):
 - **LangChain**: Agent framework and tool integration
 - **LangGraph**: Agent orchestration and workflow management
 - **MLX/MLX-LM**: Local model inference (Apple Silicon optimized)
+- **LangChain Ollama**: Ollama integration for local models
 - **Gradio**: Web interface
 - **ChromaDB**: Vector database for RAG
 - **Tavily**: Web search API
@@ -298,9 +324,16 @@ Key dependencies (see `pyproject.toml` for complete list):
 
 ### Groq API Issues
 
-- **Quota exhausted**: The system automatically falls back to local MLX model
+- **Quota exhausted**: The system automatically falls back to Ollama (if enabled) or local MLX model
 - **API errors**: Check your `GROQ_API_KEY` in `.env` file
 
+### Ollama Issues
+
+- **Ollama not starting**: Ensure Ollama service is running (`ollama serve`)
+- **Model not found**: Download the model first (`ollama pull llama3.2:3b`)
+- **Connection errors**: Check `OLLAMA_BASE_URL` in `.env` (default: `http://localhost:11434`)
+- **Memory issues**: Llama 3.2:3B requires ~2GB RAM, suitable for systems with 16GB memory
+
 ### RAG System Issues
 
 - **PDF not found**: Ensure the PDF file exists at the path specified in `config.py`

deep_agent_rag/config.py

@@ -49,6 +49,13 @@ GROQ_MAX_TOKENS = 2048
 GROQ_TEMPERATURE = 0.7
 USE_GROQ_FIRST = True  # 是否优先使用 Groq API
 
+# Ollama 配置
+OLLAMA_BASE_URL = os.getenv("OLLAMA_BASE_URL", "http://localhost:11434")
+OLLAMA_MODEL = os.getenv("OLLAMA_MODEL", "llama3.2:3b")  # Llama 3.2 3B
+OLLAMA_MAX_TOKENS = 2048
+OLLAMA_TEMPERATURE = 0.7
+USE_OLLAMA = os.getenv("USE_OLLAMA", "false").lower() == "true"  # 是否啟用 Ollama
+
 # Email 配置 - 使用 Gmail API
 EMAIL_SENDER = "matthuang46@gmail.com"
 # Gmail API 配置

deep_agent_rag/utils/llm_utils.py

@@ -1,11 +1,12 @@
 """
 LLM 工具函數
 提供 LLM 實例的創建和管理
-優先使用 Groq API，額度用完後自動切換到本地 MLX 模型
+優先順序：Groq API > Ollama > MLX 模型
 """
 import warnings
 from typing import Optional
 from langchain_groq import ChatGroq
+from langchain_ollama import ChatOllama
 from ..models import MLXChatModel, load_mlx_model
 from ..config import (
     MLX_MAX_TOKENS, 
@@ -14,7 +15,12 @@ from ..config import (
     GROQ_MODEL,
     GROQ_MAX_TOKENS,
     GROQ_TEMPERATURE,
-    USE_GROQ_FIRST
+    USE_GROQ_FIRST,
+    OLLAMA_BASE_URL,
+    OLLAMA_MODEL,
+    OLLAMA_MAX_TOKENS,
+    OLLAMA_TEMPERATURE,
+    USE_OLLAMA,
 )
 
 # 全局變量：跟踪當前使用的 LLM 類型
@@ -29,30 +35,17 @@ def get_llm_type() -> str:
 
 def is_using_local_llm() -> bool:
     """檢查是否正在使用本地 LLM"""
-    return _current_llm_type == "mlx" or _groq_quota_exceeded
+    return _current_llm_type in ["mlx", "ollama"] or _groq_quota_exceeded
 
 
 def get_llm():
     """
     獲取 LLM 實例
-    優先使用 Groq API，額度用完後自動切換到本地 MLX 模型
+    優先順序：Groq API > Ollama > MLX 模型
     """
     global _current_llm_type, _groq_quota_exceeded
     
-    # 如果已經知道 Groq 額度用完，直接使用本地模型
-    if _groq_quota_exceeded:
-        if _current_llm_type != "mlx":
-            print("⚠️ 警告：Groq API 額度已用完，已切換到本地 MLX 模型 (Qwen2.5)")
-        _current_llm_type = "mlx"
-        model, tokenizer = load_mlx_model()
-        return MLXChatModel(
-            model=model,
-            tokenizer=tokenizer,
-            max_tokens=MLX_MAX_TOKENS,
-            temperature=MLX_TEMPERATURE
-        )
-    
-    # 嘗試使用 Groq API
+    # 優先順序 1: Groq API
     if USE_GROQ_FIRST and GROQ_API_KEY:
         try:
             groq_llm = ChatGroq(
@@ -61,48 +54,61 @@ def get_llm():
                 max_tokens=GROQ_MAX_TOKENS,
                 temperature=GROQ_TEMPERATURE
             )
-            # 測試連接（通過一個簡單的調用來驗證）
-            # 注意：這裡不實際調用，只是創建實例
             _current_llm_type = "groq"
             print("✅ 使用 Groq API (優先)")
             return groq_llm
         except Exception as e:
-            # 如果創建失敗，可能是 API key 無效
+            # 如果創建失敗，繼續嘗試其他選項
             print(f"⚠️ Groq API 初始化失敗: {e}")
-            _groq_quota_exceeded = True
-            _current_llm_type = "mlx"
-            print("⚠️ 警告：已切換到本地 MLX 模型 (Qwen2.5)")
-            model, tokenizer = load_mlx_model()
-            return MLXChatModel(
-                model=model,
-                tokenizer=tokenizer,
-                max_tokens=MLX_MAX_TOKENS,
-                temperature=MLX_TEMPERATURE
+            # 不立即設置 _groq_quota_exceeded，先嘗試 Ollama
+    
+    # 優先順序 2: Ollama (Llama 3.2 或其他模型)
+    if USE_OLLAMA:
+        try:
+            ollama_llm = ChatOllama(
+                base_url=OLLAMA_BASE_URL,
+                model=OLLAMA_MODEL,
+                num_predict=OLLAMA_MAX_TOKENS,
+                temperature=OLLAMA_TEMPERATURE,
             )
-    else:
-        # 如果沒有配置 Groq 或選擇不使用，直接使用本地模型
-        if not GROQ_API_KEY:
-            print("ℹ️ 未配置 GROQ_API_KEY，使用本地 MLX 模型")
-        _current_llm_type = "mlx"
-        model, tokenizer = load_mlx_model()
-        return MLXChatModel(
-            model=model,
-            tokenizer=tokenizer,
-            max_tokens=MLX_MAX_TOKENS,
-            temperature=MLX_TEMPERATURE
-        )
+            _current_llm_type = "ollama"
+            print(f"✅ 使用 Ollama 模型 ({OLLAMA_MODEL})")
+            return ollama_llm
+        except Exception as e:
+            print(f"⚠️ Ollama 初始化失敗: {e}")
+            print("   請確保 Ollama 服務正在運行: ollama serve")
+            print("   或檢查模型是否已下載: ollama pull " + OLLAMA_MODEL)
+    
+    # 優先順序 3: MLX 模型（備援）
+    # 如果 Groq 額度用完，記錄狀態
+    if _groq_quota_exceeded and _current_llm_type != "mlx":
+        print("⚠️ 警告：Groq API 額度已用完，已切換到本地 MLX 模型 (Qwen2.5)")
+    elif _current_llm_type != "mlx":
+        if not GROQ_API_KEY and not USE_OLLAMA:
+            print("ℹ️ 未配置 Groq API 或 Ollama，使用本地 MLX 模型")
+        elif not USE_OLLAMA:
+            print("ℹ️ Ollama 未啟用，使用本地 MLX 模型作為備援")
+    
+    _current_llm_type = "mlx"
+    model, tokenizer = load_mlx_model()
+    return MLXChatModel(
+        model=model,
+        tokenizer=tokenizer,
+        max_tokens=MLX_MAX_TOKENS,
+        temperature=MLX_TEMPERATURE
+    )
 
 
 def handle_groq_error(error: Exception) -> Optional[MLXChatModel]:
     """
     處理 Groq API 錯誤
-    如果是額度用完錯誤，切換到本地模型
+    如果是額度用完錯誤，先嘗試切換到 Ollama，否則切換到 MLX 模型
     
     Args:
         error: 捕獲的異常
     
     Returns:
-        如果切換到本地模型，返回 MLXChatModel；否則返回 None
+        如果切換到本地模型，返回 ChatOllama 或 MLXChatModel；否則返回 None
     """
     global _current_llm_type, _groq_quota_exceeded
     
@@ -121,10 +127,27 @@ def handle_groq_error(error: Exception) -> Optional[MLXChatModel]:
     if any(indicator in error_str for indicator in quota_indicators):
         if not _groq_quota_exceeded:
             _groq_quota_exceeded = True
-            warning_msg = "⚠️ 警告：Groq API 額度已用完，正在切換到本地 MLX 模型 (Qwen2.5)"
+            warning_msg = "⚠️ 警告：Groq API 額度已用完"
             print(warning_msg)
             warnings.warn(warning_msg, UserWarning)
         
+        # 先嘗試使用 Ollama
+        if USE_OLLAMA:
+            try:
+                ollama_llm = ChatOllama(
+                    base_url=OLLAMA_BASE_URL,
+                    model=OLLAMA_MODEL,
+                    num_predict=OLLAMA_MAX_TOKENS,
+                    temperature=OLLAMA_TEMPERATURE,
+                )
+                _current_llm_type = "ollama"
+                print(f"✅ 已切換到 Ollama 模型 ({OLLAMA_MODEL})")
+                return ollama_llm
+            except Exception as e:
+                print(f"⚠️ Ollama 切換失敗: {e}")
+                print("   回退到 MLX 模型")
+        
+        # 回退到 MLX 模型
         _current_llm_type = "mlx"
         model, tokenizer = load_mlx_model()
         return MLXChatModel(

pyproject.toml

@@ -17,6 +17,7 @@ dependencies = [
     "yfinance>=0.2.66",
     "langgraph>=1.0.4",
     "langchain-groq>=1.1.0",
+    "langchain-ollama>=0.1.0",
     "grandalf>=0.8",
     "langserve[all]>=0.3.3",
     "fastapi>=0.124.2",

uv.lock

@@ -636,6 +636,7 @@ dependencies = [
     { name = "langchain-community" },
     { name = "langchain-google-genai" },
     { name = "langchain-groq" },
+    { name = "langchain-ollama" },
     { name = "langchain-tavily" },
     { name = "langgraph" },
     { name = "langserve", extra = ["all"] },
@@ -671,6 +672,7 @@ requires-dist = [
     { name = "langchain-community", specifier = ">=0.4.1" },
     { name = "langchain-google-genai", specifier = ">=4.0.0" },
     { name = "langchain-groq", specifier = ">=1.1.0" },
+    { name = "langchain-ollama", specifier = ">=0.1.0" },
     { name = "langchain-tavily", specifier = ">=0.2.13" },
     { name = "langgraph", specifier = ">=1.0.4" },
     { name = "langserve", extras = ["all"], specifier = ">=0.3.3" },
@@ -1591,6 +1593,19 @@ wheels = [
     { url = "https://files.pythonhosted.org/packages/af/4a/3d6227a16fe9f79968414b50e50869519378b20653805e2e8fab283908e6/langchain_groq-1.1.1-py3-none-any.whl", hash = "sha256:1c6d5146f60205dcde09d7e47bb5291c295d3f0c7bcd2417e4d3a73a04bd1050", size = 19039, upload-time = "2025-12-12T22:00:45.86Z" },
 ]
 
+[[package]]
+name = "langchain-ollama"
+version = "1.0.1"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+    { name = "langchain-core" },
+    { name = "ollama" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/73/51/72cd04d74278f3575f921084f34280e2f837211dc008c9671c268c578afe/langchain_ollama-1.0.1.tar.gz", hash = "sha256:e37880c2f41cdb0895e863b1cfd0c2c840a117868b3f32e44fef42569e367443", size = 153850, upload-time = "2025-12-12T21:48:28.68Z" }
+wheels = [
+    { url = "https://files.pythonhosted.org/packages/e3/46/f2907da16dc5a5a6c679f83b7de21176178afad8d2ca635a581429580ef6/langchain_ollama-1.0.1-py3-none-any.whl", hash = "sha256:37eb939a4718a0255fe31e19fbb0def044746c717b01b97d397606ebc3e9b440", size = 29207, upload-time = "2025-12-12T21:48:27.832Z" },
+]
+
 [[package]]
 name = "langchain-tavily"
 version = "0.2.16"
@@ -2243,6 +2258,19 @@ wheels = [
     { url = "https://files.pythonhosted.org/packages/be/9c/92789c596b8df838baa98fa71844d84283302f7604ed565dafe5a6b5041a/oauthlib-3.3.1-py3-none-any.whl", hash = "sha256:88119c938d2b8fb88561af5f6ee0eec8cc8d552b7bb1f712743136eb7523b7a1", size = 160065, upload-time = "2025-06-19T22:48:06.508Z" },
 ]
 
+[[package]]
+name = "ollama"
+version = "0.6.1"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+    { name = "httpx" },
+    { name = "pydantic" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/9d/5a/652dac4b7affc2b37b95386f8ae78f22808af09d720689e3d7a86b6ed98e/ollama-0.6.1.tar.gz", hash = "sha256:478c67546836430034b415ed64fa890fd3d1ff91781a9d548b3325274e69d7c6", size = 51620, upload-time = "2025-11-13T23:02:17.416Z" }
+wheels = [
+    { url = "https://files.pythonhosted.org/packages/47/4f/4a617ee93d8208d2bcf26b2d8b9402ceaed03e3853c754940e2290fed063/ollama-0.6.1-py3-none-any.whl", hash = "sha256:fc4c984b345735c5486faeee67d8a265214a31cbb828167782dc642ce0a2bf8c", size = 14354, upload-time = "2025-11-13T23:02:16.292Z" },
+]
+
 [[package]]
 name = "onnxruntime"
 version = "1.23.2"