diff --git a/doc/docs/en/user-guide/agent-development/model-configuration.md b/doc/docs/en/user-guide/agent-development/model-configuration.md index 33905b5988..d5a3d75da7 100644 --- a/doc/docs/en/user-guide/agent-development/model-configuration.md +++ b/doc/docs/en/user-guide/agent-development/model-configuration.md @@ -1,23 +1,17 @@ # Model Configuration -In the Model Management module, you can configure your app’s basic information and connect every model the platform needs, including large language models, embedding models, and vision-language models. Nexent supports multiple providers so you can pick the best option for each scenario. +The Model Configuration module lets you add and configure AI models — large language models, embedding models, rerank models, multimodal models, and voice models. Nexent supports multiple providers so you can pick the best option for each scenario. ## 🤖 Model Configuration -### 🔄 Sync ModelEngine Models - -Nexent supports seamless integration with the ModelEngine platform. - -👉 Click **Edit ModelEngine Configuration** in the upper right corner of the page, enter your API key, and you can retrieve all models deployed on ModelEngine. - ### 🛠️ Add Custom Models #### Add a Single Model 1. **Add a custom model** - - Click **Add Custom Model** to open the dialog. + - Click **Add Model** to open the dialog. 2. **Select model type** - - Choose Large Language Model, Embedding Model, or Vision Language Model. + - Choose Large Language Model, Embedding Model, Image Understanding Model, Image Generation Model, Video Understanding Model, Rerank Model, Speech-to-Text Model or Text-to-Speech Model. 3. **Configure model parameters** - **Model Name (required):** The name you send in API requests. - **Display Name:** Optional label shown in the UI (defaults to the model name). @@ -27,7 +21,7 @@ Nexent supports seamless integration with the ModelEngine platform. > ⚠️ **Notes** > 1. Model names usually follow `series/model`. Example: `Qwen/Qwen3-8B`. > 2. API endpoints come from the provider docs. For SiliconFlow, examples include `https://api.siliconflow.cn/v1` (LLM, VLM) and `https://api.siliconflow.cn/v1/embeddings` (embedding). -> 3. Generate API keys from the provider’s key management console. +> 3. Generate API keys from the provider's key management console. 4. **Connectivity verification** - Click **Verify** to send a test request and confirm connectivity. @@ -44,7 +38,7 @@ Use batch import to speed up onboarding: 1. Enable the **Batch Add Models** toggle in the dialog. 2. Select a **model provider**. -3. Choose the **model type** (LLM/Embedding/Vision). +3. Choose the **model type** (LLM/Embedding/VLM/Rerank/STT/TTS). 4. Enter the **API Key** (required). 5. Click **Fetch Models** to retrieve the provider list. 6. Toggle on the models you need (disabled by default). @@ -59,7 +53,7 @@ Use batch import to speed up onboarding: Modify or delete models anytime: 1. Click **Edit Custom Models**. -2. Select the model type (LLM/Embedding/Vision). +2. Select the model type (LLM/Embedding/VLM/Rerank/STT/TTS). 3. Choose between batch editing or single-model editing. 4. For batch edits, toggle models on/off or click **Edit Config** in the upper-right to change settings in bulk. 5. For single models, click the trash icon 🗑️ to delete, or click the model name to open the edit dialog. @@ -88,6 +82,12 @@ After adding models, assign the platform-level defaults. These models handle sys - Used for core platform features (title generation, real-time file access, basic text processing). - Choose any added large language model from the dropdown. +#### Large Language Model + +The large language model serves as the system's core reasoning engine, responsible for processing users' natural language requests, generating responses, executing code, analyzing data, and other complex tasks. Choosing an appropriate large language model can significantly improve the agent's conversational quality and task-handling capabilities. + +- Click the Large Language Model dropdown and select one from the added large language models. + #### Embedding Model Embedding models are primarily used for vectorization processing of text, images, and other data in knowledge bases, forming the foundation for efficient retrieval and semantic understanding. Configuring an appropriate embedding model can significantly improve knowledge base search accuracy and multimodal data processing capabilities. @@ -101,15 +101,33 @@ Choose appropriate document chunk size and chunks per request based on model cap -#### Vision-Language Model +#### Rerank Model -- Required for multimodal chat scenarios (for example, when users upload images). -- Pick one of the added vision-language models. +The rerank model performs semantic matching and scoring on initially filtered documents to ensure the most relevant answers are ranked first, improving retrieval accuracy and efficiency. Configuring an appropriate rerank model can significantly improve knowledge base retrieval effectiveness. -
- - - +- Click the Rerank Model dropdown to select one from the added rerank models. + +#### Multimodal Models + +Multimodal models combine visual and language capabilities to handle complex scenarios containing text, images, and other types of information. + +- **Image Understanding Model**: Can analyze and understand image content, extract key information, and answer questions related to images. Click the Image Understanding Model dropdown to select one from the added models. +- **Image Generation Model**: Can generate images based on text descriptions, supporting creative design, content creation, and other scenarios. Click the Image Generation Model dropdown to select one from the added models. +- **Video Understanding Model**: Can analyze and understand video content, extract key information, generate summaries, or answer questions related to videos. Click the Video Understanding Model dropdown to select one from the added models. + +#### Voice Models + +Voice models enable bidirectional conversion between speech and text, supporting voice interaction scenarios. + +- **Text-to-Speech Model**: Converts text content into natural, fluent speech output in real-time, enabling the system to interact with users in a near-human voice. With low latency and high-fidelity speech generation capabilities, it ensures a smooth and natural auditory experience during conversations. Click the Text-to-Speech Model dropdown to select one from the added models. +- **Speech-to-Text Model**: Converts user voice input into text in real-time, enabling accurate understanding and parsing of voice commands and natural language. With high-precision speech transcription and noise robustness, it ensures stable recognition of user intent even in complex environments. Click the Speech-to-Text Model dropdown to select one from the added models. + +
+ + + + +
### ✅ Check Model Connectivity @@ -129,7 +147,7 @@ Troubleshooting tips: - Confirm network stability. - Ensure the API key is valid and not expired. -- Check the provider’s service status. +- Check the provider's service status. - Review firewall and security policies. ### 🤖 Supported Providers @@ -148,12 +166,12 @@ Nexent supports any **OpenAI-compatible** provider, including: Getting started: -1. Sign up at the provider’s portal. +1. Sign up at the provider's portal. 2. Create and copy an API key. 3. Locate the API endpoint (usually ending with `/v1`). 4. Click **Add Custom Model** in Nexent and fill in the required fields. -#### Multimodal Vision Models +#### Multimodal Models Use the same API key and URL as LLMs but specify a multimodal model name, for example **Qwen/Qwen2.5-VL-32B-Instruct** on SiliconFlow. @@ -161,35 +179,50 @@ Use the same API key and URL as LLMs but specify a multimodal model name, for ex Use the same API key as LLMs but typically a different endpoint (often `/v1/embeddings`), for example **BAAI/bge-m3** from SiliconFlow. +#### Rerank Models + +Use the same API key as LLMs but typically a different endpoint (often `/v1/rerank`). + #### Speech Models -Currently only **VolcEngine Voice** is supported and must be configured via `.env`: +Currently supports VolcEngine Voice and Aliyun Bailian voice models. VolcEngine requires `appid` and `token`, while Aliyun Bailian uses the same API key as the large language model. + +**VolcEngine** +- **Website**: [volcengine.com/product/voice-tech](https://www.volcengine.com/product/voice-tech) +- **Free tier**: Available for individual use +- **Highlights**: High-quality Chinese/English TTS +- Recommended models: **Doubao Text-to-Speech Model 2.0** and **Large Model Streaming Speech Recognition** +- **Getting started**: -- **Website:** [volcengine.com/product/voice-tech](https://www.volcengine.com/product/voice-tech) -- **Free tier:** Available for individual use -- **Highlights:** High-quality Chinese/English TTS + 1. Register a VolcEngine account. + 2. Enable the Voice Technology service. + 3. Create an app and generate `appid` and `token`. + 4. Configure the TTS/STT settings in the Add Model page. -Steps: +**Aliyun Bailian** +- **Website**: [aliyun.com/benefit/scene/voice](https://www.aliyun.com/benefit/scene/voice) +- Recommended models: **Qwen3-TTS-Instruct-Flash-Realtime / Qwen3-TTS-Flash-Realtime** and **Qwen3-ASR-Flash-Realtime** +- **Getting started**: -1. Register a VolcEngine account. -2. Enable the Voice Technology service. -3. Create an app and generate an API key. -4. Configure the TTS/STT settings in your environment. + 1. Register an Aliyun account. + 2. Enable the Qwen real-time voice service. + 3. Create an app and generate an API Key. + 4. Configure the TTS/STT settings in the Add Model page. ## 💡 Need Help If you run into provider issues: -1. Review the provider’s documentation. +1. Review the provider's documentation. 2. Check API key permissions and quotas. -3. Test with the provider’s official samples. +3. Test with the provider's official samples. 4. Ask the community in our [Discord server](https://discord.gg/tb5H3S3wyv). ## 🚀 Next Steps -After closing the Model Management flow, continue with: +After closing the Model Configuration flow, continue with: -1. **[Knowledge Base](./knowledge-configuration.md)** – Create and manage knowledge bases. -2. **[Agent Development](../agent-development.md)** – Build and configure agents. +1. **[Knowledge Base](./knowledge-configuration)** – Create and manage knowledge bases. +2. **[Agent Configuration](./agent-configuration)** – Build and configure agents. -Need help? Check the **[FAQ](../../quick-start/faq.md)** or open a thread in [GitHub Discussions](https://github.com/ModelEngine-Group/nexent/discussions). +Need help? Check the **[FAQ](../../quick-start/faq)** or open a thread in [GitHub Discussions](https://github.com/ModelEngine-Group/nexent/discussions). diff --git a/doc/docs/en/user-guide/assets/model-management/edit-model-1.png b/doc/docs/en/user-guide/assets/model-management/edit-model-1.png index 45ddaaf084..8286cfb485 100644 Binary files a/doc/docs/en/user-guide/assets/model-management/edit-model-1.png and b/doc/docs/en/user-guide/assets/model-management/edit-model-1.png differ diff --git a/doc/docs/en/user-guide/assets/model-management/select-model-3.png b/doc/docs/en/user-guide/assets/model-management/select-model-3.png index 34d83df560..c4b0033ca1 100644 Binary files a/doc/docs/en/user-guide/assets/model-management/select-model-3.png and b/doc/docs/en/user-guide/assets/model-management/select-model-3.png differ diff --git a/doc/docs/en/user-guide/assets/model-management/select-model-4.png b/doc/docs/en/user-guide/assets/model-management/select-model-4.png new file mode 100644 index 0000000000..a09010ad10 Binary files /dev/null and b/doc/docs/en/user-guide/assets/model-management/select-model-4.png differ diff --git a/doc/docs/en/user-guide/assets/model-management/select-model-5.png b/doc/docs/en/user-guide/assets/model-management/select-model-5.png new file mode 100644 index 0000000000..025a2b456c Binary files /dev/null and b/doc/docs/en/user-guide/assets/model-management/select-model-5.png differ diff --git a/doc/docs/zh/user-guide/agent-development/model-configuration.md b/doc/docs/zh/user-guide/agent-development/model-configuration.md index 23c0ed2c72..6630f02ff3 100644 --- a/doc/docs/zh/user-guide/agent-development/model-configuration.md +++ b/doc/docs/zh/user-guide/agent-development/model-configuration.md @@ -1,23 +1,17 @@ # 模型配置 -在模型管理模块中,您可以配置应用的基本信息,并接入各类AI模型,包括大语言模型、向量化模型和视觉语言模型。Nexent支持多种模型提供商,帮助您根据实际需求灵活选择最适合的模型。 +在模型配置模块中,您可以接入并配置各类 AI 模型,包括大语言模型、向量化模型、重排序模型、多模态模型和语音模型。Nexent 支持多种模型提供商,帮助您根据实际需求灵活选择最适合的模型。 ## 🤖 模型配置 -### 🔄 同步ModelEngine模型 - -Nexent支持与ModelEngine平台的无缝对接 - -👉 点击页面右上方 **\*\*\*\***ModelEngine配置**\***\*\*\*,输入您的 API ,即可获取您在 ModelEgnine 上部署的所有模型 - ### 🛠️ 添加自定义模型 #### 添加单个模型 1. **添加自定义模型** - - 点击"添加自定义模型"按钮,进入添加模型弹窗。 + - 点击"添加模型"按钮,进入添加模型弹窗。 2. **选择模型类型** - - 点击模型类型下拉框,选择要添加的模型类型(大语言模型/向量化模型/视觉语言模型/重排模型)。 + - 点击模型类型下拉框,选择要添加的模型类型(大语言模型/向量模型/图片理解模型/图片生成模型/视频理解模型/重排模型/语音识别模型/语音合成模型)。 3. **配置模型参数** - **模型名称(必填)**:输入请求体中的模型名称。 - **展示名称**:可为模型设置一个展示名称,默认与模型名称相同。 @@ -25,7 +19,6 @@ Nexent支持与ModelEngine平台的无缝对接 - **API Key**:输入您的API密钥。 > ⚠️ **注意事项**: -> > 1. 模型名称通过模型提供商获取,通常格式为`模型系列/模型名字`。以模型系列是`Qwen`,模型名字是`Qwen3-8B`为例,模型名称为`Qwen/Qwen3-8B`。 > 2. 模型URL通过模型提供商的API文档获取。以模型提供商是硅基流动为例,大语言模型URL为`https://api.siliconflow.cn/v1` ,向量模型URL为`https://api.siliconflow.cn/v1/embeddings` ,视觉语言模型URL为`https://api.siliconflow.cn/v1` 。 > 3. API Key通过模型提供商的API Key密钥管理页面创建并获取API Key。 @@ -48,7 +41,7 @@ Nexent支持与ModelEngine平台的无缝对接 2. **选择模型提供商** - 点击模型提供商下拉框,选择模型提供商。 3. **选择模型类型** - - 点击模型类型下拉框,选择要添加的模型类型(大语言模型/向量化模型/视觉语言模型/重排模型)。 + - 点击模型类型下拉框,选择要添加的模型类型(大语言模型/向量模型/图片理解模型/图片生成模型/视频理解模型/重排模型/语音识别模型/语音合成模型)。 4. **输入API Key(必填)** - 输入您的API密钥。 5. **获取模型** @@ -67,7 +60,7 @@ Nexent支持与ModelEngine平台的无缝对接 当您需要修改模型配置或删除不再使用的模型时,可以通过以下步骤进行操作: 1. 点击"修改自定义模型"按钮。 -2. 选择要修改或删除的模型类型(大语言模型/向量化模型/视觉语言模型)。 +2. 选择要修改或删除的模型类型(大语言模型/向量模型/图片理解模型/图片生成模型/视频理解模型/重排模型/语音识别模型/语音合成模型)。 3. 选择是批量修改模型,还是修改单个自定义模型。 4. 如果批量修改模型,可以通过启动或关闭模型开关来添加或删除模型。您也可以通过点击右上角的"修改配置"按钮,对选中的模型进行批量配置修改。 5. 如果是修改单个自定义模型,点击删除按钮 🗑️ 即可删除目标模型;想要修改相关配置,点击模型名称即可弹出修改弹窗进行修改。 @@ -103,6 +96,11 @@ Nexent支持与ModelEngine平台的无缝对接 - 点击基础模型下拉框,从已添加的大语言模型中选择一个作为系统基础模型。 +#### 大语言模型 +大语言模型是系统的核心推理引擎,负责处理用户的自然语言请求、生成回复、执行代码、分析数据等复杂任务。选择合适的大语言模型,可以显著提升智能体的对话质量和任务处理能力。 +- 点击大语言模型下拉框,从已添加的大语言模型中选择一个。 + + #### 向量模型 向量模型主要用于知识库的文本、图片等数据的向量化处理,是实现高效检索和语义理解的基础。配置合适的向量模型,可以显著提升知识库的搜索准确率和多模态数据的处理能力。 @@ -117,35 +115,36 @@ Nexent支持与ModelEngine平台的无缝对接
#### 重排模型 - 重排模型用于初筛后的文档进行语义匹配与评分,确保最相关的核心答案能够排在首位,以提升检索的准确性和效率。配置合适的重排模型,可以显著提升知识库的检索效果。 - 点击重排模型下拉框,从已添加的重排模型中选择一个。 + #### 多模态模型 多模态模型结合了视觉和语言能力,能够处理包含文本、图片等多种信息的复杂场景。例如,在对话页面上传图片文件时,系统会自动调用多模态模型进行内容解析和智能对话。 -- 点击视觉语言模型下拉框,从已添加的视觉语言模型中选择一个。 +- **图片理解模型**:能够分析和理解图片内容,提取关键信息、回答与图片相关的问题。点击图片理解模型下拉框,从已添加的图片理解模型中选择一个。 +- **图片生成模型**:能够根据文本描述生成对应的图像,支持创意设计、内容创作等多种场景。点击图片生成模型下拉框,从已添加的图片生成模型中选择一个。 +- **视频理解模型**:能够分析和理解视频内容,提取关键信息、生成摘要或回答与视频相关的问题。点击视频理解模型下拉框,从已添加的视频理解模型中选择一个。 -
- - - - -
+#### 语音模型 -#### 语音合成模型 +语音模型用于实现语音与文本之间的双向转换,支持语音交互场景。 -语音合成模型用于将文本内容即时转换为自然流畅的语音输出,使系统能够以接近真人的方式进行语音交互与反馈。通过低延迟、高拟真度的语音生成能力,确保用户在对话过程中获得连贯、自然的听觉体验。配置合适的实时语音合成模型,可以显著提升语音交互系统的表现力和用户体验。 +- **语音合成模型**:用于将文本内容即时转换为自然流畅的语音输出,使系统能够以接近真人的方式进行语音交互与反馈。通过低延迟、高拟真度的语音生成能力,确保用户在对话过程中获得连贯、自然的听觉体验。点击语音合成模型下拉框,从已添加的语音合成模型中选择一个。 +- **语音识别模型**:用于将用户输入的语音内容实时转换为文本,实现对语音指令和自然语言的准确理解与解析。通过高精度的语音转写与噪声鲁棒能力,确保在复杂环境下依然能够稳定识别用户意图。点击语音识别模型下拉框,从已添加的语音识别模型中选择一个。 -- 点击语音合成模型下拉框,从已添加的视觉语言模型中选择一个。 -#### 语音识别模型 +
+ + + + + +
-语音识别模型用于将用户输入的语音内容实时转换为文本,实现对语音指令和自然语言的准确理解与解析。通过高精度的语音转写与噪声鲁棒能力,确保在复杂环境下依然能够稳定识别用户意图。配置合适的语音识别模型,可以显著提升语音交互系统的理解能力和整体响应效率。 -- 点击语音识别模型下拉框,从已添加的视觉语言模型中选择一个。 ### ✅ 检查模型连通性 @@ -190,7 +189,7 @@ Nexent 支持任何 **遵循OpenAI API规范** 的大语言模型供应商,包 3. 在文档中查看API端点(即模型URL,一般以`/v1`为结尾); 4. 在Nexent模型配置页面点击添加自定义模型,填入必备信息,即可接入。 -#### 🎭 多模态视觉模型 +#### 🎭 多模态模型 使用与大语言模型相同的API Key和模型URL,但指定多模态模型名称,如硅基流动提供的**Qwen/Qwen2.5-VL-32B-Instruct**。 @@ -199,34 +198,32 @@ Nexent 支持任何 **遵循OpenAI API规范** 的大语言模型供应商,包 使用与大语言模型相同的API Key,但模型URL一般会有所差异,一般以`/v1/embeddings`为结尾,同时指定向量模型名称,如硅基流动提供的**BAAI/bge-m3**。 #### 🔃 重排模型 - 使用与大语言模型相同的API Key,但模型URL一般会有所差异,一般以`/v1/rerank`为结尾。 - #### 🎤 语音模型 目前支持阿里灵积和火山引擎语音模型,阿里灵积需配置与大语言模型相同的apikey,火山引擎模型需配置appid与token **火山引擎** - - **网站**: [volcengine.com/product/voice-tech](https://www.volcengine.com/product/voice-tech) - **免费额度**: 个人使用可用 - **特色**: 高质量中英文语音合成 - 推荐使用**豆包语音合成模型2.0和大模型流式语音识别模型** - **开始使用**: - 1. 注册火山引擎账户 - 2. 访问语音技术服务 - 3. 创建应用并获取appid和token - 4. 在添加模型页面中配置 TTS/STT 设置 -**阿里灵积** + 1. 注册火山引擎账户 + 2. 访问语音技术服务 + 3. 创建应用并获取appid和token + 4. 在添加模型页面中配置 TTS/STT 设置 +**阿里灵积** - **网站**: [aliyun.com/benefit/scene/voice](https://www.aliyun.com/benefit/scene/voice) - 推荐使用**千问3-TTS-Instruct-Flash-Realtime/千问3-TTS-Flash-Realtime和千问3-ASR-Flash-Realtime** - **开始使用**: - 1. 注册阿里云账户 - 2. 访问阿里千问实时语音技术服务 - 3. 创建应用并获取 API Key - 4. 在添加模型页面中配置 TTS/STT 设置 + + 1. 注册阿里云账户 + 2. 访问阿里千问实时语音技术服务 + 3. 创建应用并获取 API Key + 4. 在添加模型页面中配置 TTS/STT 设置 ## 💡 需要帮助 @@ -239,9 +236,9 @@ Nexent 支持任何 **遵循OpenAI API规范** 的大语言模型供应商,包 ## 🚀 下一步 -完成模型管理配置后,建议您继续配置: +完成模型配置后,建议您继续配置: -1. **[知识库配置](./knowledge-configuration)** - 创建和管理知识库。 -2. **[智能体开发](../agent-development)** - 创建和配置智能体。 +1. **[知识库](./knowledge-configuration)** - 创建和管理知识库。 +2. **[智能体配置](./agent-configuration)** - 创建和配置智能体。 -如果您在使用过程中遇到任何问题,请参考我们的 **[常见问题](../../quick-start/faq.md)** 或在[GitHub Discussions](https://github.com/ModelEngine-Group/nexent/discussions)中进行提问获取支持。 +如果您在使用过程中遇到任何问题,请参考我们的 **[常见问题](../../quick-start/faq)** 或在[GitHub Discussions](https://github.com/ModelEngine-Group/nexent/discussions)中进行提问获取支持。 diff --git a/doc/docs/zh/user-guide/assets/model-management/edit-model-1.png b/doc/docs/zh/user-guide/assets/model-management/edit-model-1.png index cabd518339..02c3ae16ef 100644 Binary files a/doc/docs/zh/user-guide/assets/model-management/edit-model-1.png and b/doc/docs/zh/user-guide/assets/model-management/edit-model-1.png differ diff --git a/doc/docs/zh/user-guide/assets/model-management/select-model-3.png b/doc/docs/zh/user-guide/assets/model-management/select-model-3.png index 9cda79ae6d..78ed606337 100644 Binary files a/doc/docs/zh/user-guide/assets/model-management/select-model-3.png and b/doc/docs/zh/user-guide/assets/model-management/select-model-3.png differ diff --git a/doc/docs/zh/user-guide/assets/model-management/select-model-4.png b/doc/docs/zh/user-guide/assets/model-management/select-model-4.png index 78ed606337..182f53b9eb 100644 Binary files a/doc/docs/zh/user-guide/assets/model-management/select-model-4.png and b/doc/docs/zh/user-guide/assets/model-management/select-model-4.png differ diff --git a/doc/docs/zh/user-guide/assets/model-management/select-model-5.png b/doc/docs/zh/user-guide/assets/model-management/select-model-5.png new file mode 100644 index 0000000000..330ef9c89b Binary files /dev/null and b/doc/docs/zh/user-guide/assets/model-management/select-model-5.png differ