1
This commit is contained in:
@@ -0,0 +1,10 @@
|
||||
.git
|
||||
.idea
|
||||
data
|
||||
workspace
|
||||
docs
|
||||
ui-src/node_modules
|
||||
ui-src/dist
|
||||
Dockerfile
|
||||
docker-compose.yml
|
||||
README.md
|
||||
+12
@@ -0,0 +1,12 @@
|
||||
# 运行时数据与本地环境
|
||||
data/
|
||||
workspace/
|
||||
.idea/
|
||||
.DS_Store
|
||||
|
||||
# 前端
|
||||
ui-src/node_modules/
|
||||
ui-src/dist/
|
||||
|
||||
# 构建产物
|
||||
rag-local
|
||||
+31
@@ -0,0 +1,31 @@
|
||||
# ==================== 阶段 1：构建前端 SPA ====================
|
||||
FROM node:20-alpine AS ui
|
||||
WORKDIR /ui
|
||||
COPY ui-src/package.json ui-src/package-lock.json ./
|
||||
RUN npm ci
|
||||
COPY ui-src/ ./
|
||||
RUN npm run build
|
||||
|
||||
# ==================== 阶段 2：构建后端二进制（modernc sqlite 纯 Go，无 CGO） ====================
|
||||
FROM golang:1.26 AS build
|
||||
WORKDIR /app
|
||||
COPY go.mod go.sum ./
|
||||
RUN go mod download
|
||||
COPY . .
|
||||
RUN CGO_ENABLED=0 go build -ldflags="-s -w" -o rag-local .
|
||||
|
||||
# ==================== 阶段 3：精简运行镜像 ====================
|
||||
FROM alpine:3.20
|
||||
WORKDIR /app
|
||||
RUN apk add --no-cache tzdata && \
|
||||
adduser -D -u 1000 app && \
|
||||
mkdir -p /app/data /app/workspace && \
|
||||
chown -R app:app /app
|
||||
ENV TZ=Asia/Shanghai
|
||||
COPY --from=build /app/rag-local /app/rag-local
|
||||
COPY --from=build /app/config.yml /app/config.yml
|
||||
COPY --from=ui /ui/dist /app/ui-src/dist
|
||||
USER app
|
||||
EXPOSE 8080
|
||||
VOLUME ["/app/data", "/app/workspace"]
|
||||
ENTRYPOINT ["/app/rag-local"]
|
||||
@@ -0,0 +1,119 @@
|
||||
# rag-local 本地知识库
|
||||
|
||||
纯本地的 RAG（检索增强生成）知识库系统，基于 SQLite 全栈文件型存储，无外部数据库依赖。
|
||||
|
||||
## 功能
|
||||
|
||||
- **文档流水线**：上传 txt / md / pdf / docx / html → 自动解析 → 标题感知分块 → 向量化 + 全文索引
|
||||
- **混合检索**：sqlite-vec 向量 KNN + FTS5 全文 BM25，RRF 融合排序，中文 gse 分词
|
||||
- **RAG 问答**：SSE 流式对话，检索引用（含来源与得分）随回答展示，会话历史持久化
|
||||
- **知识图谱**：解析时 LLM 抽取实体与关系，问答时实体链接 + 一跳邻居注入提示词，图谱页可视化实体/关系
|
||||
- **单用户门禁**：启动生成访问令牌，登录后 JWT 鉴权
|
||||
|
||||
## 技术栈
|
||||
|
||||
| 层 | 技术 |
|
||||
|---|---|
|
||||
| 后端 | Go + GoFrame v2 + Eino（LLM 编排） |
|
||||
| 存储 | SQLite（modernc 纯 Go 驱动 + sqlite-vec 扩展） |
|
||||
| 全文 | SQLite FTS5 + gse 中文分词 |
|
||||
| 前端 | Vue 3 + Element Plus + Vite（前后端不分离，单端口） |
|
||||
| 部署 | Docker Compose 一键启动 |
|
||||
|
||||
三个 SQLite 文件各司其职：
|
||||
|
||||
| 库 | 文件 | 内容 |
|
||||
|---|---|---|
|
||||
| business | `data/business.db` | 数据集/文档/分块/向量/全文/解析任务/知识图谱 |
|
||||
| system | `data/system.db` | 系统配置（令牌、默认模型）与模型配置 |
|
||||
| chat | `data/chat.db` | 会话与消息 |
|
||||
|
||||
## 快速开始（Docker）
|
||||
|
||||
```bash
|
||||
docker compose up -d --build
|
||||
```
|
||||
|
||||
- 浏览器打开 http://localhost:8080
|
||||
- 启动日志中查看访问令牌（`docker compose logs rag-local` 搜索"访问令牌"）
|
||||
- 登录后按以下流程使用：
|
||||
1. **设置** 页添加对话模型与向量模型（OpenAI 兼容接口，如 Ollama / vLLM / one-api），并点击"测试"验证连通
|
||||
2. **数据集** 页新建数据集，绑定向量模型（不绑定则仅全文检索）
|
||||
3. 进入数据集上传文档，等待解析完成
|
||||
4. **问答** 页选择知识库开始提问，回答可展开查看引用来源
|
||||
5. **知识图谱** 页查看解析时抽取的实体与关系
|
||||
|
||||
数据保存在 `./data` 与 `./workspace`，删除容器不丢失。
|
||||
|
||||
## 本地开发
|
||||
|
||||
```bash
|
||||
# 后端（Go 1.26+）
|
||||
go run . # 监听 :8080，启动日志打印访问令牌
|
||||
|
||||
# 前端（开发热更新）
|
||||
cd ui-src
|
||||
npm install
|
||||
npm run dev # Vite 开发服务器，API 代理见 vite.config.js
|
||||
```
|
||||
|
||||
生产构建时前端产物在 `ui-src/dist`，由 Go 直接托管。
|
||||
|
||||
## 模型配置
|
||||
|
||||
任意 OpenAI 兼容接口均可使用：
|
||||
|
||||
- **对话模型**（`/v1/chat/completions`，支持 SSE 流式）：用于问答与知识图谱实体抽取
|
||||
- **向量模型**（`/v1/embeddings`）：用于分块向量化与语义检索；维度须与 `config.yml` 中 `vector.dim` 一致（默认 1024）
|
||||
|
||||
示例（Ollama）：
|
||||
|
||||
```bash
|
||||
# 拉取模型
|
||||
ollama pull qwen2.5:7b
|
||||
ollama pull nomic-embed-text
|
||||
|
||||
# 设置页配置：
|
||||
# 对话模型：endpoint http://localhost:11434/v1，模型名 qwen2.5:7b
|
||||
# 向量模型：endpoint http://localhost:11434/v1，模型名 nomic-embed-text，维度 768
|
||||
```
|
||||
|
||||
> 注意：向量维度变更需清空 `data/business.db` 重建（vec0 表建表维度固定）。
|
||||
|
||||
## 配置说明（config.yml）
|
||||
|
||||
| 配置 | 说明 |
|
||||
|---|---|
|
||||
| `server.address` | 监听地址，默认 `:8080` |
|
||||
| `server.clientMaxBodySize` | 上传文件上限，默认 200MB |
|
||||
| `vector.dim` | 向量维度，须与向量模型一致 |
|
||||
| `database.cache.ttl` | DAO 查询缓存秒数 |
|
||||
|
||||
## API 概览
|
||||
|
||||
| 分组 | 接口 |
|
||||
|---|---|
|
||||
| `/system-config` | 登录、设置读写、令牌查看/重新生成 |
|
||||
| `/model-config` | 模型配置 CRUD、连通性测试 |
|
||||
| `/dataset` | 数据集 CRUD |
|
||||
| `/document` | 上传、列表、删除、重新向量化 |
|
||||
| `/chunk` | 分块列表、编辑（改后自动重向量化） |
|
||||
| `/parse-task` | 解析任务列表、失败重试 |
|
||||
| `/conversation` `/message` | 会话管理、消息列表、SSE 问答流（`/message/chat`） |
|
||||
| `/kg-entity` `/kg-relation` | 知识图谱实体/关系列表 |
|
||||
|
||||
所有接口除 `/system-config/login` 外均需 `Authorization: Bearer <JWT>`。
|
||||
|
||||
## 项目结构
|
||||
|
||||
```
|
||||
common/ 通用层：HTTP 服务/鉴权/文件解析/中文分词/向量 JSON
|
||||
kb/
|
||||
consts/ 表名、状态、常量
|
||||
model/ entity / dto / domain
|
||||
dao/ 数据访问（每表一个文件）
|
||||
service/ 业务逻辑（每表一个文件 + chat_service 问答编排）
|
||||
controller/ 接口层（每表一个文件）
|
||||
ui-src/ Vue 3 前端
|
||||
docs/ 实现方案文档
|
||||
```
|
||||
@@ -0,0 +1,47 @@
|
||||
package common
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"time"
|
||||
|
||||
"github.com/golang-jwt/jwt/v5"
|
||||
)
|
||||
|
||||
const jwtSecret = "rag-local-jwt-secret-2026"
|
||||
|
||||
const TokenExpireSeconds = 24 * 3600
|
||||
|
||||
type JwtClaims struct {
|
||||
Role string `json:"role"`
|
||||
TokenFp string `json:"token_fp"`
|
||||
jwt.RegisteredClaims
|
||||
}
|
||||
|
||||
func GetJwtSecret() string {
|
||||
return jwtSecret
|
||||
}
|
||||
|
||||
func SignToken(role, tokenFp string, expireSeconds int64) (string, error) {
|
||||
claims := JwtClaims{
|
||||
Role: role,
|
||||
TokenFp: tokenFp,
|
||||
RegisteredClaims: jwt.RegisteredClaims{
|
||||
ExpiresAt: jwt.NewNumericDate(time.Now().Add(time.Duration(expireSeconds) * time.Second)),
|
||||
},
|
||||
}
|
||||
return jwt.NewWithClaims(jwt.SigningMethodHS256, claims).SignedString([]byte(jwtSecret))
|
||||
}
|
||||
|
||||
func ParseToken(tokenStr string) (*JwtClaims, error) {
|
||||
token, err := jwt.ParseWithClaims(tokenStr, &JwtClaims{}, func(token *jwt.Token) (interface{}, error) {
|
||||
return []byte(jwtSecret), nil
|
||||
})
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
claims, ok := token.Claims.(*JwtClaims)
|
||||
if !ok || !token.Valid {
|
||||
return nil, errors.New("invalid token")
|
||||
}
|
||||
return claims, nil
|
||||
}
|
||||
@@ -0,0 +1,81 @@
|
||||
package common
|
||||
|
||||
import (
|
||||
"context"
|
||||
"net/http"
|
||||
"strings"
|
||||
|
||||
"github.com/gogf/gf/v2/net/ghttp"
|
||||
)
|
||||
|
||||
var publicPaths = []string{
|
||||
"/system-config/login",
|
||||
}
|
||||
|
||||
// CheckTokenFingerprint 由 kb/service 注入，避免 common → service 循环依赖
|
||||
var CheckTokenFingerprint func(ctx context.Context, fp string) bool
|
||||
|
||||
func Auth(r *ghttp.Request) {
|
||||
path := r.URL.Path
|
||||
|
||||
// 公开路径（精确匹配）
|
||||
for _, p := range publicPaths {
|
||||
if path == p {
|
||||
r.Middleware.Next()
|
||||
return
|
||||
}
|
||||
}
|
||||
|
||||
// workspace 源文件前缀放行（浏览器下载/预览请求不带 Authorization）
|
||||
if strings.HasPrefix(path, "/workspace/") {
|
||||
r.Middleware.Next()
|
||||
return
|
||||
}
|
||||
|
||||
// 前端静态资源放行：仅 GET / 与 GET /assets/*（hash 路由下 SPA 只请求这两个路径）
|
||||
if r.Method == http.MethodGet && (path == "/" || strings.HasPrefix(path, "/assets/")) {
|
||||
r.Middleware.Next()
|
||||
return
|
||||
}
|
||||
|
||||
auth := r.Header.Get("Authorization")
|
||||
if auth == "" || !strings.HasPrefix(auth, "Bearer ") {
|
||||
r.Response.WriteJson(ghttp.DefaultHandlerResponse{
|
||||
Code: http.StatusUnauthorized,
|
||||
Message: "未登录或登录已过期",
|
||||
})
|
||||
r.Exit()
|
||||
return
|
||||
}
|
||||
|
||||
claims, err := ParseToken(auth[7:])
|
||||
if err != nil {
|
||||
r.Response.WriteJson(ghttp.DefaultHandlerResponse{
|
||||
Code: http.StatusUnauthorized,
|
||||
Message: "登录已过期，请重新登录",
|
||||
})
|
||||
r.Exit()
|
||||
return
|
||||
}
|
||||
|
||||
// 指纹校验：访问令牌被重新生成后，旧会话立即失效
|
||||
if CheckTokenFingerprint != nil && !CheckTokenFingerprint(r.Context(), claims.TokenFp) {
|
||||
r.Response.WriteJson(ghttp.DefaultHandlerResponse{
|
||||
Code: http.StatusUnauthorized,
|
||||
Message: "访问令牌已变更，请重新登录",
|
||||
})
|
||||
r.Exit()
|
||||
return
|
||||
}
|
||||
|
||||
r.SetCtxVar("role", claims.Role)
|
||||
r.Middleware.Next()
|
||||
}
|
||||
|
||||
func GetRole(r *ghttp.Request) string {
|
||||
v := r.GetCtxVar("role")
|
||||
if v == nil {
|
||||
return ""
|
||||
}
|
||||
return v.String()
|
||||
}
|
||||
@@ -0,0 +1,55 @@
|
||||
package common
|
||||
|
||||
import (
|
||||
"context"
|
||||
|
||||
"github.com/gogf/gf/v2/database/gdb"
|
||||
"github.com/gogf/gf/v2/frame/g"
|
||||
"github.com/gogf/gf/v2/os/gtime"
|
||||
"github.com/gogf/gf/v2/util/gconv"
|
||||
)
|
||||
|
||||
func prepareInsertData(data any) map[string]any {
|
||||
m := gconv.Map(data, gconv.MapOption{Tags: []string{"orm"}})
|
||||
delete(m, "id")
|
||||
m["created_at"] = gtime.Now().Format("Y-m-d H:i:s")
|
||||
m["updated_at"] = gtime.Now().Format("Y-m-d H:i:s")
|
||||
delete(m, "deleted_at")
|
||||
return m
|
||||
}
|
||||
|
||||
func InsertAndReturnId(ctx context.Context, table string, data any) (id int64, err error) {
|
||||
m := prepareInsertData(data)
|
||||
r, err := g.DB().Model(table).Ctx(ctx).Data(m).Insert()
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
if r == nil {
|
||||
return 0, nil
|
||||
}
|
||||
return r.LastInsertId()
|
||||
}
|
||||
|
||||
func GetOneByPk[T any](ctx context.Context, table string, pk int64) (res *T, err error) {
|
||||
r, err := g.DB().Model(table).Ctx(ctx).
|
||||
Cache(gdb.CacheOption{Duration: CacheTTL(), Name: table + "_GetOneByPk_" + gconv.String(pk)}).
|
||||
Where("id", pk).One()
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if r == nil {
|
||||
return nil, nil
|
||||
}
|
||||
err = r.Struct(&res)
|
||||
return
|
||||
}
|
||||
|
||||
func UpdateByPk(ctx context.Context, table string, pk int64, data any) error {
|
||||
_, err := g.DB().Model(table).Ctx(ctx).Data(data).Where("id", pk).Update()
|
||||
return err
|
||||
}
|
||||
|
||||
func DeleteByPk(ctx context.Context, table string, pk int64) error {
|
||||
_, err := g.DB().Model(table).Ctx(ctx).Unscoped().Where("id", pk).Delete()
|
||||
return err
|
||||
}
|
||||
@@ -0,0 +1,22 @@
|
||||
package common
|
||||
|
||||
import (
|
||||
"context"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"github.com/gogf/gf/v2/frame/g"
|
||||
)
|
||||
|
||||
var (
|
||||
cacheTTL time.Duration
|
||||
cacheTTLOnce sync.Once
|
||||
)
|
||||
|
||||
// CacheTTL returns the database query cache TTL from config
|
||||
func CacheTTL() time.Duration {
|
||||
cacheTTLOnce.Do(func() {
|
||||
cacheTTL = time.Duration(g.Cfg().MustGet(context.Background(), "database.cache.ttl", 60).Int()) * time.Second
|
||||
})
|
||||
return cacheTTL
|
||||
}
|
||||
@@ -0,0 +1,87 @@
|
||||
package common
|
||||
|
||||
import (
|
||||
"archive/zip"
|
||||
"io"
|
||||
"strings"
|
||||
)
|
||||
|
||||
type docxParser struct{}
|
||||
|
||||
// Parse docx：解 zip 读 word/document.xml，按 <w:p> 段落提取文本
|
||||
func (p *docxParser) Parse(path string) (string, error) {
|
||||
r, err := zip.OpenReader(path)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
defer r.Close()
|
||||
|
||||
var sb strings.Builder
|
||||
for _, f := range r.File {
|
||||
if f.Name != "word/document.xml" {
|
||||
continue
|
||||
}
|
||||
rc, err := f.Open()
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
body, err := io.ReadAll(rc)
|
||||
rc.Close()
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
extractDocxText(string(body), &sb)
|
||||
return sb.String(), nil
|
||||
}
|
||||
return "", nil
|
||||
}
|
||||
|
||||
// extractDocxText 解析 document.xml：每个 <w:p> 一段，段内文本去 XML 标签
|
||||
func extractDocxText(xml string, sb *strings.Builder) {
|
||||
for {
|
||||
start := strings.Index(xml, "<w:p")
|
||||
if start < 0 {
|
||||
break
|
||||
}
|
||||
end := findTagEnd(xml, start)
|
||||
if end < 0 {
|
||||
break
|
||||
}
|
||||
para := xml[start:end]
|
||||
xml = xml[end:]
|
||||
|
||||
var paraSb strings.Builder
|
||||
texts := strings.Split(para, "<w:t")
|
||||
for i, seg := range texts {
|
||||
if i == 0 {
|
||||
continue
|
||||
}
|
||||
if !strings.HasPrefix(seg, ">") {
|
||||
continue
|
||||
}
|
||||
value := seg[1:]
|
||||
if j := strings.Index(value, "</w:t>"); j >= 0 {
|
||||
value = value[:j]
|
||||
}
|
||||
paraSb.WriteString(value)
|
||||
}
|
||||
line := strings.TrimSpace(paraSb.String())
|
||||
if line != "" {
|
||||
sb.WriteString(line)
|
||||
sb.WriteString("\n")
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// findTagEnd 找到 <w:p ...> 对应的 </w:p> 结束位置（返回开标签之后的位置）
|
||||
func findTagEnd(xml string, start int) int {
|
||||
openEnd := strings.Index(xml[start:], ">")
|
||||
if openEnd < 0 {
|
||||
return -1
|
||||
}
|
||||
closeTag := strings.Index(xml[start:], "</w:p>")
|
||||
if closeTag < 0 {
|
||||
return -1
|
||||
}
|
||||
return start + closeTag + len("</w:p>")
|
||||
}
|
||||
@@ -0,0 +1,34 @@
|
||||
package common
|
||||
|
||||
import (
|
||||
"os"
|
||||
"strings"
|
||||
|
||||
"github.com/PuerkitoBio/goquery"
|
||||
)
|
||||
|
||||
type htmlParser struct{}
|
||||
|
||||
// Parse html：取 body 文本，标题(h1-h6)前后补空行供分块器识别
|
||||
func (p *htmlParser) Parse(path string) (string, error) {
|
||||
b, err := os.ReadFile(path)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
doc, err := goquery.NewDocumentFromReader(strings.NewReader(string(b)))
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
var sb strings.Builder
|
||||
doc.Find("body").Children().Each(func(i int, sel *goquery.Selection) {
|
||||
switch goquery.NodeName(sel) {
|
||||
case "h1", "h2", "h3", "h4", "h5", "h6", "p", "div", "li", "pre", "blockquote":
|
||||
text := strings.TrimSpace(sel.Text())
|
||||
if text != "" {
|
||||
sb.WriteString(text)
|
||||
sb.WriteString("\n\n")
|
||||
}
|
||||
}
|
||||
})
|
||||
return sb.String(), nil
|
||||
}
|
||||
@@ -0,0 +1,49 @@
|
||||
package common
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"reflect"
|
||||
"regexp"
|
||||
"strings"
|
||||
|
||||
"github.com/gogf/gf/v2/frame/g"
|
||||
"github.com/gogf/gf/v2/net/ghttp"
|
||||
"github.com/gogf/gf/v2/os/gtime"
|
||||
)
|
||||
|
||||
var Httpserver = g.Server()
|
||||
|
||||
func init() {
|
||||
err := gtime.SetTimeZone("Asia/Shanghai")
|
||||
if err != nil {
|
||||
panic("设置时区失败")
|
||||
}
|
||||
Httpserver.SetOpenApiPath("/api.json")
|
||||
// 全局 panic 恢复（最先注册，作为最外层包裹）
|
||||
Httpserver.BindMiddlewareDefault(ghttp.MiddlewareHandlerResponse)
|
||||
// CORS - allow all origins
|
||||
Httpserver.BindMiddlewareDefault(func(r *ghttp.Request) {
|
||||
r.Response.CORS(r.Response.DefaultCORSOptions())
|
||||
r.Middleware.Next()
|
||||
})
|
||||
// 访问令牌鉴权
|
||||
Httpserver.BindMiddlewareDefault(Auth)
|
||||
}
|
||||
|
||||
// RouteRegister 根据控制器结构体名称自动注册路由
|
||||
func RouteRegister(controllers []interface{}) {
|
||||
re := regexp.MustCompile("[A-Z]")
|
||||
for _, t := range controllers {
|
||||
sName := reflect.ValueOf(t).Elem().Type().Name()
|
||||
convertedStr := re.ReplaceAllStringFunc(sName, func(s string) string {
|
||||
return fmt.Sprintf("-%s", strings.ToLower(s))
|
||||
})
|
||||
if len(convertedStr) > 0 && convertedStr[0] == '-' {
|
||||
convertedStr = convertedStr[1:]
|
||||
}
|
||||
Httpserver.Group("/"+convertedStr, func(group *ghttp.RouterGroup) {
|
||||
group.Bind(t)
|
||||
})
|
||||
}
|
||||
go Httpserver.Run()
|
||||
}
|
||||
@@ -0,0 +1,53 @@
|
||||
package common
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// Parser 文档解析器接口：从源文件抽取纯文本
|
||||
type Parser interface {
|
||||
// Parse 返回文档全文（保留标题/段落/空行结构，供分块器使用）
|
||||
Parse(path string) (string, error)
|
||||
}
|
||||
|
||||
var parsers = map[string]Parser{}
|
||||
|
||||
func init() {
|
||||
registerParser("txt", &textParser{})
|
||||
registerParser("md", &textParser{})
|
||||
registerParser("pdf", &pdfParser{})
|
||||
registerParser("docx", &docxParser{})
|
||||
registerParser("html", &htmlParser{})
|
||||
}
|
||||
|
||||
func registerParser(ext string, p Parser) {
|
||||
parsers[ext] = p
|
||||
}
|
||||
|
||||
// SupportedExts 返回支持的扩展名列表（不含点）
|
||||
func SupportedExts() []string {
|
||||
exts := make([]string, 0, len(parsers))
|
||||
for ext := range parsers {
|
||||
exts = append(exts, ext)
|
||||
}
|
||||
return exts
|
||||
}
|
||||
|
||||
// ParseFile 按扩展名分发解析，返回全文
|
||||
func ParseFile(path string) (string, error) {
|
||||
ext := strings.TrimPrefix(extOf(path), ".")
|
||||
p, ok := parsers[ext]
|
||||
if !ok {
|
||||
return "", fmt.Errorf("不支持的文档类型: .%s", ext)
|
||||
}
|
||||
return p.Parse(path)
|
||||
}
|
||||
|
||||
func extOf(path string) string {
|
||||
idx := strings.LastIndexByte(path, '.')
|
||||
if idx < 0 {
|
||||
return ""
|
||||
}
|
||||
return path[idx:]
|
||||
}
|
||||
@@ -0,0 +1,97 @@
|
||||
package common
|
||||
|
||||
import (
|
||||
"archive/zip"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestParseTxt(t *testing.T) {
|
||||
path := filepath.Join(t.TempDir(), "a.txt")
|
||||
if err := os.WriteFile(path, []byte("第一行\n\n## 标题\n第二行"), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
text, err := ParseFile(path)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
for _, part := range []string{"第一行", "## 标题", "第二行"} {
|
||||
if !strings.Contains(text, part) {
|
||||
t.Errorf("missing %q in %q", part, text)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestParseHtml(t *testing.T) {
|
||||
path := filepath.Join(t.TempDir(), "b.html")
|
||||
html := `<html><body><h1>文档标题</h1><p>正文第一段。</p><p>正文第二段。</p></body></html>`
|
||||
if err := os.WriteFile(path, []byte(html), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
text, err := ParseFile(path)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
for _, part := range []string{"文档标题", "正文第一段", "正文第二段"} {
|
||||
if !strings.Contains(text, part) {
|
||||
t.Errorf("missing %q in %q", part, text)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestParseDocx(t *testing.T) {
|
||||
path := filepath.Join(t.TempDir(), "c.docx")
|
||||
f, err := os.Create(path)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
zw := zip.NewWriter(f)
|
||||
w, err := zw.Create("word/document.xml")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
xml := `<?xml version="1.0"?><w:document xmlns:w="urn:x"><w:body><w:p><w:r><w:t>第一段落</w:t></w:r></w:p><w:p><w:r><w:t>第二段落</w:t></w:r></w:p></w:body></w:document>`
|
||||
if _, err := w.Write([]byte(xml)); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := zw.Close(); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
f.Close()
|
||||
|
||||
text, err := ParseFile(path)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
for _, part := range []string{"第一段落", "第二段落"} {
|
||||
if !strings.Contains(text, part) {
|
||||
t.Errorf("missing %q in %q", part, text)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestParsePdf(t *testing.T) {
|
||||
path := "/Users/zhangbin/go/pkg/mod/github.com/pdfcpu/pdfcpu@v0.14.0/pkg/testdata/testRot.pdf"
|
||||
if _, err := os.Stat(path); err != nil {
|
||||
t.Skip("pdfcpu testdata not found")
|
||||
}
|
||||
text, err := ParseFile(path)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if strings.TrimSpace(text) == "" {
|
||||
t.Fatal("pdf text is empty")
|
||||
}
|
||||
}
|
||||
|
||||
func TestParseUnsupported(t *testing.T) {
|
||||
path := filepath.Join(t.TempDir(), "d.xyz")
|
||||
if err := os.WriteFile(path, []byte("x"), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := ParseFile(path); err == nil {
|
||||
t.Fatal("expected error for unsupported type")
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,35 @@
|
||||
package common
|
||||
|
||||
import (
|
||||
"io"
|
||||
"os"
|
||||
"strings"
|
||||
|
||||
"github.com/pdfcpu/pdfcpu/pkg/api"
|
||||
)
|
||||
|
||||
type pdfParser struct{}
|
||||
|
||||
// Parse PDF：抽取全部文本，按页分隔（分页符供分块器识别）
|
||||
func (p *pdfParser) Parse(path string) (string, error) {
|
||||
f, err := os.Open(path)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
defer f.Close()
|
||||
|
||||
var sb strings.Builder
|
||||
err = api.ExtractContent(f, nil, func(r io.Reader, _ int) error {
|
||||
b, err := io.ReadAll(r)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
sb.Write(b)
|
||||
sb.WriteString("\n\n")
|
||||
return nil
|
||||
}, nil)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
return sb.String(), nil
|
||||
}
|
||||
@@ -0,0 +1,20 @@
|
||||
package common
|
||||
|
||||
import (
|
||||
"os"
|
||||
"strings"
|
||||
)
|
||||
|
||||
type textParser struct{}
|
||||
|
||||
// Parse txt / md：直接读全文。md 保留 # 标题行供分块器识别
|
||||
func (p *textParser) Parse(path string) (string, error) {
|
||||
b, err := os.ReadFile(path)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
text := string(b)
|
||||
text = strings.ReplaceAll(text, "\r\n", "\n")
|
||||
text = strings.ReplaceAll(text, "\r", "\n")
|
||||
return text, nil
|
||||
}
|
||||
@@ -0,0 +1,47 @@
|
||||
package common
|
||||
|
||||
import (
|
||||
"strings"
|
||||
|
||||
"github.com/go-ego/gse"
|
||||
)
|
||||
|
||||
var seg gse.Segmenter
|
||||
|
||||
func init() {
|
||||
if err := seg.LoadDict(); err != nil {
|
||||
panic("gse load dict failed: " + err.Error())
|
||||
}
|
||||
}
|
||||
|
||||
// Tokenize 中文分词，空格连接（FTS5 索引列格式：content_tokens）
|
||||
func Tokenize(text string) string {
|
||||
tokens := seg.Cut(text, true)
|
||||
var sb strings.Builder
|
||||
for _, t := range tokens {
|
||||
t = strings.TrimSpace(t)
|
||||
if t == "" {
|
||||
continue
|
||||
}
|
||||
sb.WriteString(t)
|
||||
sb.WriteString(" ")
|
||||
}
|
||||
return strings.TrimSpace(sb.String())
|
||||
}
|
||||
|
||||
// TokenizeQuery FTS5 查询串：每个词用双引号包裹（AND 语义），过滤 FTS5 特殊字符
|
||||
func TokenizeQuery(text string) string {
|
||||
tokens := seg.Cut(text, true)
|
||||
var parts []string
|
||||
for _, t := range tokens {
|
||||
t = strings.TrimSpace(t)
|
||||
if t == "" {
|
||||
continue
|
||||
}
|
||||
if strings.ContainsAny(t, "\"*:()") {
|
||||
continue
|
||||
}
|
||||
parts = append(parts, "\""+t+"\"")
|
||||
}
|
||||
return strings.Join(parts, " ")
|
||||
}
|
||||
@@ -0,0 +1,22 @@
|
||||
package common
|
||||
|
||||
import (
|
||||
"crypto/rand"
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
)
|
||||
|
||||
// RandomToken 生成 n 字节随机数的十六进制字符串（2n 位）
|
||||
func RandomToken(n int) string {
|
||||
b := make([]byte, n)
|
||||
if _, err := rand.Read(b); err != nil {
|
||||
panic("crypto/rand failed: " + err.Error())
|
||||
}
|
||||
return hex.EncodeToString(b)
|
||||
}
|
||||
|
||||
// TokenFingerprint 计算令牌 SHA-256 指纹前 16 位，用于 JWT 会话校验
|
||||
func TokenFingerprint(token string) string {
|
||||
sum := sha256.Sum256([]byte(token))
|
||||
return hex.EncodeToString(sum[:8])
|
||||
}
|
||||
+30
@@ -0,0 +1,30 @@
|
||||
database:
|
||||
default:
|
||||
name: data/business.db
|
||||
type: sqlite
|
||||
debug: false
|
||||
system:
|
||||
name: data/system.db
|
||||
type: sqlite
|
||||
debug: false
|
||||
chat:
|
||||
name: data/chat.db
|
||||
type: sqlite
|
||||
debug: false
|
||||
cache:
|
||||
ttl: 60 # DAO查询缓存时间(秒)，0为禁用缓存
|
||||
server:
|
||||
address: :8080
|
||||
name: rag-local
|
||||
workerId: 1
|
||||
clientMaxBodySize: 209715200 # 200MB 上传文件限制
|
||||
requestTimeout: 3000 # HTTP请求超时(秒)，支持 AI 长响应
|
||||
|
||||
# 向量配置
|
||||
vector:
|
||||
dim: 1024 # 向量维度（须与所用 embedding 模型一致，变更需清库重建）
|
||||
|
||||
# AI模型调用配置
|
||||
chat:
|
||||
timeout: 600 # 对话模型API请求超时时间(秒)
|
||||
max_retries: 3 # 请求失败最大重试次数
|
||||
@@ -0,0 +1,13 @@
|
||||
services:
|
||||
rag-local:
|
||||
build: .
|
||||
container_name: rag-local
|
||||
restart: unless-stopped
|
||||
ports:
|
||||
- "8080:8080"
|
||||
volumes:
|
||||
# 数据目录（三个 SQLite 库）与文档工作区持久化
|
||||
- ./data:/app/data
|
||||
- ./workspace:/app/workspace
|
||||
environment:
|
||||
- TZ=Asia/Shanghai
|
||||
+881
@@ -0,0 +1,881 @@
|
||||
# rag-local 本地知识库 实现方案
|
||||
|
||||
> 技术栈：GoFrame v2 + Vue 3 + Eino（字节 CloudWeGo LLM 编排框架）
|
||||
> 数据库：全部文件型 —— SQL = SQLite，向量 = sqlite-vec（SQLite 扩展），全文检索 = SQLite FTS5
|
||||
> 部署：docker-compose 一键部署，前后端不分离，统一端口暴露
|
||||
> 鉴权：单用户 + 启动生成的访问令牌（面向边缘设备部署）
|
||||
|
||||
---
|
||||
|
||||
## 1. 项目概述
|
||||
|
||||
本地私有知识库系统（面向 NAS / 小主机等边缘设备单机部署），支持：
|
||||
|
||||
- 文档管理：上传（txt / md / pdf / docx / html）、自动解析、分块、向量化
|
||||
- 混合检索：语义（向量）检索 + 关键词（全文）检索，RRF 融合排序
|
||||
- RAG 问答：基于 Eino 编排的流式对话，答案附带引用来源
|
||||
- 模型可配置：对话模型 / 嵌入模型均为 OpenAI 兼容 API，可对接任意供应商（本地 Ollama、DeepSeek、硅基流动、OpenAI 等）
|
||||
- 单用户访问令牌登录：token 首次启动随机生成并打印在控制台，登录页输入即可使用
|
||||
|
||||
设计原则：
|
||||
|
||||
1. **全文件型数据库**：业务 SQL、向量、全文检索全部落地为普通文件，随数据目录一键备份/迁移，无任何外部数据库服务
|
||||
2. **按业务领域分库文件**：系统 / 数据集 / 问答 三个 SQLite 文件，互不干扰
|
||||
3. **分层文件与表对齐**：每张业务表对应一组 `entity / dao / service / controller / dto` 文件，数量严格对齐（参照 video-factory 17 表 × 5 层的组织方式）
|
||||
4. **前后端不分离**：Vue 构建产物由 GoFrame 统一端口托管，单容器部署
|
||||
5. **零配置**：不设用户体系与角色权限（单机单人场景），启动即用
|
||||
|
||||
---
|
||||
|
||||
## 2. 技术选型
|
||||
|
||||
| 能力 | 选型 | 说明 |
|
||||
|---|---|---|
|
||||
| Web 框架 | GoFrame v2.10+ | 与 video-factory 一致，复用其分层/路由/鉴权模式 |
|
||||
| SQL 数据库 | SQLite（modernc.org/sqlite 纯 Go 实现） | GoFrame `contrib/drivers/sqlite/v2`，免 CGO，`CGO_ENABLED=0` |
|
||||
| 向量检索 | **sqlite-vec**（`modernc.org/sqlite/vec` 子包） | modernc 从 **v1.47.0 起内置** sqlite-vec 的 CGO-free 版本，`vec0` 虚拟表提供 KNN 检索；与业务表**同库同事务**，天然文件型 |
|
||||
| 全文检索 | SQLite FTS5（modernc 内置） | BM25 打分；中文通过**应用层分词**（纯 Go `github.com/go-ego/gse`）写入分词列，simple tokenizer 索引 |
|
||||
| LLM 编排 | Eino `github.com/cloudwego/eino` | ChatModel / Embedding 自研 OpenAI 兼容 HTTP 实现（eino-ext 为空壳，仅依赖 eino 接口）；Indexer / Retriever 自研（SQLite 后端），实现 Eino 标准接口 |
|
||||
| 文档解析 | pdfcpu（pdf）、纯 Go zip+xml 解析（docx）、goquery（html）、标准库（txt/md） | 全部纯 Go，免 CGO |
|
||||
| 前端 | Vue 3 + Vite + Element Plus + axios | hash 路由，构建产物 `ui-src/dist`，由 Go 统一端口托管 |
|
||||
| 部署 | Docker 多阶段构建 + docker-compose | 单服务单端口，数据目录挂载宿主机卷 |
|
||||
|
||||
### 2.1 关键版本说明（向量扩展）
|
||||
|
||||
- sqlite-vec 需要 `modernc.org/sqlite >= v1.47.0`（2026-03-17 起内置，免 CGO）
|
||||
- GoFrame v2.10.2 驱动链默认锁定 `modernc.org/sqlite v1.23.1`（过旧），**必须在 go.mod 中显式升级**：
|
||||
```bash
|
||||
go get modernc.org/sqlite@v1.47.0
|
||||
```
|
||||
Go modules 最小版本选择（MVS）会使全链路统一使用 v1.47+；glebarez 为薄封装，API 兼容。主程序只需：
|
||||
```go
|
||||
import _ "modernc.org/sqlite/vec" // 空导入，init 自动注册 vec0 扩展
|
||||
```
|
||||
- **风险与验证**：升级后首步执行 `go build` 验证 glebarez 与 v1.47 兼容；若出现编译错误，启用备选方案（见 §14.1）
|
||||
|
||||
---
|
||||
|
||||
## 3. 总体架构
|
||||
|
||||
```
|
||||
┌─────────────────────────── 浏览器 ───────────────────────────┐
|
||||
│ http://host:8080 │
|
||||
└──────────────────────────────┬───────────────────────────────┘
|
||||
│ 统一端口（SPA 静态资源 + REST API + SSE 流式）
|
||||
┌──────────────────────────────▼───────────────────────────────┐
|
||||
│ GoFrame HTTP Server (:8080) │
|
||||
│ ┌───────────┐ ┌──────────────┐ ┌────────────────────────┐ │
|
||||
│ │ 静态资源 │ │ API 路由 │ │ 访问令牌鉴权 / CORS / │ │
|
||||
│ │ ui-src/dist│ │ /dataset │ │ panic 恢复 中间件 │ │
|
||||
│ └───────────┘ │ /document ... │ └────────────────────────┘ │
|
||||
│ └──────┬───────┘ │
|
||||
│ ┌──────────────┼──────────────┐ │
|
||||
│ ┌──────▼─────┐ ┌──────▼─────┐ ┌──────▼──────┐ │
|
||||
│ │ controller │→│ service │→│ dao │ │
|
||||
│ │ (10 个文件) │ │ (10 个文件) │ │ (10 个文件) │ │
|
||||
│ └────────────┘ └──────┬─────┘ └──────┬──────┘ │
|
||||
│ │ │ │
|
||||
│ ┌─────────▼──────┐ ┌────▼─────────────────────┐ │
|
||||
│ │ Eino RAG 编排 │ │ SQLite × 3（文件型） │ │
|
||||
│ │ Graph/Chain │ │ business.db: 业务表+向量+ │ │
|
||||
│ │ Indexer/Retr. │ │ FTS（vec0 + fts5） │ │
|
||||
│ │ ChatModel │ │ system.db: 令牌/配置 │ │
|
||||
│ │ Embedding │ │ chat.db: 会话/消息 │ │
|
||||
│ └───────┬────────┘ └───────────────────────────┘ │
|
||||
│ │ HTTP（OpenAI 兼容 API） │
|
||||
└──────────────────────┼────────────────────────────────────────┘
|
||||
┌────────▼────────┐
|
||||
│ 模型供应商（可配置）│
|
||||
│ Ollama/OpenAI/ │
|
||||
│ DeepSeek/硅基流动 │
|
||||
└─────────────────┘
|
||||
```
|
||||
|
||||
数据流：
|
||||
|
||||
```
|
||||
上传文档 → 解析文本 → 分块 → Embedding 向量化 ──┐
|
||||
├─→ business.db（chunk 表 + vec0 + fts5 同库同事务）
|
||||
关键词检索（FTS5 BM25）───────────────────────┤
|
||||
↓
|
||||
问题 → 混合检索（向量 + 关键词 + RRF 融合）→ 组装上下文 → ChatModel 流式回答 → SSE
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 4. 目录结构
|
||||
|
||||
```
|
||||
rag-local/
|
||||
├── main.go # 入口：路由注册、静态资源托管、任务轮询启动、访问令牌生成/打印
|
||||
├── config.yml # 多库配置 + 服务配置
|
||||
├── go.mod / go.sum
|
||||
├── Dockerfile # 多阶段构建（ui-builder → builder → runtime）
|
||||
├── docker-compose.yml # 一键部署，挂载数据卷
|
||||
├── common/ # 公共层（与 video-factory 对齐）
|
||||
│ ├── http.go # RouteRegister 自动路由 + OpenAPI + 中间件注册
|
||||
│ ├── auth.go # JWT 签发/解析 + 访问令牌指纹校验
|
||||
│ ├── auth_middleware.go # 鉴权中间件（白名单 + 静态资源放行）
|
||||
│ ├── base_dao.go # InsertAndReturnId / GetOneByPk / UpdateByPk / DeleteByPk
|
||||
│ ├── cache.go # DAO 查询缓存 TTL（database.cache.ttl）
|
||||
│ ├── util.go # 通用工具
|
||||
│ ├── parser.go # 解析器接口 + 注册表（按扩展名分发）
|
||||
│ ├── pdf_parser.go # pdfcpu 解析 PDF
|
||||
│ ├── docx_parser.go # zip+xml 解析 word/document.xml
|
||||
│ ├── html_parser.go # goquery 解析 HTML
|
||||
│ ├── text_parser.go # txt / md
|
||||
│ └── tokenizer.go # gse 中文分词（写索引/检索共用）
|
||||
├── kb/ # 业务模块（knowledge base，对应 video-factory 的 shortdrama）
|
||||
│ ├── consts/
|
||||
│ │ ├── table_name.go # 表名常量 + 数据库组常量（DbGroupSystem 等）
|
||||
│ │ ├── status.go # 文档状态 / 任务状态 / 消息角色
|
||||
│ │ ├── content_type.go # 文件类型 / 模型类型（chat/embedding）
|
||||
│ │ └── consts.go # 通用常量（向量维度默认值、TopK、access_token 配置键）
|
||||
│ ├── model/
|
||||
│ │ ├── entity/ # 10 个文件，与 10 张实体表一一对应
|
||||
│ │ ├── dto/ # 10 个文件，与 entity 对应（含 g.Meta 路由定义）
|
||||
│ │ └── domain/ # 领域对象（RAG 检索结果、引用来源、流式事件等）
|
||||
│ ├── dao/ # 10 个文件，与实体表一一对应
|
||||
│ ├── service/ # 10 个文件，与 dao 对应（文档流水线、RAG 问答编排归入对应 service）
|
||||
│ ├── controller/ # 10 个文件，与 service 对应
|
||||
├── ui-src/ # Vue 3 前端工程
|
||||
│ ├── package.json
|
||||
│ ├── vite.config.js # base:'/'、build.outDir:'dist'
|
||||
│ └── src/
|
||||
│ ├── main.js / App.vue
|
||||
│ ├── router/index.js # hash 路由（登录守卫）
|
||||
│ ├── api/ # axios 封装 + 各模块 API（与后端 dto 对齐）
|
||||
│ ├── store/ # Pinia（auth / dataset 状态）
|
||||
│ ├── views/
|
||||
│ │ ├── Login.vue # 输入访问令牌登录
|
||||
│ │ ├── Layout.vue
|
||||
│ │ ├── DatasetList.vue # 数据集列表
|
||||
│ │ ├── DatasetDetail.vue # 文档管理 + 上传 + 解析状态
|
||||
│ │ ├── Chat.vue # RAG 问答（SSE 流式 + 引用来源）
|
||||
│ │ └── Settings.vue # 模型配置 + 访问令牌管理
|
||||
│ └── components/
|
||||
│ ├── DocumentUpload.vue
|
||||
│ ├── ChunkList.vue
|
||||
│ └── MessageBubble.vue # 流式消息 + 引用折叠面板
|
||||
├── data/ # 仅数据库文件（gitignore）
|
||||
│ ├── business.db # 数据集领域
|
||||
│ ├── system.db # 系统领域
|
||||
│ └── chat.db # 问答领域
|
||||
├── workspace/ # 上传的文档源文件（gitignore，与数据库分离）
|
||||
│ └── {datasetId}/{yyyymmdd}/{uuid}.{ext}
|
||||
└── docs/
|
||||
└── 实现方案.md
|
||||
```
|
||||
|
||||
> 分层文件与表对齐规则（硬性约定，参照 video-factory）：
|
||||
> - `model/entity/` 每个文件定义一张实体表的结构体，`orm` 标签命名
|
||||
> - `dao/` 每个文件 = 一张表的单例 DAO（`var Xxx = &xxxDao{}`），`init()` 内建表 + 索引 + 迁移
|
||||
> - `service/` 每个文件对应一个 DAO，承载业务逻辑（文档流水线、RAG 调用）
|
||||
> - `controller/` 每个文件对应一个 service，暴露 REST 接口（`g.Meta` 定义 path/method）
|
||||
> - `model/dto/` 每个文件定义一张表的 Req/Res 结构体
|
||||
> - 虚拟表（vec0 / FTS5）是 chunk 表的附属索引，**不单独建分层文件**，由 `chunk_dao.go` 统一管理
|
||||
> - **不建 parser/rag 等技术目录**：纯技术能力（文档解析、中文分词）平铺在 `common/`；业务编排（分块、Indexer、Retriever、工作流）归入对应 service 文件
|
||||
|
||||
---
|
||||
|
||||
## 5. 数据库设计
|
||||
|
||||
### 5.1 数据库文件划分（按业务领域）
|
||||
|
||||
| 文件 | 配置组名 | 领域 | 说明 |
|
||||
|---|---|---|---|
|
||||
| `data/business.db` | `default` | 数据集 | 数据集、文档、分块、向量（vec0）、全文索引（FTS5）、解析任务 |
|
||||
| `data/system.db` | `system` | 系统 | 访问令牌、模型配置、系统配置 |
|
||||
| `data/chat.db` | `chat` | 问答 | 会话、消息 |
|
||||
|
||||
config.yml：
|
||||
|
||||
```yaml
|
||||
database:
|
||||
default:
|
||||
name: data/business.db
|
||||
type: sqlite
|
||||
debug: true
|
||||
system:
|
||||
name: data/system.db
|
||||
type: sqlite
|
||||
debug: true
|
||||
chat:
|
||||
name: data/chat.db
|
||||
type: sqlite
|
||||
debug: true
|
||||
cache:
|
||||
ttl: 60
|
||||
|
||||
server:
|
||||
address: :8080
|
||||
name: rag-local
|
||||
workerId: 1
|
||||
clientMaxBodySize: 209715200 # 200MB，支持大文件上传
|
||||
requestTimeout: 3000 # 秒；支持 AI 长响应
|
||||
```
|
||||
|
||||
### 5.2 表清单（10 张实体表 + 2 张虚拟表）
|
||||
|
||||
| # | 表名 | 库 | 类型 | 说明 |
|
||||
|---|---|---|---|---|
|
||||
| 1 | `system_config` | system | 实体 | 访问令牌、默认模型等键值配置 |
|
||||
| 2 | `model_config` | system | 实体 | 模型配置（chat / embedding） |
|
||||
| 3 | `kb_dataset` | business | 实体 | 数据集 |
|
||||
| 4 | `kb_document` | business | 实体 | 文档 |
|
||||
| 5 | `kb_chunk` | business | 实体 | 分块 |
|
||||
| 6 | `kb_chunk_vec` | business | **虚拟表 vec0** | 分块向量（chunk_id 与 kb_chunk 1:1） |
|
||||
| 7 | `kb_chunk_fts` | business | **虚拟表 FTS5** | 分块全文索引 |
|
||||
| 8 | `kb_parse_task` | business | 实体 | 文档解析/向量化任务 |
|
||||
| 9 | `kg_entity` | business | 实体 | 知识图谱实体（LLM 抽取） |
|
||||
| 10 | `kg_relation` | business | 实体 | 知识图谱关系（头实体-关系-尾实体） |
|
||||
| 11 | `chat_conversation` | chat | 实体 | 问答会话 |
|
||||
| 12 | `chat_message` | chat | 实体 | 问答消息 |
|
||||
|
||||
> 实体表 10 张 → `entity / dao / service / controller / dto` 各 10 个文件，严格对齐。
|
||||
> 无 user / login_log / 角色权限：单用户场景，访问令牌存于 system_config。
|
||||
> 知识图谱（`kg_entity` / `kg_relation`）属数据集领域，见 §7.7。
|
||||
> 注：用 `sqlite3 .tables` 会看到 `kb_chunk_fts_*`（5 张）与 `kb_chunk_vec_*`（4 张）等额外表，
|
||||
> 它们是 FTS5 / vec0 虚拟表自动生成的**内部影子表**（倒排索引、向量分块等存储），由 SQLite 自动维护，
|
||||
> 不是业务表，不可删除，删除会导致虚拟表损坏。
|
||||
|
||||
### 5.3 建表 DDL
|
||||
|
||||
表名前缀：`kb_`（数据集领域）、`chat_`（问答领域）；`consts.TableNameXxx` 常量集中管理，参照 video-factory。
|
||||
|
||||
```sql
|
||||
-- ==================== system.db ====================
|
||||
|
||||
CREATE TABLE IF NOT EXISTS system_config (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
cfg_key TEXT NOT NULL DEFAULT '',
|
||||
cfg_value TEXT NOT NULL DEFAULT '',
|
||||
updated_at DATETIME DEFAULT (datetime('now','localtime'))
|
||||
);
|
||||
CREATE UNIQUE INDEX IF NOT EXISTS idx_system_config_key ON system_config(cfg_key);
|
||||
|
||||
-- 预置键：
|
||||
-- access_token 访问令牌（首次启动生成，明文存储，见 §8）
|
||||
-- default_chat_model 默认对话模型配置 id
|
||||
-- default_dataset 默认问答数据集 id
|
||||
|
||||
CREATE TABLE IF NOT EXISTS model_config (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
name TEXT NOT NULL DEFAULT '', -- 配置名称，如 "DeepSeek Chat"
|
||||
model_type TEXT NOT NULL DEFAULT 'chat', -- chat / embedding
|
||||
model_name TEXT NOT NULL DEFAULT '', -- 模型名，如 deepseek-chat
|
||||
endpoint_url TEXT NOT NULL DEFAULT '', -- 如 https://api.deepseek.com
|
||||
api_key TEXT NOT NULL DEFAULT '',
|
||||
dimension INTEGER NOT NULL DEFAULT 1024, -- 仅 embedding 有效（向量维度）
|
||||
extra TEXT NOT NULL DEFAULT '', -- JSON 扩展（temperature 等）
|
||||
created_at DATETIME DEFAULT (datetime('now','localtime')),
|
||||
updated_at DATETIME DEFAULT (datetime('now','localtime'))
|
||||
);
|
||||
|
||||
-- ==================== business.db ====================
|
||||
|
||||
CREATE TABLE IF NOT EXISTS kb_dataset (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
name TEXT NOT NULL DEFAULT '',
|
||||
description TEXT NOT NULL DEFAULT '',
|
||||
embedding_cfg_id INTEGER NOT NULL DEFAULT 0, -- 绑定 embedding 模型配置；切换需重新向量化
|
||||
status INTEGER NOT NULL DEFAULT 1, -- 1 正常 / 0 禁用
|
||||
created_at DATETIME DEFAULT (datetime('now','localtime')),
|
||||
updated_at DATETIME DEFAULT (datetime('now','localtime'))
|
||||
);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS kb_document (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
dataset_id INTEGER NOT NULL DEFAULT 0,
|
||||
filename TEXT NOT NULL DEFAULT '',
|
||||
file_path TEXT NOT NULL DEFAULT '', -- workspace 下相对路径（workspace/3/20260804/xxx.pdf）
|
||||
file_size INTEGER NOT NULL DEFAULT 0,
|
||||
file_type TEXT NOT NULL DEFAULT '', -- txt/md/pdf/docx/html
|
||||
status INTEGER NOT NULL DEFAULT 0, -- 0 待处理 1 处理中 2 完成 3 失败
|
||||
chunk_count INTEGER NOT NULL DEFAULT 0,
|
||||
error_msg TEXT NOT NULL DEFAULT '',
|
||||
created_at DATETIME DEFAULT (datetime('now','localtime')),
|
||||
updated_at DATETIME DEFAULT (datetime('now','localtime'))
|
||||
);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS kb_chunk (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
dataset_id INTEGER NOT NULL DEFAULT 0,
|
||||
document_id INTEGER NOT NULL DEFAULT 0,
|
||||
seq INTEGER NOT NULL DEFAULT 0, -- 分块序号
|
||||
content TEXT NOT NULL DEFAULT '', -- 分块原文
|
||||
meta TEXT NOT NULL DEFAULT '', -- JSON：来源页码/标题路径等
|
||||
created_at DATETIME DEFAULT (datetime('now','localtime'))
|
||||
);
|
||||
CREATE INDEX IF NOT EXISTS idx_kb_chunk_document ON kb_chunk(document_id);
|
||||
|
||||
-- 向量虚拟表（sqlite-vec，维度按 embedding 模型定，默认 1024）
|
||||
CREATE VIRTUAL TABLE IF NOT EXISTS kb_chunk_vec USING vec0(
|
||||
chunk_id INTEGER PRIMARY KEY,
|
||||
embedding float[1024]
|
||||
);
|
||||
|
||||
-- 全文索引虚拟表（FTS5，默认 unicode61 tokenizer + 应用层 gse 中文分词；
|
||||
-- FTS5 不支持 simple tokenizer，中文分词在应用层完成，content_tokens 存分词后空格连接文本）
|
||||
CREATE VIRTUAL TABLE IF NOT EXISTS kb_chunk_fts USING fts5(
|
||||
chunk_id UNINDEXED,
|
||||
dataset_id UNINDEXED,
|
||||
title,
|
||||
content_tokens -- gse 分词后空格连接
|
||||
);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS kb_parse_task (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
document_id INTEGER NOT NULL DEFAULT 0,
|
||||
dataset_id INTEGER NOT NULL DEFAULT 0,
|
||||
task_type TEXT NOT NULL DEFAULT 'parse', -- parse / reembed
|
||||
status INTEGER NOT NULL DEFAULT 0, -- 0 待处理 1 处理中 2 完成 3 失败
|
||||
error_msg TEXT NOT NULL DEFAULT '',
|
||||
created_at DATETIME DEFAULT (datetime('now','localtime')),
|
||||
updated_at DATETIME DEFAULT (datetime('now','localtime'))
|
||||
);
|
||||
CREATE INDEX IF NOT EXISTS idx_kb_parse_task_status ON kb_parse_task(status);
|
||||
|
||||
-- 知识图谱（LLM 抽取，见 §7.7）
|
||||
CREATE TABLE IF NOT EXISTS kg_entity (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
dataset_id INTEGER NOT NULL DEFAULT 0,
|
||||
name TEXT NOT NULL DEFAULT '', -- 实体名
|
||||
entity_type TEXT NOT NULL DEFAULT '', -- 类型（person/org/location/...）
|
||||
meta TEXT NOT NULL DEFAULT '', -- JSON 扩展
|
||||
created_at DATETIME DEFAULT (datetime('now','localtime'))
|
||||
);
|
||||
CREATE INDEX IF NOT EXISTS idx_kg_entity_dataset_name ON kg_entity(dataset_id, name);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS kg_relation (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
dataset_id INTEGER NOT NULL DEFAULT 0,
|
||||
head_id INTEGER NOT NULL DEFAULT 0, -- 头实体 kg_entity.id
|
||||
relation TEXT NOT NULL DEFAULT '', -- 关系类型
|
||||
tail_id INTEGER NOT NULL DEFAULT 0, -- 尾实体 kg_entity.id
|
||||
meta TEXT NOT NULL DEFAULT '', -- JSON：来源 chunk、置信度
|
||||
created_at DATETIME DEFAULT (datetime('now','localtime'))
|
||||
);
|
||||
CREATE INDEX IF NOT EXISTS idx_kg_relation_head ON kg_relation(head_id);
|
||||
CREATE INDEX IF NOT EXISTS idx_kg_relation_tail ON kg_relation(tail_id);
|
||||
|
||||
-- ==================== chat.db ====================
|
||||
|
||||
CREATE TABLE IF NOT EXISTS chat_conversation (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
dataset_id INTEGER NOT NULL DEFAULT 0, -- 问答绑定的数据集
|
||||
title TEXT NOT NULL DEFAULT '',
|
||||
created_at DATETIME DEFAULT (datetime('now','localtime')),
|
||||
updated_at DATETIME DEFAULT (datetime('now','localtime'))
|
||||
);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS chat_message (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
conversation_id INTEGER NOT NULL DEFAULT 0,
|
||||
role TEXT NOT NULL DEFAULT 'user', -- user / assistant / system
|
||||
content TEXT NOT NULL DEFAULT '',
|
||||
citations TEXT NOT NULL DEFAULT '', -- JSON：引用来源（文档/分块/得分）
|
||||
created_at DATETIME DEFAULT (datetime('now','localtime'))
|
||||
);
|
||||
CREATE INDEX IF NOT EXISTS idx_chat_message_conversation ON chat_message(conversation_id);
|
||||
```
|
||||
|
||||
### 5.4 关键设计决策
|
||||
|
||||
1. **向量与业务同库同事务**：`kb_chunk_vec` 与 `kb_chunk` 同在 business.db。删除文档时，`chunk → vec → fts` 在一个事务内删除，无一致性问题。
|
||||
2. **数据库与源文件分离**：`data/` 仅存 3 个 db 文件；上传源文件存 `workspace/`。备份 = 两个目录分别打包（db 小而关键、workspace 大而可重建），清理与迁移互不影响。
|
||||
3. **向量维度跟随模型**：`model_config.dimension` 决定 vec0 建表维度；数据集绑定 embedding 配置（`kb_dataset.embedding_cfg_id`）。切换 embedding 模型时前端提示"重新向量化"，触发 `reembed` 任务（重建该库全部向量 + FTS）。
|
||||
4. **中文分词在应用层**：SQLite 内置 tokenizer 无法正确切分中文。写入 FTS5 时用 gse 分词后以空格连接存入 `content_tokens`；查询时对 query 同样分词。备选：trigram tokenizer（召回差但零依赖）。
|
||||
5. **FTS5 表自包含**：只存 `chunk_id + dataset_id + title + content_tokens`，原文在 `kb_chunk`，命中后回表 join 取原文与元数据，避免 FTS 表膨胀。
|
||||
6. **解析任务用表驱动**：`kb_parse_task` 轮询模式（与 video-factory 的 `StartVideoPoller` 一致），单 goroutine 串行消费，避免 SQLite 并发写冲突。
|
||||
7. **SQLite 并发**：3 个库文件各自独立连接；写操作集中在任务轮询 goroutine 与用户操作，使用 WAL 模式（`PRAGMA journal_mode=WAL`）提升读写并发。
|
||||
|
||||
---
|
||||
|
||||
## 6. 分层实现规范（参照 video-factory）
|
||||
|
||||
### 6.1 consts 层
|
||||
|
||||
```go
|
||||
// kb/consts/table_name.go
|
||||
const (
|
||||
TableNameSystemConfig = "system_config"
|
||||
TableNameModelConfig = "model_config"
|
||||
TableNameDataset = "kb_dataset"
|
||||
TableNameDocument = "kb_document"
|
||||
TableNameChunk = "kb_chunk"
|
||||
TableNameChunkVec = "kb_chunk_vec"
|
||||
TableNameChunkFts = "kb_chunk_fts"
|
||||
TableNameParseTask = "kb_parse_task"
|
||||
TableNameConversation = "chat_conversation"
|
||||
TableNameMessage = "chat_message"
|
||||
)
|
||||
|
||||
// 数据库组
|
||||
const (
|
||||
DbGroupDefault = "" // business.db
|
||||
DbGroupSystem = "system" // system.db
|
||||
DbGroupChat = "chat" // chat.db
|
||||
)
|
||||
```
|
||||
|
||||
### 6.2 entity 层
|
||||
|
||||
每文件一张表，`orm` 标签与列名一致，时间用 `*gtime.Time`（参照 video-factory 的 `user.go`）：
|
||||
|
||||
```go
|
||||
type Chunk struct {
|
||||
Id int64 `orm:"id" json:"id"`
|
||||
DatasetId int64 `orm:"dataset_id" json:"dataset_id"`
|
||||
DocumentId int64 `orm:"document_id" json:"document_id"`
|
||||
Seq int `orm:"seq" json:"seq"`
|
||||
Content string `orm:"content" json:"content"`
|
||||
Meta string `orm:"meta" json:"meta"`
|
||||
CreatedAt *gtime.Time `orm:"created_at" json:"created_at"`
|
||||
}
|
||||
```
|
||||
|
||||
### 6.3 dao 层
|
||||
|
||||
- 单例模式：`var Chunk = &chunkDao{}`
|
||||
- `init()` 内执行 `CREATE TABLE IF NOT EXISTS` + 索引 + 迁移（参照 video-factory `user_dao.go`）
|
||||
- 每表一个文件；`chunk_dao.go` 额外管理 `kb_chunk_vec`（向量 KNN 查询、批量插入）与 `kb_chunk_fts`（BM25 检索、分词写索引），并提供事务内删除（chunk+vec+fts）
|
||||
- 查询缓存：`gdb.CacheOption{Duration: common.CacheTTL(), Name: ...}`，写操作后清理（参照 video-factory `clearUserCache`）
|
||||
- 通用 CRUD 用 `common/base_dao.go`（InsertAndReturnId / GetOneByPk / UpdateByPk / DeleteByPk）
|
||||
|
||||
向量与全文检索 DAO 示例：
|
||||
|
||||
```go
|
||||
// 向量 KNN（余弦）
|
||||
func (d *chunkDao) VecSearch(ctx context.Context, datasetId int64, vector []float32, topK int) ([]VecHit, error) {
|
||||
vecStr := vectorToJson(vector) // [0.1,0.2,...]
|
||||
r, err := g.DB(consts.DbGroupDefault).GetCtx(ctx).Raw(
|
||||
"SELECT chunk_id, vec_distance_cosine(embedding, vec_f32(?)) AS d "+
|
||||
"FROM "+consts.TableNameChunkVec+" WHERE embedding MATCH ? ORDER BY d LIMIT ?",
|
||||
vecStr, vecStr, topK,
|
||||
)
|
||||
...
|
||||
}
|
||||
|
||||
// 全文检索（BM25）
|
||||
func (d *chunkDao) FtsSearch(ctx context.Context, datasetId int64, query string, topK int) ([]FtsHit, error) {
|
||||
tokens := common.TokenizeQuery(query)
|
||||
r, err := g.DB(consts.DbGroupDefault).GetCtx(ctx).Raw(
|
||||
"SELECT chunk_id, bm25(kb_chunk_fts) AS score FROM kb_chunk_fts "+
|
||||
"WHERE kb_chunk_fts MATCH ? AND dataset_id = ? ORDER BY score LIMIT ?",
|
||||
tokens, datasetId, topK,
|
||||
)
|
||||
...
|
||||
}
|
||||
```
|
||||
|
||||
### 6.4 service 层
|
||||
|
||||
每表一个文件 + 业务方法（解析与分词等纯技术能力在 `common/`，业务编排全部归入对应 service）：
|
||||
- `system_config_service.go`：**访问令牌**生成/校验/重新生成、系统设置读写
|
||||
- `dataset_service.go`：数据集 CRUD
|
||||
- `document_service.go`：上传落盘、创建文档记录、提交解析任务（解析器调 `common/parser.go`）
|
||||
- `chunk_service.go`：文本分块（splitter，标题感知+800字+100重叠）、`InsertAll`（批量向量化 + 写 chunk+vec0+FTS5，等价 Indexer 职责）、分块列表、编辑（改文本后重分词）
|
||||
- `parse_task_service.go`：任务队列消费（`StartParsePoller`，与 video-factory `StartVideoPoller` 同模式；解析→分块→索引全流程编排）
|
||||
- `conversation_service.go` / `message_service.go`：会话 CRUD、历史消息
|
||||
- `chat_service.go`：**RAG 问答入口**（模型组件工厂、`HybridRetriever`（vec0+FTS5+RRF）、问答工作流编排，SSE 流式回传，落库 message；检索调 `common/tokenizer.go` 分词）
|
||||
- `model_config_service.go`：模型配置 CRUD + 连通性测试
|
||||
|
||||
### 6.5 controller 层 + dto 层
|
||||
|
||||
- 结构体命名决定路由前缀：`dataset` → `/dataset`，`system-config` → `/system-config`（`common.RouteRegister` 自动注册）
|
||||
- 接口定义在 `model/dto/`，`g.Meta` 携带 path/method/summary；OpenAPI 文档自动生成（`/api.json`）
|
||||
- 主要接口清单：
|
||||
|
||||
| 路由 | 说明 |
|
||||
|---|---|
|
||||
| `POST /system-config/login` | **访问令牌登录**（body: {token} → 返回 JWT） |
|
||||
| `GET/PUT /system-config` | 系统设置（默认模型、默认数据集） |
|
||||
| `POST /system-config/regenerate-token` | 重新生成访问令牌（旧会话立即失效） |
|
||||
| `GET/POST/PUT/DELETE /dataset` | 数据集 CRUD |
|
||||
| `POST /document/upload`（multipart）`DELETE /document` `GET /document/list` | 文档管理 |
|
||||
| `GET /document/{id}/chunks` `PUT /chunk` | 分块查看/编辑 |
|
||||
| `POST /document/{id}/reembed` | 重新向量化 |
|
||||
| `GET /parse-task/list` `POST /parse-task/{id}/retry` | 任务管理 |
|
||||
| `GET/POST/DELETE /conversation` | 会话 CRUD |
|
||||
| `GET /message/list` | 历史消息 |
|
||||
| **`POST /message/chat`** | **RAG 问答（SSE 流式）** |
|
||||
| `GET/POST/PUT/DELETE /model-config` | 模型配置 CRUD |
|
||||
| `GET /workspace/*` | 源文件访问（main.go BindHandler 静态服务，鉴权放行 + 路径穿越防护） |
|
||||
| `POST /model-config/{id}/test` | 连通性测试 |
|
||||
|
||||
---
|
||||
|
||||
## 7. RAG 问答设计
|
||||
|
||||
> 本层不再单独建目录：分块/Indexer 归入 `chunk_service.go`，Retriever/组件工厂/工作流归入 `chat_service.go`，分词归入 `common/tokenizer.go`。
|
||||
|
||||
### 7.1 组件清单
|
||||
|
||||
| 组件 | 实现 | 归属 |
|
||||
|---|---|---|
|
||||
| ChatModel | 自研 `OpenAIChatModel` | `chat_service.go`：OpenAI 兼容 `/chat/completions`（Generate/Stream），baseURL 可配，兼容 Ollama/DeepSeek/OpenAI 等 |
|
||||
| Embedding | 自研 `OpenAIEmbedder` | `chat_service.go`：OpenAI 兼容 `/embeddings`；按库绑定的 embedding 配置创建，模型必须与索引端一致 |
|
||||
| Indexer | `ChunkService.InsertAll` | `chunk_service.go`：批量向量化 + 写入 chunk + vec0 + FTS5（解析流水线非图节点，不单独实现 eino Indexer 接口） |
|
||||
| Retriever | **自研 `HybridRetriever`** | `chat_service.go`：实现 `eino.Retriever` 接口，vec0 向量检索 + FTS5 关键词检索 + RRF 融合 |
|
||||
| Splitter | 自研（标题感知分块） | `chunk_service.go`：标题感知 + 800字 + 100重叠 |
|
||||
| Tokenizer | gse 中文分词 | `common/tokenizer.go`：写索引/检索共用 |
|
||||
|
||||
### 7.2 写入路径（`ChunkService.InsertAll`）
|
||||
|
||||
```go
|
||||
func (s *chunkService) InsertAll(ctx context.Context, datasetId, documentId int64,
|
||||
chunks []string, embedder embedding.Embedder) error
|
||||
```
|
||||
|
||||
流程：
|
||||
1. 数据集绑定 embedding 配置时构建 `OpenAIEmbedder`（`parse_task_service` 解析流水线中完成，无配置降级仅写 FTS5）
|
||||
2. 按 `EmbedBatchSize`（16）批量调用 `Embedder.EmbedStrings` 生成向量
|
||||
3. 逐 chunk 同一事务：`kb_chunk` 自增 id → `kb_chunk_vec(chunk_id, vec_f32(?))` → gse 分词后 `kb_chunk_fts(chunk_id, dataset_id, title, content_tokens)`
|
||||
4. 更新 `kb_document.chunk_count`、`status=2`
|
||||
|
||||
> 解析流水线为轮询任务而非图节点，故不单独实现 eino Indexer 接口，`InsertAll` 承担等价职责。
|
||||
|
||||
### 7.3 自研 HybridRetriever（读取路径）
|
||||
|
||||
```go
|
||||
type HybridRetriever struct {
|
||||
datasetId int64
|
||||
embedder embedding.Embedder
|
||||
}
|
||||
|
||||
// 实现 eino Retriever 接口
|
||||
func (h *HybridRetriever) Retrieve(ctx context.Context, query string, opts ...retriever.Option) ([]*schema.Document, error)
|
||||
```
|
||||
|
||||
流程：
|
||||
1. **向量检索**：query → Embedding → `vec0` KNN 余弦 TopK（如 20）
|
||||
2. **关键词检索**：query → gse 分词 → FTS5 MATCH TopK（如 20）
|
||||
3. **RRF 融合**：`score = Σ 1/(60 + rank)`，取 TopK（如 10）
|
||||
4. 回表 `kb_chunk` 取原文与元数据，组装 `schema.Document`，`doc.WithScore(score)` 记录得分与引用信息
|
||||
5. 支持 `retriever.WithTopK` / `WithScoreThreshold` 选项
|
||||
|
||||
```go
|
||||
// RRF 融合
|
||||
type hit struct{ chunkId int64; ranks []int; score float64 }
|
||||
score = Σ(1 / (60 + rank))
|
||||
```
|
||||
|
||||
### 7.4 问答工作流（chat_service.go 编排）
|
||||
|
||||
```
|
||||
用户问题
|
||||
│
|
||||
▼
|
||||
┌─────────┐ ┌──────────────┐ ┌────────────┐ ┌───────────┐
|
||||
│ Hybrid │──▶│ Prompt │──▶│ ChatModel │──▶│ 流式输出 │
|
||||
│ Retriever│ │ (上下文+问题) │ │ (OpenAI兼容)│ │ (SSE) │
|
||||
└─────────┘ └──────────────┘ └────────────┘ └───────────┘
|
||||
```
|
||||
|
||||
- `chat_service.go` 中直接编排 eino 组件：`ChatService.Ask` = HybridRetriever.Retrieve → 组装引用列表 + 系统提示词 → OpenAIChatModel.Stream 流式生成（组件已实现 eino 接口，可随时迁入 graph/chain 拓扑；eino v0.9.13 的 graph 节点类型约束与"中间取引用"需求不匹配，故直接编排）
|
||||
- `message_service.go`：会话解析/创建、用户消息落库、历史消息组装（最近 10 轮）、助手消息 + citations JSON 落库
|
||||
- Prompt 模板：
|
||||
```
|
||||
你是一个本地知识库助手。请仅根据以下资料回答用户问题；若资料不足以回答，请明确说明。
|
||||
回答引用资料时，在对应位置标注 [编号]。
|
||||
|
||||
【资料】
|
||||
[1] 内容...
|
||||
[2] 内容...
|
||||
```
|
||||
- SSE 事件顺序：`citations`（先推，含引用列表与 conversation_id）→ `delta`（增量文本）→ `done`；错误推 `error` 事件
|
||||
- 引用列表编号 [1][2] 与提示词中资料编号一一对应，随助手消息以 JSON 落库（chat_message.citations）
|
||||
|
||||
### 7.5 中文分词
|
||||
|
||||
- `github.com/go-ego/gse`（纯 Go，无 CGO），初始化标准中文词典（约 35 万词）
|
||||
- 写索引：`gse.Cut(text)` → 过滤停用词/单字 → 空格 join → `content_tokens`
|
||||
- 查询：同样流程；额外保留原 query 子串供 trigram 兜底（可选）
|
||||
- 词典可随镜像内置 `dict/zh/dict.txt`
|
||||
|
||||
### 7.6 文档解析流水线
|
||||
|
||||
```
|
||||
POST /document/upload
|
||||
│ 保存文件到 workspace/{datasetId}/{yyyymmdd}/{uuid}.ext
|
||||
│ 插入 kb_document(status=0) + kb_parse_task(status=0)
|
||||
▼
|
||||
StartParsePoller（main.go 启动，3 秒轮询）
|
||||
│ 取 status=0 任务 → 置 status=1
|
||||
│ 1. 按扩展名分发解析器（txt/md 直接读；pdf 用 pdfcpu 抽取文本；
|
||||
│ docx 解 zip 读 word/document.xml 提取段落；html 用 goquery 取 body 文本）
|
||||
│ 2. 分块：优先按标题/段落切（# 标题、空行、PDF 分页），
|
||||
│ 超过 max_chunk_size(800字) 时按句号/换行回退切分，重叠 100 字
|
||||
│ 3. 批量 Embedding → SqliteIndexer.Store
|
||||
│ 4. 更新 document.status=2、chunk_count；任务 status=2
|
||||
│ 失败 → status=3 + error_msg，前端可重试
|
||||
```
|
||||
|
||||
### 7.7 知识图谱（LLM 抽取 + 图增强检索）
|
||||
|
||||
面向"数据集整体关系"类问题（如"谁与谁合作过""公司有哪些产品线"），在向量/关键词检索之外补充图谱能力。图谱**不是替代 RAG**，而是为问答注入结构化关系上下文。
|
||||
|
||||
**构建（LLM 抽取，挂在解析流水线第 3.5 步）**：
|
||||
|
||||
```
|
||||
解析 → 分块 → 向量化 ──▶ 批量抽取（每 chunk 一次 LLM 调用，JSON 输出）
|
||||
▼
|
||||
{entities:[{name, type}], relations:[{head, relation, tail}]}
|
||||
▼
|
||||
upsert kg_entity（按 dataset_id+name 去重）→ 写 kg_relation（带来源 chunk_id）
|
||||
```
|
||||
|
||||
- 复用 M4 的 chat 组件工厂，抽取 Prompt 要求模型只输出 JSON：
|
||||
```json
|
||||
{"entities": [{"name": "张明", "type": "person"}], "relations": [{"head": "张明", "relation": "任职于", "tail": "XX科技"}]}
|
||||
```
|
||||
- 实体按 `(dataset_id, name)` 去重（同一实体多 chunk 出现只建一次，UNIQUE 约束 + ON CONFLICT upsert），关系带来源 chunk_id；删除文档时按分块 id 级联清理
|
||||
- 抽取失败不阻断流水线（记录错误，文档状态仍为完成）；未配置默认对话模型时整体跳过
|
||||
|
||||
**使用（图增强检索，挂在 HybridRetriever 之后）**：
|
||||
|
||||
```
|
||||
用户问题
|
||||
│
|
||||
├─▶ 混合检索（向量+FTS5）──┐
|
||||
├─▶ 实体链接：问题文本分词后与 kg_entity.name 精确/子串匹配，取 Top 3 命中实体
|
||||
│ └─▶ 一跳邻居：取这些实体的关系三元组（头/尾任意一端命中即取，上限 20 条）
|
||||
▼
|
||||
组装上下文：检索片段 + 三元组列表（"【知识图谱】张明 -任职于-> XX科技"）→ ChatModel
|
||||
```
|
||||
|
||||
- 实体链接用 `common/tokenizer.go` 分词 + 名称匹配，零模型调用；打分规则：问题含完整实体名 +5，命中 token 按长度加权，Top3 后名称长者优先
|
||||
- 三元组作为辅助上下文注入 prompt（排序在检索片段之后），增强模型对关系类问题的回答
|
||||
- `kg_entity` / `kg_relation` 两表归属 business.db，各配独立 entity/dao/service/controller 文件（分层 10 文件对齐）
|
||||
|
||||
---
|
||||
|
||||
## 8. 鉴权设计（单用户 + 启动令牌）
|
||||
|
||||
面向边缘设备部署，不做用户体系，只有一个"门禁"级别的访问令牌：
|
||||
|
||||
### 8.1 令牌生命周期
|
||||
|
||||
```
|
||||
首次启动
|
||||
│ 读 system_config(access_token)
|
||||
│ ┌─ 不存在 → 随机生成 32 位十六进制 token，写入 system_config，日志打印：
|
||||
│ │ ============================================
|
||||
│ │ 访问令牌(登录用): a1b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4
|
||||
│ │ 请在登录页输入上述令牌
|
||||
│ │ ============================================
|
||||
│ └─ 已存在 → 直接打印当前 token（便于找回，重启不重新生成）
|
||||
▼
|
||||
登录页：输入 token → POST /system-config/login → 校验通过 → 签发 JWT
|
||||
│ JWT claims: { role: "owner", token_fp: 指纹 }，有效期 24h
|
||||
▼
|
||||
后续请求：Authorization: Bearer <JWT>，中间件校验 token_fp 与当前令牌指纹一致
|
||||
```
|
||||
|
||||
要点：
|
||||
|
||||
- **指纹校验**：JWT 中携带 access_token 的 SHA-256 指纹（前 16 位 hex）；鉴权中间件将 JWT 指纹与 `system_config` 中当前令牌指纹比对（带缓存），不一致即 401。因此**重新生成令牌后所有旧会话立即失效**
|
||||
- **重新生成**：设置页 `POST /system-config/regenerate-token` → 生成新令牌覆盖并打印日志，前端强制跳转登录页
|
||||
- **令牌存储**：system_config 明文存储（本地单机、日志与设置页本就可见明文，且设置页需展示）
|
||||
- **公开路径白名单**：`/system-config/login`、`GET /`、`GET /assets/*`（hash 路由下 SPA 只请求这两类静态路径，无鉴权绕过）；`GET /workspace/*` 前缀放行（同 video-factory `/workspace/*`：浏览器下载/预览请求不带 Authorization），仅做路径穿越防护；其余路径一律校验
|
||||
- 无注册、无角色、无用户表；登录日志不落库（如需审计可在日志文件输出）
|
||||
|
||||
```go
|
||||
// common/auth_middleware.go（伪码）
|
||||
func Auth(r *ghttp.Request) {
|
||||
if isPublicPath(r.URL.Path) { r.Middleware.Next(); return }
|
||||
claims, err := ParseToken(bearer(r))
|
||||
if err != nil || claims.TokenFp != currentTokenFingerprint() {
|
||||
r.Response.WriteJson(401); r.Exit(); return
|
||||
}
|
||||
r.Middleware.Next()
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 9. 前端设计（Vue 3 + Vite + Element Plus）
|
||||
|
||||
### 9.1 工程
|
||||
|
||||
- `ui-src/` 独立 Vite 工程：`vite.config.js` 设 `base: '/'`、`build.outDir: 'dist'`
|
||||
- hash 路由（无需服务端 SPA fallback，与 video-factory 一致），路由守卫：无 JWT → 跳登录页
|
||||
- 依赖：`vue@3`、`vue-router@4`、`pinia`、`element-plus`、`axios`
|
||||
- 构建产物 `ui-src/dist`，本地开发 `npm run build` 后由 Go 统一端口托管；开发期可用 Vite dev server + proxy
|
||||
|
||||
### 9.2 页面
|
||||
|
||||
| 页面 | 功能 |
|
||||
|---|---|
|
||||
| 登录页 | 输入访问令牌登录（提示语：令牌见服务启动日志 / `docker compose logs`）；无注册入口 |
|
||||
| 数据集列表 | 卡片式列表、新建/删除/设置（绑定 embedding 模型） |
|
||||
| 数据集详情 | 文档上传（拖拽 + 进度）、文档列表（解析状态/分块数/失败重试）、分块预览与编辑 |
|
||||
| 问答页 | 左侧会话列表，右侧消息流；SSE 流式渲染，答案带引用折叠面板（点击定位到分块原文） |
|
||||
| 设置页 | 模型配置 CRUD（chat/embedding）、连通性测试；**访问令牌管理**（查看/复制/重新生成，重新生成后强制重新登录） |
|
||||
|
||||
### 9.3 API 封装
|
||||
|
||||
- `api/http.js`：axios 实例（baseURL `/`、token 注入、401 跳转、错误 toast）
|
||||
- `api/xxx.js`：按后端模块组织，与 dto 对齐
|
||||
- 流式：`fetch` + `ReadableStream` 解析 SSE（`data: {"content":"..."}` 增量 / `data: {"citations":[...]}` 引用 / `data: [DONE]`）
|
||||
|
||||
---
|
||||
|
||||
## 10. Docker 部署
|
||||
|
||||
### 10.1 Dockerfile（多阶段，参照 video-factory）
|
||||
|
||||
```dockerfile
|
||||
# ==================== 前端构建 ====================
|
||||
FROM node:20-alpine AS ui-builder
|
||||
RUN apk add --no-cache git
|
||||
WORKDIR /build-ui
|
||||
COPY rag-local/ui-src/package.json rag-local/ui-src/package-lock.json ./
|
||||
RUN npm ci --registry=https://registry.npmmirror.com
|
||||
COPY rag-local/ui-src/ ./
|
||||
RUN npm run build
|
||||
|
||||
# ==================== 后端构建 ====================
|
||||
FROM golang:alpine AS builder
|
||||
RUN sed -i 's/dl-cdn.alpinelinux.org/mirrors.aliyun.com/g' /etc/apk/repositories \
|
||||
&& apk add --no-cache git ca-certificates tzdata
|
||||
ENV TZ=Asia/Shanghai GO111MODULE=on \
|
||||
GOPROXY=https://goproxy.cn,direct \
|
||||
CGO_ENABLED=0 GOTOOLCHAIN=auto
|
||||
WORKDIR /build
|
||||
COPY rag-local/ .
|
||||
RUN go mod download && go build -ldflags="-s -w" -o main ./main.go
|
||||
|
||||
# ==================== 运行镜像 ====================
|
||||
FROM alpine:3.19
|
||||
RUN sed -i 's/dl-cdn.alpinelinux.org/mirrors.aliyun.com/g' /etc/apk/repositories \
|
||||
&& apk add --no-cache ca-certificates tzdata
|
||||
ENV TZ=Asia/Shanghai
|
||||
RUN ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && echo $TZ > /etc/timezone
|
||||
WORKDIR /app
|
||||
COPY --from=builder /build/main .
|
||||
COPY --from=builder /build/config.yml .
|
||||
COPY --from=ui-builder /build-ui/dist ./ui-src/dist
|
||||
RUN mkdir -p /app/data /app/workspace \
|
||||
&& printf '#!/bin/sh\nfor db in business.db system.db chat.db; do\n if [ -d /app/data/$db ]; then rm -rf /app/data/$db; fi\n touch /app/data/$db 2>/dev/null || true\ndone\nexec ./main\n' > /app/entrypoint.sh \
|
||||
&& chmod +x /app/entrypoint.sh
|
||||
EXPOSE 8080
|
||||
ENTRYPOINT ["/app/entrypoint.sh"]
|
||||
```
|
||||
|
||||
### 10.2 docker-compose.yml
|
||||
|
||||
```yaml
|
||||
services:
|
||||
rag-local:
|
||||
build:
|
||||
context: ..
|
||||
dockerfile: rag-local/Dockerfile
|
||||
container_name: rag-local
|
||||
ports:
|
||||
- "8080:8080" # 统一端口：前后端一体
|
||||
volumes:
|
||||
- /data/rag-local/data:/app/data # 数据库（3 个 db 文件）
|
||||
- /data/rag-local/workspace:/app/workspace # 上传的文档源文件
|
||||
restart: unless-stopped
|
||||
|
||||
networks:
|
||||
default:
|
||||
name: rag-local-network
|
||||
```
|
||||
|
||||
> 数据即文件：备份 = 打包 `data`（3 个 db）+ `workspace`（源文件）两个目录；迁移 = 拷贝到新机器即可。
|
||||
> 首次启动后通过 `docker compose logs rag-local` 查看访问令牌。
|
||||
|
||||
---
|
||||
|
||||
## 11. 开发与构建命令
|
||||
|
||||
```bash
|
||||
# 后端
|
||||
cd rag-local
|
||||
go mod tidy
|
||||
go get modernc.org/sqlite@v1.47.0 # 关键：向量扩展依赖
|
||||
go build ./... # 验证编译（见 §2.1 风险提示）
|
||||
go run main.go # 启动日志中查看访问令牌
|
||||
|
||||
# 前端
|
||||
cd rag-local/ui-src
|
||||
npm install
|
||||
npm run build # 产物 ui-src/dist，Go 统一端口托管
|
||||
npm run dev # 开发期（Vite + proxy）
|
||||
|
||||
# 部署
|
||||
docker compose up -d --build
|
||||
docker compose logs rag-local # 查看访问令牌
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 12. 实施步骤（里程碑）
|
||||
|
||||
| 阶段 | 内容 | 验收标准 |
|
||||
|---|---|---|
|
||||
| **M1 骨架** | go mod、config.yml 三库、common 层（http/auth/base_dao/cache）、consts、system_config（令牌生成/登录）与 model_config 全链路（entity/dao/service/controller/dto）、ui-src 空壳 SPA 由 Go 托管 | 8080 端口可打开页面，启动日志打印令牌，登录页输入令牌可进入，3 个 db 文件生成 |
|
||||
| **M2 文档流水线** | dataset/document/parse_task 全链路、解析器（txt/md/pdf/docx/html）、分块、任务轮询 | 上传文档 → 解析 → 分块 → 状态流转，前端可看 chunk 列表 |
|
||||
| **M3 向量 + 全文** | 升级 modernc v1.47、验证 vec0、chunk_vec/chunk_fts 建表、gse 分词、SqliteIndexer/向量检索/FTS5 检索、RRF 融合 | 库内文档可被语义检索与关键词检索命中 |
|
||||
| **M4 RAG 问答** ✅ | model_config 全链路、chat/embedding 组件工厂、Eino Graph 工作流、SSE 流式、conversation/message 落库、引用展示 | 问答页流式对话，回答有引用来源，历史消息可回溯 |
|
||||
| **M5 知识图谱** ✅ | kg_entity/kg_relation 全链路、LLM 抽取（挂在解析流水线）、实体链接 + 一跳邻居注入 prompt | 图谱页可看实体/关系，问答能回答关系类问题 |
|
||||
| **M6 打磨与部署** ✅ | 分块编辑（改后自动重向量化）、文档重新向量化、令牌查看/重新生成、模型连通性测试、设置/数据集/详情三页填充、Dockerfile + docker-compose + README | docker compose up 一键启动，全功能可用 |
|
||||
|
||||
---
|
||||
|
||||
## 13. 依赖清单（go.mod）
|
||||
|
||||
```
|
||||
github.com/gogf/gf/v2 v2.10.x
|
||||
github.com/gogf/gf/contrib/drivers/sqlite/v2 v2.10.x
|
||||
modernc.org/sqlite v1.47.0+ # 显式升级，内置 sqlite-vec
|
||||
github.com/cloudwego/eino # RAG 编排
|
||||
github.com/cloudwego/eino-ext # chat/openai、embedding/openai
|
||||
github.com/go-ego/gse # 中文分词（纯 Go）
|
||||
github.com/pdfcpu/pdfcpu # PDF 文本抽取
|
||||
github.com/PuerkitoBio/goquery # HTML 解析
|
||||
github.com/golang-jwt/jwt/v5 # JWT
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 14. 风险与备选方案
|
||||
|
||||
### 14.1 向量扩展兼容性（主要风险）
|
||||
|
||||
**风险**：glebarez/go-sqlite（GoFrame 驱动链）锁定 modernc v1.23.1，MVS 升级到 v1.47.0 后可能存在 API 编译不兼容。
|
||||
|
||||
**应对**：M3 阶段第一步即验证 `go build`。若失败，按序切换：
|
||||
|
||||
- **备选 A（推荐）**：自研平面扫描向量检索 —— `kb_chunk_vec` 改用普通表存向量 JSON，检索时过滤 + 内存余弦相似度排序（批量预取 + 并发分片）。10 万 chunk × 1024 维单次查询约 20–50ms，本地知识库规模完全够用。Indexer/Retriever 接口不变，仅换后端实现。
|
||||
- **备选 B**：`gosqlite.org` 模块（CGo-free，类型化 vec + FTS5 API），业务 SQL 走 GoFrame、向量/FTS 走独立连接，应用层 RRF 融合。
|
||||
- **备选 C**：LanceDB（文件型向量库，纯 Go SDK），向量独立目录存储。
|
||||
|
||||
### 14.2 中文检索质量
|
||||
|
||||
FTS5 召回依赖 gse 分词质量；专有名词（人名/产品名）可能切碎。缓解：检索时同时保留 trigram 兜底，或对命中率低的 query 降级为向量检索为主。
|
||||
|
||||
### 14.3 SQLite 写并发
|
||||
|
||||
解析任务轮询与用户操作可能并发写 business.db。缓解：WAL 模式 + 任务串行消费 + 写操作集中到 service 层（参照 video-factory 单机场景）。
|
||||
|
||||
### 14.4 切换 embedding 模型
|
||||
|
||||
不同模型的向量空间不可混用。方案：数据集绑定 embedding 配置，切换时强制"重新向量化"（`reembed` 任务重建全部向量与 FTS 索引），并清空缓存。
|
||||
|
||||
---
|
||||
|
||||
## 15. 与 video-factory 的差异点说明
|
||||
|
||||
| 项 | video-factory | rag-local |
|
||||
|---|---|---|
|
||||
| 数据库 | 3 个 SQLite（业务/系统/财务） | 3 个 SQLite（业务/系统/问答） |
|
||||
| 向量/全文 | 无 | **SQLite 同库内嵌**（vec0 + FTS5），无额外服务 |
|
||||
| AI 层 | 自研 ReAct agent（openai 直连） | **Eino** 组件化编排（Indexer/Retriever/Graph） |
|
||||
| 鉴权 | 多用户 + 角色 + JWT（user/login_log 表） | **单用户访问令牌**：启动生成打印，登录换取 JWT，无用户表 |
|
||||
| 前端 | HTML + 少量 Vue | 纯 Vue 3 SPA（构建产物 Go 托管） |
|
||||
| 异步任务 | 视频生成轮询 | 文档解析/向量化轮询 |
|
||||
| 其余 | 分层/路由/鉴权/缓存/部署模式 | 完全对齐 |
|
||||
@@ -0,0 +1,81 @@
|
||||
module rag-local
|
||||
|
||||
go 1.26.1
|
||||
|
||||
require (
|
||||
github.com/PuerkitoBio/goquery v1.12.0
|
||||
github.com/cloudwego/eino v0.9.13
|
||||
github.com/go-ego/gse v1.0.2
|
||||
github.com/gogf/gf/contrib/drivers/sqlite/v2 v2.10.2
|
||||
github.com/gogf/gf/v2 v2.10.2
|
||||
github.com/golang-jwt/jwt/v5 v5.3.1
|
||||
github.com/pdfcpu/pdfcpu v0.14.0
|
||||
modernc.org/sqlite v1.47.0
|
||||
)
|
||||
|
||||
require (
|
||||
github.com/BurntSushi/toml v1.5.0 // indirect
|
||||
github.com/andybalholm/cascadia v1.3.3 // indirect
|
||||
github.com/bahlo/generic-list-go v0.2.0 // indirect
|
||||
github.com/buger/jsonparser v1.1.1 // indirect
|
||||
github.com/bytedance/gopkg v0.1.3 // indirect
|
||||
github.com/bytedance/sonic v1.15.0 // indirect
|
||||
github.com/bytedance/sonic/loader v0.5.0 // indirect
|
||||
github.com/clbanning/mxj/v2 v2.7.0 // indirect
|
||||
github.com/clipperhouse/uax29/v2 v2.7.0 // indirect
|
||||
github.com/cloudwego/base64x v0.1.6 // indirect
|
||||
github.com/dustin/go-humanize v1.0.1 // indirect
|
||||
github.com/eino-contrib/jsonschema v1.0.3 // indirect
|
||||
github.com/emirpasic/gods/v2 v2.0.0-alpha // indirect
|
||||
github.com/fatih/color v1.18.0 // indirect
|
||||
github.com/fsnotify/fsnotify v1.9.0 // indirect
|
||||
github.com/glebarez/go-sqlite v1.21.2 // indirect
|
||||
github.com/go-logr/logr v1.4.3 // indirect
|
||||
github.com/go-logr/stdr v1.2.2 // indirect
|
||||
github.com/google/uuid v1.6.0 // indirect
|
||||
github.com/goph/emperror v0.17.2 // indirect
|
||||
github.com/gorilla/websocket v1.5.3 // indirect
|
||||
github.com/grokify/html-strip-tags-go v0.1.0 // indirect
|
||||
github.com/hhrutter/tiff v1.0.6 // indirect
|
||||
github.com/json-iterator/go v1.1.12 // indirect
|
||||
github.com/klauspost/cpuid/v2 v2.2.9 // indirect
|
||||
github.com/magiconair/properties v1.8.10 // indirect
|
||||
github.com/mailru/easyjson v0.7.7 // indirect
|
||||
github.com/mattn/go-colorable v0.1.13 // indirect
|
||||
github.com/mattn/go-isatty v0.0.20 // indirect
|
||||
github.com/mattn/go-runewidth v0.0.27 // indirect
|
||||
github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect
|
||||
github.com/modern-go/reflect2 v1.0.2 // indirect
|
||||
github.com/ncruces/go-strftime v1.0.0 // indirect
|
||||
github.com/nikolalohinski/gonja v1.5.3 // indirect
|
||||
github.com/olekukonko/errors v1.1.0 // indirect
|
||||
github.com/olekukonko/ll v0.0.9 // indirect
|
||||
github.com/olekukonko/tablewriter v1.1.0 // indirect
|
||||
github.com/pelletier/go-toml/v2 v2.0.9 // indirect
|
||||
github.com/pkg/errors v0.9.1 // indirect
|
||||
github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec // indirect
|
||||
github.com/rivo/uniseg v0.2.0 // indirect
|
||||
github.com/sirupsen/logrus v1.9.3 // indirect
|
||||
github.com/slongfield/pyfmt v0.0.0-20220222012616-ea85ff4c361f // indirect
|
||||
github.com/twitchyliquid64/golang-asm v0.15.1 // indirect
|
||||
github.com/vcaesar/cedar v0.30.0 // indirect
|
||||
github.com/wk8/go-ordered-map/v2 v2.1.8 // indirect
|
||||
github.com/yargevad/filepathx v1.0.0 // indirect
|
||||
go.opentelemetry.io/auto/sdk v1.1.0 // indirect
|
||||
go.opentelemetry.io/otel v1.38.0 // indirect
|
||||
go.opentelemetry.io/otel/metric v1.38.0 // indirect
|
||||
go.opentelemetry.io/otel/sdk v1.38.0 // indirect
|
||||
go.opentelemetry.io/otel/trace v1.38.0 // indirect
|
||||
go.yaml.in/yaml/v3 v3.0.5 // indirect
|
||||
golang.org/x/arch v0.11.0 // indirect
|
||||
golang.org/x/crypto v0.54.0 // indirect
|
||||
golang.org/x/exp v0.0.0-20251023183803-a4bb9ffd2546 // indirect
|
||||
golang.org/x/image v0.44.0 // indirect
|
||||
golang.org/x/net v0.56.0 // indirect
|
||||
golang.org/x/sys v0.47.0 // indirect
|
||||
golang.org/x/text v0.40.0 // indirect
|
||||
gopkg.in/yaml.v3 v3.0.1 // indirect
|
||||
modernc.org/libc v1.70.0 // indirect
|
||||
modernc.org/mathutil v1.7.1 // indirect
|
||||
modernc.org/memory v1.11.0 // indirect
|
||||
)
|
||||
@@ -0,0 +1,332 @@
|
||||
github.com/BurntSushi/toml v1.5.0 h1:W5quZX/G/csjUnuI8SUYlsHs9M38FC7znL0lIO+DvMg=
|
||||
github.com/BurntSushi/toml v1.5.0/go.mod h1:ukJfTF/6rtPPRCnwkur4qwRxa8vTRFBF0uk2lLoLwho=
|
||||
github.com/PuerkitoBio/goquery v1.12.0 h1:pAcL4g3WRXekcB9AU/y1mbKez2dbY2AajVhtkO8RIBo=
|
||||
github.com/PuerkitoBio/goquery v1.12.0/go.mod h1:802ej+gV2y7bbIhOIoPY5sT183ZW0YFofScC4q/hIpQ=
|
||||
github.com/airbrake/gobrake v3.6.1+incompatible/go.mod h1:wM4gu3Cn0W0K7GUuVWnlXZU11AGBXMILnrdOU8Kn00o=
|
||||
github.com/andybalholm/cascadia v1.3.3 h1:AG2YHrzJIm4BZ19iwJ/DAua6Btl3IwJX+VI4kktS1LM=
|
||||
github.com/andybalholm/cascadia v1.3.3/go.mod h1:xNd9bqTn98Ln4DwST8/nG+H0yuB8Hmgu1YHNnWw0GeA=
|
||||
github.com/bahlo/generic-list-go v0.2.0 h1:5sz/EEAK+ls5wF+NeqDpk5+iNdMDXrh3z3nPnH1Wvgk=
|
||||
github.com/bahlo/generic-list-go v0.2.0/go.mod h1:2KvAjgMlE5NNynlg/5iLrrCCZ2+5xWbdbCW3pNTGyYg=
|
||||
github.com/bitly/go-simplejson v0.5.0/go.mod h1:cXHtHw4XUPsvGaxgjIAn8PhEWG9NfngEKAMDJEczWVA=
|
||||
github.com/bmizerany/assert v0.0.0-20160611221934-b7ed37b82869/go.mod h1:Ekp36dRnpXw/yCqJaO+ZrUyxD+3VXMFFr56k5XYrpB4=
|
||||
github.com/buger/jsonparser v1.1.1 h1:2PnMjfWD7wBILjqQbt530v576A/cAbQvEW9gGIpYMUs=
|
||||
github.com/buger/jsonparser v1.1.1/go.mod h1:6RYKKt7H4d4+iWqouImQ9R2FZql3VbhNgx27UK13J/0=
|
||||
github.com/bugsnag/bugsnag-go v1.4.0/go.mod h1:2oa8nejYd4cQ/b0hMIopN0lCRxU0bueqREvZLWFrtK8=
|
||||
github.com/bugsnag/panicwrap v1.2.0/go.mod h1:D/8v3kj0zr8ZAKg1AQ6crr+5VwKN5eIywRkfhyM/+dE=
|
||||
github.com/bytedance/gopkg v0.1.3 h1:TPBSwH8RsouGCBcMBktLt1AymVo2TVsBVCY4b6TnZ/M=
|
||||
github.com/bytedance/gopkg v0.1.3/go.mod h1:576VvJ+eJgyCzdjS+c4+77QF3p7ubbtiKARP3TxducM=
|
||||
github.com/bytedance/sonic v1.15.0 h1:/PXeWFaR5ElNcVE84U0dOHjiMHQOwNIx3K4ymzh/uSE=
|
||||
github.com/bytedance/sonic v1.15.0/go.mod h1:tFkWrPz0/CUCLEF4ri4UkHekCIcdnkqXw9VduqpJh0k=
|
||||
github.com/bytedance/sonic/loader v0.5.0 h1:gXH3KVnatgY7loH5/TkeVyXPfESoqSBSBEiDd5VjlgE=
|
||||
github.com/bytedance/sonic/loader v0.5.0/go.mod h1:AR4NYCk5DdzZizZ5djGqQ92eEhCCcdf5x77udYiSJRo=
|
||||
github.com/certifi/gocertifi v0.0.0-20190105021004-abcd57078448/go.mod h1:GJKEexRPVJrBSOjoqN5VNOIKJ5Q3RViH6eu3puDRwx4=
|
||||
github.com/clbanning/mxj/v2 v2.7.0 h1:WA/La7UGCanFe5NpHF0Q3DNtnCsVoxbPKuyBNHWRyME=
|
||||
github.com/clbanning/mxj/v2 v2.7.0/go.mod h1:hNiWqW14h+kc+MdF9C6/YoRfjEJoR3ou6tn/Qo+ve2s=
|
||||
github.com/clipperhouse/uax29/v2 v2.7.0 h1:+gs4oBZ2gPfVrKPthwbMzWZDaAFPGYK72F0NJv2v7Vk=
|
||||
github.com/clipperhouse/uax29/v2 v2.7.0/go.mod h1:EFJ2TJMRUaplDxHKj1qAEhCtQPW2tJSwu5BF98AuoVM=
|
||||
github.com/cloudwego/base64x v0.1.6 h1:t11wG9AECkCDk5fMSoxmufanudBtJ+/HemLstXDLI2M=
|
||||
github.com/cloudwego/base64x v0.1.6/go.mod h1:OFcloc187FXDaYHvrNIjxSe8ncn0OOM8gEHfghB2IPU=
|
||||
github.com/cloudwego/eino v0.9.13 h1:iD/ETS+lxnNp1VeNPqWVGPWdND6Dbf4LyINbLUlDRcM=
|
||||
github.com/cloudwego/eino v0.9.13/go.mod h1:OBD1mrkfkt/pJa4rkg1P0VnaMeOVl7l8IAdEqY//3IQ=
|
||||
github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
|
||||
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
|
||||
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
|
||||
github.com/dustin/go-humanize v1.0.1 h1:GzkhY7T5VNhEkwH0PVJgjz+fX1rhBrR7pRT3mDkpeCY=
|
||||
github.com/dustin/go-humanize v1.0.1/go.mod h1:Mu1zIs6XwVuF/gI1OepvI0qD18qycQx+mFykh5fBlto=
|
||||
github.com/eino-contrib/jsonschema v1.0.3 h1:2Kfsm1xlMV0ssY2nuxshS4AwbLFuqmPmzIjLVJ1Fsp0=
|
||||
github.com/eino-contrib/jsonschema v1.0.3/go.mod h1:cpnX4SyKjWjGC7iN2EbhxaTdLqGjCi0e9DxpLYxddD4=
|
||||
github.com/emirpasic/gods/v2 v2.0.0-alpha h1:dwFlh8pBg1VMOXWGipNMRt8v96dKAIvBehtCt6OtunU=
|
||||
github.com/emirpasic/gods/v2 v2.0.0-alpha/go.mod h1:W0y4M2dtBB9U5z3YlghmpuUhiaZT2h6yoeE+C1sCp6A=
|
||||
github.com/fatih/color v1.18.0 h1:S8gINlzdQ840/4pfAwic/ZE0djQEH3wM94VfqLTZcOM=
|
||||
github.com/fatih/color v1.18.0/go.mod h1:4FelSpRwEGDpQ12mAdzqdOukCy4u8WUtOY6lkT/6HfU=
|
||||
github.com/fsnotify/fsnotify v1.4.7/go.mod h1:jwhsz4b93w/PPRr/qN1Yymfu8t87LnFCMoQvtojpjFo=
|
||||
github.com/fsnotify/fsnotify v1.9.0 h1:2Ml+OJNzbYCTzsxtv8vKSFD9PbJjmhYF14k/jKC7S9k=
|
||||
github.com/fsnotify/fsnotify v1.9.0/go.mod h1:8jBTzvmWwFyi3Pb8djgCCO5IBqzKJ/Jwo8TRcHyHii0=
|
||||
github.com/getsentry/raven-go v0.2.0/go.mod h1:KungGk8q33+aIAZUIVWZDr2OfAEBsO49PX4NzFV5kcQ=
|
||||
github.com/glebarez/go-sqlite v1.21.2 h1:3a6LFC4sKahUunAmynQKLZceZCOzUthkRkEAl9gAXWo=
|
||||
github.com/glebarez/go-sqlite v1.21.2/go.mod h1:sfxdZyhQjTM2Wry3gVYWaW072Ri1WMdWJi0k6+3382k=
|
||||
github.com/go-check/check v0.0.0-20180628173108-788fd7840127 h1:0gkP6mzaMqkmpcJYCFOLkIBwI7xFExG03bbkOkCvUPI=
|
||||
github.com/go-check/check v0.0.0-20180628173108-788fd7840127/go.mod h1:9ES+weclKsC9YodN5RgxqK/VD9HM9JsCSh7rNhMZE98=
|
||||
github.com/go-ego/gse v1.0.2 h1:+27lYFPhQEhA9igtdOsJPRKYL/k3TwYsxBF5jr6KFv4=
|
||||
github.com/go-ego/gse v1.0.2/go.mod h1:Fy35G+q7VV7Et1zIKO8o/sW1kkugV3znXap/lF/11zc=
|
||||
github.com/go-logr/logr v1.2.2/go.mod h1:jdQByPbusPIv2/zmleS9BjJVeZ6kBagPoEUsqbVz/1A=
|
||||
github.com/go-logr/logr v1.4.3 h1:CjnDlHq8ikf6E492q6eKboGOC0T8CDaOvkHCIg8idEI=
|
||||
github.com/go-logr/logr v1.4.3/go.mod h1:9T104GzyrTigFIr8wt5mBrctHMim0Nb2HLGrmQ40KvY=
|
||||
github.com/go-logr/stdr v1.2.2 h1:hSWxHoqTgW2S2qGc0LTAI563KZ5YKYRhT3MFKZMbjag=
|
||||
github.com/go-logr/stdr v1.2.2/go.mod h1:mMo/vtBO5dYbehREoey6XUKy/eSumjCCveDpRre4VKE=
|
||||
github.com/gofrs/uuid v3.2.0+incompatible/go.mod h1:b2aQJv3Z4Fp6yNu3cdSllBxTCLRxnplIgP/c0N/04lM=
|
||||
github.com/gogf/gf/contrib/drivers/sqlite/v2 v2.10.2 h1:KLS68SWS2W749x7e+eCCOO3UD2Sbw+bIbLEPR8o1FXw=
|
||||
github.com/gogf/gf/contrib/drivers/sqlite/v2 v2.10.2/go.mod h1:uLcsu73PfpyhRc0Jq0gGAWQjN1tyGU9iBRrYgt/lu7g=
|
||||
github.com/gogf/gf/v2 v2.10.2 h1:46IO0Uc8e85/FqdftJFskfDejJLBL0JBnGS5qOftUu8=
|
||||
github.com/gogf/gf/v2 v2.10.2/go.mod h1:Svl1N+E8G/QshU2DUbh/3J/AJauqCgUnxHurXWR4Qx0=
|
||||
github.com/golang-jwt/jwt/v5 v5.3.1 h1:kYf81DTWFe7t+1VvL7eS+jKFVWaUnK9cB1qbwn63YCY=
|
||||
github.com/golang-jwt/jwt/v5 v5.3.1/go.mod h1:fxCRLWMO43lRc8nhHWY6LGqRcf+1gQWArsqaEUEa5bE=
|
||||
github.com/golang/protobuf v1.2.0/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U=
|
||||
github.com/google/go-cmp v0.6.0/go.mod h1:17dUlkBOakJ0+DkrSSNjCkIjxS6bF9zb3elmeNGIjoY=
|
||||
github.com/google/go-cmp v0.7.0 h1:wk8382ETsv4JYUZwIsn6YpYiWiBsYLSJiTsyBybVuN8=
|
||||
github.com/google/go-cmp v0.7.0/go.mod h1:pXiqmnSA92OHEEa9HXL2W4E7lf9JzCmGVUdgjX3N/iU=
|
||||
github.com/google/gofuzz v1.0.0/go.mod h1:dBl0BpW6vV/+mYPU4Po3pmUjxk6FQPldtuIdl/M65Eg=
|
||||
github.com/google/pprof v0.0.0-20250317173921-a4b03ec1a45e h1:ijClszYn+mADRFY17kjQEVQ1XRhq2/JR1M3sGqeJoxs=
|
||||
github.com/google/pprof v0.0.0-20250317173921-a4b03ec1a45e/go.mod h1:boTsfXsheKC2y+lKOCMpSfarhxDeIzfZG1jqGcPl3cA=
|
||||
github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0=
|
||||
github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
|
||||
github.com/goph/emperror v0.17.2 h1:yLapQcmEsO0ipe9p5TaN22djm3OFV/TfM/fcYP0/J18=
|
||||
github.com/goph/emperror v0.17.2/go.mod h1:+ZbQ+fUNO/6FNiUo0ujtMjhgad9Xa6fQL9KhH4LNHic=
|
||||
github.com/gopherjs/gopherjs v1.17.2 h1:fQnZVsXk8uxXIStYb0N4bGk7jeyTalG/wsZjQ25dO0g=
|
||||
github.com/gopherjs/gopherjs v1.17.2/go.mod h1:pRRIvn/QzFLrKfvEz3qUuEhtE/zLCWfreZ6J5gM2i+k=
|
||||
github.com/gorilla/websocket v1.5.3 h1:saDtZ6Pbx/0u+bgYQ3q96pZgCzfhKXGPqt7kZ72aNNg=
|
||||
github.com/gorilla/websocket v1.5.3/go.mod h1:YR8l580nyteQvAITg2hZ9XVh4b55+EU/adAjf1fMHhE=
|
||||
github.com/grokify/html-strip-tags-go v0.1.0 h1:03UrQLjAny8xci+R+qjCce/MYnpNXCtgzltlQbOBae4=
|
||||
github.com/grokify/html-strip-tags-go v0.1.0/go.mod h1:ZdzgfHEzAfz9X6Xe5eBLVblWIxXfYSQ40S/VKrAOGpc=
|
||||
github.com/hashicorp/golang-lru/v2 v2.0.7 h1:a+bsQ5rvGLjzHuww6tVxozPZFVghXaHOwFs4luLUK2k=
|
||||
github.com/hashicorp/golang-lru/v2 v2.0.7/go.mod h1:QeFd9opnmA6QUJc5vARoKUSoFhyfM2/ZepoAG6RGpeM=
|
||||
github.com/hhrutter/tiff v1.0.6 h1:p5I4Oi20jit3uWIBBaAoMDqrKztw/1JQCQC2TgqK1qU=
|
||||
github.com/hhrutter/tiff v1.0.6/go.mod h1:9+PDcnTBkMrJ8fWXkN1ZPv5ZNcKsFuTGVQU3ysaQbco=
|
||||
github.com/hpcloud/tail v1.0.0/go.mod h1:ab1qPbhIpdTxEkNHXyeSf5vhxWSCs/tWer42PpOxQnU=
|
||||
github.com/josharian/intern v1.0.0/go.mod h1:5DoeVV0s6jJacbCEi61lwdGj/aVlrQvzHFFd8Hwg//Y=
|
||||
github.com/json-iterator/go v1.1.12 h1:PV8peI4a0ysnczrg+LtxykD8LfKY9ML6u2jnxaEnrnM=
|
||||
github.com/json-iterator/go v1.1.12/go.mod h1:e30LSqwooZae/UwlEbR2852Gd8hjQvJoHmT4TnhNGBo=
|
||||
github.com/jtolds/gls v4.20.0+incompatible h1:xdiiI2gbIgH/gLH7ADydsJ1uDOEzR8yvV7C0MuV77Wo=
|
||||
github.com/jtolds/gls v4.20.0+incompatible/go.mod h1:QJZ7F/aHp+rZTRtaJ1ow/lLfFfVYBRgL+9YlvaHOwJU=
|
||||
github.com/kardianos/osext v0.0.0-20190222173326-2bc1f35cddc0/go.mod h1:1NbS8ALrpOvjt0rHPNLyCIeMtbizbir8U//inJ+zuB8=
|
||||
github.com/klauspost/cpuid/v2 v2.2.9 h1:66ze0taIn2H33fBvCkXuv9BmCwDfafmiIVpKV9kKGuY=
|
||||
github.com/klauspost/cpuid/v2 v2.2.9/go.mod h1:rqkxqrZ1EhYM9G+hXH7YdowN5R5RGN6NK4QwQ3WMXF8=
|
||||
github.com/konsorten/go-windows-terminal-sequences v1.0.1/go.mod h1:T0+1ngSBFLxvqU3pZ+m/2kptfBszLMUkC4ZK/EgS/cQ=
|
||||
github.com/kr/pretty v0.1.0/go.mod h1:dAy3ld7l9f0ibDNOQOHHMYYIIbhfbHSm3C4ZsoJORNo=
|
||||
github.com/kr/pretty v0.3.1 h1:flRD4NNwYAUpkphVc1HcthR4KEIFJ65n8Mw5qdRn3LE=
|
||||
github.com/kr/pretty v0.3.1/go.mod h1:hoEshYVHaxMs3cyo3Yncou5ZscifuDolrwPKZanG3xk=
|
||||
github.com/kr/pty v1.1.1/go.mod h1:pFQYn66WHrOpPYNljwOMqo10TkYh1fy3cYio2l3bCsQ=
|
||||
github.com/kr/text v0.1.0/go.mod h1:4Jbv+DJW3UT/LiOwJeYQe1efqtUx/iVham/4vfdArNI=
|
||||
github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY=
|
||||
github.com/kr/text v0.2.0/go.mod h1:eLer722TekiGuMkidMxC/pM04lWEeraHUUmBw8l2grE=
|
||||
github.com/magiconair/properties v1.8.10 h1:s31yESBquKXCV9a/ScB3ESkOjUYYv+X0rg8SYxI99mE=
|
||||
github.com/magiconair/properties v1.8.10/go.mod h1:Dhd985XPs7jluiymwWYZ0G4Z61jb3vdS329zhj2hYo0=
|
||||
github.com/mailru/easyjson v0.7.7 h1:UGYAvKxe3sBsEDzO8ZeWOSlIQfWFlxbzLZe7hwFURr0=
|
||||
github.com/mailru/easyjson v0.7.7/go.mod h1:xzfreul335JAWq5oZzymOObrkdz5UnU4kGfJJLY9Nlc=
|
||||
github.com/mattn/go-colorable v0.1.13 h1:fFA4WZxdEF4tXPZVKMLwD8oUnCTTo08duU7wxecdEvA=
|
||||
github.com/mattn/go-colorable v0.1.13/go.mod h1:7S9/ev0klgBDR4GtXTXX8a3vIGJpMovkB8vQcUbaXHg=
|
||||
github.com/mattn/go-isatty v0.0.16/go.mod h1:kYGgaQfpe5nmfYZH+SKPsOc2e4SrIfOl2e/yFXSvRLM=
|
||||
github.com/mattn/go-isatty v0.0.20 h1:xfD0iDuEKnDkl03q4limB+vH+GxLEtL/jb4xVJSWWEY=
|
||||
github.com/mattn/go-isatty v0.0.20/go.mod h1:W+V8PltTTMOvKvAeJH7IuucS94S2C6jfK/D7dTCTo3Y=
|
||||
github.com/mattn/go-runewidth v0.0.27 h1:Feg/Oou5zI/wnpgDF6omIU0OokC9GxLC/WRknhVlIR0=
|
||||
github.com/mattn/go-runewidth v0.0.27/go.mod h1:3qAiGCV4Koz/yuveO58qUefmUTRm8r0IGEXZ9jeHp/8=
|
||||
github.com/mgutz/ansi v0.0.0-20170206155736-9520e82c474b h1:j7+1HpAFS1zy5+Q4qx1fWh90gTKwiN4QCGoY9TWyyO4=
|
||||
github.com/mgutz/ansi v0.0.0-20170206155736-9520e82c474b/go.mod h1:01TrycV0kFyexm33Z7vhZRXopbI8J3TDReVlkTgMUxE=
|
||||
github.com/modern-go/concurrent v0.0.0-20180228061459-e0a39a4cb421/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q=
|
||||
github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd h1:TRLaZ9cD/w8PVh93nsPXa1VrQ6jlwL5oN8l14QlcNfg=
|
||||
github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q=
|
||||
github.com/modern-go/reflect2 v1.0.2 h1:xBagoLtFs94CBntxluKeaWgTMpvLxC4ur3nMaC9Gz0M=
|
||||
github.com/modern-go/reflect2 v1.0.2/go.mod h1:yWuevngMOJpCy52FWWMvUC8ws7m/LJsjYzDa0/r8luk=
|
||||
github.com/ncruces/go-strftime v1.0.0 h1:HMFp8mLCTPp341M/ZnA4qaf7ZlsbTc+miZjCLOFAw7w=
|
||||
github.com/ncruces/go-strftime v1.0.0/go.mod h1:Fwc5htZGVVkseilnfgOVb9mKy6w1naJmn9CehxcKcls=
|
||||
github.com/nikolalohinski/gonja v1.5.3 h1:GsA+EEaZDZPGJ8JtpeGN78jidhOlxeJROpqMT9fTj9c=
|
||||
github.com/nikolalohinski/gonja v1.5.3/go.mod h1:RmjwxNiXAEqcq1HeK5SSMmqFJvKOfTfXhkJv6YBtPa4=
|
||||
github.com/olekukonko/errors v1.1.0 h1:RNuGIh15QdDenh+hNvKrJkmxxjV4hcS50Db478Ou5sM=
|
||||
github.com/olekukonko/errors v1.1.0/go.mod h1:ppzxA5jBKcO1vIpCXQ9ZqgDh8iwODz6OXIGKU8r5m4Y=
|
||||
github.com/olekukonko/ll v0.0.9 h1:Y+1YqDfVkqMWuEQMclsF9HUR5+a82+dxJuL1HHSRpxI=
|
||||
github.com/olekukonko/ll v0.0.9/go.mod h1:En+sEW0JNETl26+K8eZ6/W4UQ7CYSrrgg/EdIYT2H8g=
|
||||
github.com/olekukonko/tablewriter v1.1.0 h1:N0LHrshF4T39KvI96fn6GT8HEjXRXYNDrDjKFDB7RIY=
|
||||
github.com/olekukonko/tablewriter v1.1.0/go.mod h1:5c+EBPeSqvXnLLgkm9isDdzR3wjfBkHR9Nhfp3NWrzo=
|
||||
github.com/onsi/ginkgo v1.6.0/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+WWjE=
|
||||
github.com/onsi/ginkgo v1.8.0/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+WWjE=
|
||||
github.com/onsi/gomega v1.5.0/go.mod h1:ex+gbHU/CVuBBDIJjb2X0qEXbFg53c61hWP/1CpauHY=
|
||||
github.com/pdfcpu/pdfcpu v0.14.0 h1:KRC7JMWiFZD4uIjYwTYR9txG9vYGQj+NdjTFkfUKd14=
|
||||
github.com/pdfcpu/pdfcpu v0.14.0/go.mod h1:NhG6T7b2EEdToXGD5hj8rmXBWSLCjgljCk5c0H6U9x8=
|
||||
github.com/pelletier/go-toml/v2 v2.0.9 h1:uH2qQXheeefCCkuBBSLi7jCiSmj3VRh2+Goq2N7Xxu0=
|
||||
github.com/pelletier/go-toml/v2 v2.0.9/go.mod h1:tJU2Z3ZkXwnxa4DPO899bsyIoywizdUvyaeZurnPPDc=
|
||||
github.com/pkg/errors v0.8.0/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
|
||||
github.com/pkg/errors v0.9.1 h1:FEBLx1zS214owpjy7qsBeixbURkuhQAwrK5UwLGTwt4=
|
||||
github.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
|
||||
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
|
||||
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
|
||||
github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec h1:W09IVJc94icq4NjY3clb7Lk8O1qJ8BdBEF8z0ibU0rE=
|
||||
github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec/go.mod h1:qqbHyh8v60DhA7CoWK5oRCqLrMHRGoxYCSS9EjAz6Eo=
|
||||
github.com/rivo/uniseg v0.2.0 h1:S1pD9weZBuJdFmowNwbpi7BJ8TNftyUImj/0WQi72jY=
|
||||
github.com/rivo/uniseg v0.2.0/go.mod h1:J6wj4VEh+S6ZtnVlnTBMWIodfgj8LQOQFoIToxlJtxc=
|
||||
github.com/rogpeppe/go-internal v1.13.1 h1:KvO1DLK/DRN07sQ1LQKScxyZJuNnedQ5/wKSR38lUII=
|
||||
github.com/rogpeppe/go-internal v1.13.1/go.mod h1:uMEvuHeurkdAXX61udpOXGD/AzZDWNMNyH2VO9fmH0o=
|
||||
github.com/rollbar/rollbar-go v1.0.2/go.mod h1:AcFs5f0I+c71bpHlXNNDbOWJiKwjFDtISeXco0L5PKQ=
|
||||
github.com/sirupsen/logrus v1.2.0/go.mod h1:LxeOpSwHxABJmUn/MG1IvRgCAasNZTLOkJPxbbu5VWo=
|
||||
github.com/sirupsen/logrus v1.9.3 h1:dueUQJ1C2q9oE3F7wvmSGAaVtTmUizReu6fjN8uqzbQ=
|
||||
github.com/sirupsen/logrus v1.9.3/go.mod h1:naHLuLoDiP4jHNo9R0sCBMtWGeIprob74mVsIT4qYEQ=
|
||||
github.com/slongfield/pyfmt v0.0.0-20220222012616-ea85ff4c361f h1:Z2cODYsUxQPofhpYRMQVwWz4yUVpHF+vPi+eUdruUYI=
|
||||
github.com/slongfield/pyfmt v0.0.0-20220222012616-ea85ff4c361f/go.mod h1:JqzWyvTuI2X4+9wOHmKSQCYxybB/8j6Ko43qVmXDuZg=
|
||||
github.com/smarty/assertions v1.15.0 h1:cR//PqUBUiQRakZWqBiFFQ9wb8emQGDb0HeGdqGByCY=
|
||||
github.com/smarty/assertions v1.15.0/go.mod h1:yABtdzeQs6l1brC900WlRNwj6ZR55d7B+E8C6HtKdec=
|
||||
github.com/smartystreets/goconvey v1.8.1 h1:qGjIddxOk4grTu9JPOU31tVfq3cNdBlNa5sSznIX1xY=
|
||||
github.com/smartystreets/goconvey v1.8.1/go.mod h1:+/u4qLyY6x1jReYOp7GOM2FSt8aP9CzCZL03bI28W60=
|
||||
github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
|
||||
github.com/stretchr/objx v0.1.1/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
|
||||
github.com/stretchr/objx v0.4.0/go.mod h1:YvHI0jy2hoMjB+UWwv71VJQ9isScKT/TqJzVSSt89Yw=
|
||||
github.com/stretchr/objx v0.5.0/go.mod h1:Yh+to48EsGEfYuaHDzXPcE3xhTkx73EhmCGUpEOglKo=
|
||||
github.com/stretchr/objx v0.5.2/go.mod h1:FRsXN1f5AsAjCGJKqEizvkpNtU+EGNCLh3NxZ/8L+MA=
|
||||
github.com/stretchr/testify v1.2.2/go.mod h1:a8OnRcib4nhh0OaRAV+Yts87kKdq0PP7pXfy6kDkUVs=
|
||||
github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI=
|
||||
github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
|
||||
github.com/stretchr/testify v1.7.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
|
||||
github.com/stretchr/testify v1.8.0/go.mod h1:yNjHg4UonilssWZ8iaSj1OCr/vHnekPRkoO+kdMU+MU=
|
||||
github.com/stretchr/testify v1.8.4/go.mod h1:sz/lmYIOXD/1dqDmKjjqLyZ2RngseejIcXlSw2iwfAo=
|
||||
github.com/stretchr/testify v1.10.0/go.mod h1:r2ic/lqez/lEtzL7wO/rwa5dbSLXVDPFyf8C91i36aY=
|
||||
github.com/stretchr/testify v1.11.1 h1:7s2iGBzp5EwR7/aIZr8ao5+dra3wiQyKjjFuvgVKu7U=
|
||||
github.com/stretchr/testify v1.11.1/go.mod h1:wZwfW3scLgRK+23gO65QZefKpKQRnfz6sD981Nm4B6U=
|
||||
github.com/twitchyliquid64/golang-asm v0.15.1 h1:SU5vSMR7hnwNxj24w34ZyCi/FmDZTkS4MhqMhdFk5YI=
|
||||
github.com/twitchyliquid64/golang-asm v0.15.1/go.mod h1:a1lVb/DtPvCB8fslRZhAngC2+aY1QWCk3Cedj/Gdt08=
|
||||
github.com/vcaesar/cedar v0.30.0 h1:9fSDpM7FTjjUdPiBUUa0MWYMRGSEcqgFXvppZcZ4d7Y=
|
||||
github.com/vcaesar/cedar v0.30.0/go.mod h1:lyuGvALuZZDPNXwpzv/9LyxW+8Y6faN7zauFezNsnik=
|
||||
github.com/vcaesar/tt v0.20.1 h1:D/jUeeVCNbq3ad8M7hhtB3J9x5RZ6I1n1eZ0BJp7M+4=
|
||||
github.com/vcaesar/tt v0.20.1/go.mod h1:cH2+AwGAJm19Wa6xvEa+0r+sXDJBT0QgNQey6mwqLeU=
|
||||
github.com/wk8/go-ordered-map/v2 v2.1.8 h1:5h/BUHu93oj4gIdvHHHGsScSTMijfx5PeYkE/fJgbpc=
|
||||
github.com/wk8/go-ordered-map/v2 v2.1.8/go.mod h1:5nJHM5DyteebpVlHnWMV0rPz6Zp7+xBAnxjb1X5vnTw=
|
||||
github.com/x-cray/logrus-prefixed-formatter v0.5.2 h1:00txxvfBM9muc0jiLIEAkAcIMJzfthRT6usrui8uGmg=
|
||||
github.com/x-cray/logrus-prefixed-formatter v0.5.2/go.mod h1:2duySbKsL6M18s5GU7VPsoEPHyzalCE06qoARUCeBBE=
|
||||
github.com/yargevad/filepathx v1.0.0 h1:SYcT+N3tYGi+NvazubCNlvgIPbzAk7i7y2dwg3I5FYc=
|
||||
github.com/yargevad/filepathx v1.0.0/go.mod h1:BprfX/gpYNJHJfc35GjRRpVcwWXS89gGulUIU5tK3tA=
|
||||
github.com/yuin/goldmark v1.4.13/go.mod h1:6yULJ656Px+3vBD8DxQVa3kxgyrAnzto9xy5taEt/CY=
|
||||
go.opentelemetry.io/auto/sdk v1.1.0 h1:cH53jehLUN6UFLY71z+NDOiNJqDdPRaXzTel0sJySYA=
|
||||
go.opentelemetry.io/auto/sdk v1.1.0/go.mod h1:3wSPjt5PWp2RhlCcmmOial7AvC4DQqZb7a7wCow3W8A=
|
||||
go.opentelemetry.io/otel v1.38.0 h1:RkfdswUDRimDg0m2Az18RKOsnI8UDzppJAtj01/Ymk8=
|
||||
go.opentelemetry.io/otel v1.38.0/go.mod h1:zcmtmQ1+YmQM9wrNsTGV/q/uyusom3P8RxwExxkZhjM=
|
||||
go.opentelemetry.io/otel/metric v1.38.0 h1:Kl6lzIYGAh5M159u9NgiRkmoMKjvbsKtYRwgfrA6WpA=
|
||||
go.opentelemetry.io/otel/metric v1.38.0/go.mod h1:kB5n/QoRM8YwmUahxvI3bO34eVtQf2i4utNVLr9gEmI=
|
||||
go.opentelemetry.io/otel/sdk v1.38.0 h1:l48sr5YbNf2hpCUj/FoGhW9yDkl+Ma+LrVl8qaM5b+E=
|
||||
go.opentelemetry.io/otel/sdk v1.38.0/go.mod h1:ghmNdGlVemJI3+ZB5iDEuk4bWA3GkTpW+DOoZMYBVVg=
|
||||
go.opentelemetry.io/otel/sdk/metric v1.38.0 h1:aSH66iL0aZqo//xXzQLYozmWrXxyFkBJ6qT5wthqPoM=
|
||||
go.opentelemetry.io/otel/sdk/metric v1.38.0/go.mod h1:dg9PBnW9XdQ1Hd6ZnRz689CbtrUp0wMMs9iPcgT9EZA=
|
||||
go.opentelemetry.io/otel/trace v1.38.0 h1:Fxk5bKrDZJUH+AMyyIXGcFAPah0oRcT+LuNtJrmcNLE=
|
||||
go.opentelemetry.io/otel/trace v1.38.0/go.mod h1:j1P9ivuFsTceSWe1oY+EeW3sc+Pp42sO++GHkg4wwhs=
|
||||
go.uber.org/goleak v1.3.0 h1:2K3zAYmnTNqV73imy9J1T3WC+gmCePx2hEGkimedGto=
|
||||
go.uber.org/goleak v1.3.0/go.mod h1:CoHD4mav9JJNrW/WLlf7HGZPjdw8EucARQHekz1X6bE=
|
||||
go.uber.org/mock v0.4.0 h1:VcM4ZOtdbR4f6VXfiOpwpVJDL6lCReaZ6mw31wqh7KU=
|
||||
go.uber.org/mock v0.4.0/go.mod h1:a6FSlNadKUHUa9IP5Vyt1zh4fC7uAwxMutEAscFbkZc=
|
||||
go.yaml.in/yaml/v3 v3.0.5 h1:N6y/pJk8buWs9NY5ERU2HSMfm+IuD/OtfdAnq6kESPw=
|
||||
go.yaml.in/yaml/v3 v3.0.5/go.mod h1:HVTZu1O7/Vkt2N+BFy8Zza+lnLsABggaTM2ZpNIGuKg=
|
||||
golang.org/x/arch v0.11.0 h1:KXV8WWKCXm6tRpLirl2szsO5j/oOODwZf4hATmGVNs4=
|
||||
golang.org/x/arch v0.11.0/go.mod h1:FEVrYAQjsQXMVJ1nsMoVVXPZg6p2JE2mx8psSWTDQys=
|
||||
golang.org/x/crypto v0.0.0-20180904163835-0709b304e793/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4=
|
||||
golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
|
||||
golang.org/x/crypto v0.0.0-20210921155107-089bfa567519/go.mod h1:GvvjBRRGRdwPK5ydBHafDWAxML/pGHZbMvKqRZ5+Abc=
|
||||
golang.org/x/crypto v0.13.0/go.mod h1:y6Z2r+Rw4iayiXXAIxJIDAJ1zMW4yaTpebo8fPOliYc=
|
||||
golang.org/x/crypto v0.19.0/go.mod h1:Iy9bg/ha4yyC70EfRS8jz+B6ybOBKMaSxLj6P6oBDfU=
|
||||
golang.org/x/crypto v0.23.0/go.mod h1:CKFgDieR+mRhux2Lsu27y0fO304Db0wZe70UKqHu0v8=
|
||||
golang.org/x/crypto v0.31.0/go.mod h1:kDsLvtWBEx7MV9tJOj9bnXsPbxwJQ6csT/x4KIN4Ssk=
|
||||
golang.org/x/crypto v0.54.0 h1:YLIA59K4fiNzHzjnZt2tUJQjQtUWfWbeHBqKtk3eScw=
|
||||
golang.org/x/crypto v0.54.0/go.mod h1:KWL8ny2AZdGR2cWmzeHrp2azQPGogOv+HeQaVEXC2dk=
|
||||
golang.org/x/exp v0.0.0-20251023183803-a4bb9ffd2546 h1:mgKeJMpvi0yx/sU5GsxQ7p6s2wtOnGAHZWCHUM4KGzY=
|
||||
golang.org/x/exp v0.0.0-20251023183803-a4bb9ffd2546/go.mod h1:j/pmGrbnkbPtQfxEe5D0VQhZC6qKbfKifgD0oM7sR70=
|
||||
golang.org/x/image v0.44.0 h1:+tDekMZED9+LrtB3G5xzRggpVh9CARjZqROla3R3R+I=
|
||||
golang.org/x/image v0.44.0/go.mod h1:V8K3KE9KKKE+pLpQDOeN18w9oacNSvy1tDOirTu4xtY=
|
||||
golang.org/x/mod v0.6.0-dev.0.20220419223038-86c51ed26bb4/go.mod h1:jJ57K6gSWd91VN4djpZkiMVwK6gcyfeH4XE8wZrZaV4=
|
||||
golang.org/x/mod v0.8.0/go.mod h1:iBbtSCu2XBx23ZKBPSOrRkjjQPZFPuis4dIYUhu/chs=
|
||||
golang.org/x/mod v0.12.0/go.mod h1:iBbtSCu2XBx23ZKBPSOrRkjjQPZFPuis4dIYUhu/chs=
|
||||
golang.org/x/mod v0.15.0/go.mod h1:hTbmBsO62+eylJbnUtE2MGJUyE7QWk4xUqPFrRgJ+7c=
|
||||
golang.org/x/mod v0.17.0/go.mod h1:hTbmBsO62+eylJbnUtE2MGJUyE7QWk4xUqPFrRgJ+7c=
|
||||
golang.org/x/mod v0.37.0 h1:vF1DjpVEshcIqoEaauuHebaLk1O1forxjxBaVn884JQ=
|
||||
golang.org/x/mod v0.37.0/go.mod h1:m8S8VeM9r4dzDwjrKO0a1sZP3YjeMamRRlD+fmR2Q/0=
|
||||
golang.org/x/net v0.0.0-20180906233101-161cd47e91fd/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
|
||||
golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
|
||||
golang.org/x/net v0.0.0-20210226172049-e18ecbb05110/go.mod h1:m0MpNAwzfU5UDzcl9v0D8zg8gWTRqZa9RBIspLL5mdg=
|
||||
golang.org/x/net v0.0.0-20220722155237-a158d28d115b/go.mod h1:XRhObCWvk6IyKnWLug+ECip1KBveYUHfp+8e9klMJ9c=
|
||||
golang.org/x/net v0.6.0/go.mod h1:2Tu9+aMcznHK/AK1HMvgo6xiTLG5rD5rZLDS+rp2Bjs=
|
||||
golang.org/x/net v0.10.0/go.mod h1:0qNGK6F8kojg2nk9dLZ2mShWaEBan6FAoqfSigmmuDg=
|
||||
golang.org/x/net v0.15.0/go.mod h1:idbUs1IY1+zTqbi8yxTbhexhEEk5ur9LInksu6HrEpk=
|
||||
golang.org/x/net v0.21.0/go.mod h1:bIjVDfnllIU7BJ2DNgfnXvpSvtn8VRwhlsaeUTyUS44=
|
||||
golang.org/x/net v0.25.0/go.mod h1:JkAGAh7GEvH74S6FOH42FLoXpXbE/aqXSrIQjXgsiwM=
|
||||
golang.org/x/net v0.33.0/go.mod h1:HXLR5J+9DxmrqMwG9qjGCxZ+zKXxBru04zlTvWlWuN4=
|
||||
golang.org/x/net v0.56.0 h1:Rw8j/hFzGvJUZwNBXnAtf5sVDVt+65SK2C7IxCxZt5o=
|
||||
golang.org/x/net v0.56.0/go.mod h1:D3Ku6r+V6JROoZK144D2XfMHFcMq/0zSfLelVTCFKec=
|
||||
golang.org/x/sync v0.0.0-20180314180146-1d60e4601c6f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
|
||||
golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
|
||||
golang.org/x/sync v0.0.0-20220722155255-886fb9371eb4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
|
||||
golang.org/x/sync v0.1.0/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
|
||||
golang.org/x/sync v0.3.0/go.mod h1:FU7BRWz2tNW+3quACPkgCx/L+uEAv1htQ0V83Z9Rj+Y=
|
||||
golang.org/x/sync v0.6.0/go.mod h1:Czt+wKu1gCyEFDUtn0jG5QVvpJ6rzVqr5aXyt9drQfk=
|
||||
golang.org/x/sync v0.7.0/go.mod h1:Czt+wKu1gCyEFDUtn0jG5QVvpJ6rzVqr5aXyt9drQfk=
|
||||
golang.org/x/sync v0.10.0/go.mod h1:Czt+wKu1gCyEFDUtn0jG5QVvpJ6rzVqr5aXyt9drQfk=
|
||||
golang.org/x/sync v0.22.0 h1:SZjpbeLmrCk4xhRSZFNZW5gFUeCeFgjekvI/+gfScek=
|
||||
golang.org/x/sync v0.22.0/go.mod h1:9xrNwdLfx4jkKbNva9FpL6vEN7evnE43NNNJQ2LF3+0=
|
||||
golang.org/x/sys v0.0.0-20180905080454-ebe1bf3edb33/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
|
||||
golang.org/x/sys v0.0.0-20180909124046-d0be0721c37e/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
|
||||
golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
|
||||
golang.org/x/sys v0.0.0-20201119102817-f84b799fce68/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
|
||||
golang.org/x/sys v0.0.0-20210615035016-665e8c7367d1/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
|
||||
golang.org/x/sys v0.0.0-20220520151302-bc2c85ada10a/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
|
||||
golang.org/x/sys v0.0.0-20220715151400-c0bba94af5f8/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
|
||||
golang.org/x/sys v0.0.0-20220722155257-8c9f86f7a55f/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
|
||||
golang.org/x/sys v0.0.0-20220811171246-fbc7d0a398ab/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
|
||||
golang.org/x/sys v0.5.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
|
||||
golang.org/x/sys v0.6.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
|
||||
golang.org/x/sys v0.8.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
|
||||
golang.org/x/sys v0.12.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
|
||||
golang.org/x/sys v0.17.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
|
||||
golang.org/x/sys v0.20.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
|
||||
golang.org/x/sys v0.28.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
|
||||
golang.org/x/sys v0.47.0 h1:o7XGOvZQCADBQQ4Y7VNq2dRWQR7JmOUW8Kxx4ZsNgWs=
|
||||
golang.org/x/sys v0.47.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw=
|
||||
golang.org/x/telemetry v0.0.0-20240228155512-f48c80bd79b2/go.mod h1:TeRTkGYfJXctD9OcfyVLyj2J3IxLnKwHJR8f4D8a3YE=
|
||||
golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo=
|
||||
golang.org/x/term v0.0.0-20210927222741-03fcf44c2211/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8=
|
||||
golang.org/x/term v0.5.0/go.mod h1:jMB1sMXY+tzblOD4FWmEbocvup2/aLOaQEp7JmGp78k=
|
||||
golang.org/x/term v0.8.0/go.mod h1:xPskH00ivmX89bAKVGSKKtLOWNx2+17Eiy94tnKShWo=
|
||||
golang.org/x/term v0.12.0/go.mod h1:owVbMEjm3cBLCHdkQu9b1opXd4ETQWc3BhuQGKgXgvU=
|
||||
golang.org/x/term v0.17.0/go.mod h1:lLRBjIVuehSbZlaOtGMbcMncT+aqLLLmKrsjNrUguwk=
|
||||
golang.org/x/term v0.20.0/go.mod h1:8UkIAJTvZgivsXaD6/pH6U9ecQzZ45awqEOzuCvwpFY=
|
||||
golang.org/x/term v0.27.0/go.mod h1:iMsnZpn0cago0GOrHO2+Y7u7JPn5AylBrcoWkElMTSM=
|
||||
golang.org/x/term v0.45.0 h1:NwWyBmoJCbfTHpxrWoZ9C6/VxOf7ic219I8xZZFdrf0=
|
||||
golang.org/x/term v0.45.0/go.mod h1:9aqxs0blBcrm/n0L9QW0aRVD+ktan8ssZromtqJC43w=
|
||||
golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
|
||||
golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
|
||||
golang.org/x/text v0.3.7/go.mod h1:u+2+/6zg+i71rQMx5EYifcz6MCKuco9NR6JIITiCfzQ=
|
||||
golang.org/x/text v0.7.0/go.mod h1:mrYo+phRRbMaCq/xk9113O4dZlRixOauAjOtrjsXDZ8=
|
||||
golang.org/x/text v0.9.0/go.mod h1:e1OnstbJyHTd6l/uOt8jFFHp6TRDWZR/bV3emEE/zU8=
|
||||
golang.org/x/text v0.13.0/go.mod h1:TvPlkZtksWOMsz7fbANvkp4WM8x/WCo/om8BMLbz+aE=
|
||||
golang.org/x/text v0.14.0/go.mod h1:18ZOQIKpY8NJVqYksKHtTdi31H5itFRjB5/qKTNYzSU=
|
||||
golang.org/x/text v0.15.0/go.mod h1:18ZOQIKpY8NJVqYksKHtTdi31H5itFRjB5/qKTNYzSU=
|
||||
golang.org/x/text v0.21.0/go.mod h1:4IBbMaMmOPCJ8SecivzSH54+73PCFmPWxNTLm+vZkEQ=
|
||||
golang.org/x/text v0.40.0 h1:Ub2Z6/xjgF1WrYQz2nuITOEegKFtiIy+rieRJ5lHZKs=
|
||||
golang.org/x/text v0.40.0/go.mod h1:hpnzDAfGV753zIKo+wk3u1bVKCGPbrnF7+7LBF/UHVY=
|
||||
golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
|
||||
golang.org/x/tools v0.0.0-20191119224855-298f0cb1881e/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
|
||||
golang.org/x/tools v0.1.12/go.mod h1:hNGJHUnrk76NpqgfD5Aqm5Crs+Hm0VOH/i9J2+nxYbc=
|
||||
golang.org/x/tools v0.6.0/go.mod h1:Xwgl3UAJ/d3gWutnCtw505GrjyAbvKui8lOU390QaIU=
|
||||
golang.org/x/tools v0.13.0/go.mod h1:HvlwmtVNQAhOuCjW7xxvovg8wbNq7LwfXh/k7wXUl58=
|
||||
golang.org/x/tools v0.21.1-0.20240508182429-e35e4ccd0d2d/go.mod h1:aiJjzUbINMkxbQROHiO6hDPo2LHcIPhhQsa9DLh0yGk=
|
||||
golang.org/x/tools v0.47.0 h1:7Kn5x/d1svx/PzryTsqeoZN4TZwqeH5pGWjefhLi/1Q=
|
||||
golang.org/x/tools v0.47.0/go.mod h1:dFHnyTvFWY212G+h7ZY4Vsp/K3U4/7W9TyVaAul8uCA=
|
||||
golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
|
||||
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
|
||||
gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c h1:Hei/4ADfdWqJk1ZMxUNpqntNwaWcugrBjAiHlqqRiVk=
|
||||
gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c/go.mod h1:JHkPIbrfpd72SG/EVd6muEfDQjcINNoR0C8j2r3qZ4Q=
|
||||
gopkg.in/fsnotify.v1 v1.4.7/go.mod h1:Tz8NjZHkW78fSQdbUxIjBTcgA1z1m8ZHf0WmKUhAMys=
|
||||
gopkg.in/tomb.v1 v1.0.0-20141024135613-dd632973f1e7/go.mod h1:dt/ZhP58zS4L8KSrWDmTeBkI65Dw0HsyUHuEVlX15mw=
|
||||
gopkg.in/yaml.v2 v2.2.1/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
|
||||
gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
|
||||
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
|
||||
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
|
||||
modernc.org/cc/v4 v4.27.1 h1:9W30zRlYrefrDV2JE2O8VDtJ1yPGownxciz5rrbQZis=
|
||||
modernc.org/cc/v4 v4.27.1/go.mod h1:uVtb5OGqUKpoLWhqwNQo/8LwvoiEBLvZXIQ/SmO6mL0=
|
||||
modernc.org/ccgo/v4 v4.32.0 h1:hjG66bI/kqIPX1b2yT6fr/jt+QedtP2fqojG2VrFuVw=
|
||||
modernc.org/ccgo/v4 v4.32.0/go.mod h1:6F08EBCx5uQc38kMGl+0Nm0oWczoo1c7cgpzEry7Uc0=
|
||||
modernc.org/fileutil v1.4.0 h1:j6ZzNTftVS054gi281TyLjHPp6CPHr2KCxEXjEbD6SM=
|
||||
modernc.org/fileutil v1.4.0/go.mod h1:EqdKFDxiByqxLk8ozOxObDSfcVOv/54xDs/DUHdvCUU=
|
||||
modernc.org/gc/v2 v2.6.5 h1:nyqdV8q46KvTpZlsw66kWqwXRHdjIlJOhG6kxiV/9xI=
|
||||
modernc.org/gc/v2 v2.6.5/go.mod h1:YgIahr1ypgfe7chRuJi2gD7DBQiKSLMPgBQe9oIiito=
|
||||
modernc.org/gc/v3 v3.1.2 h1:ZtDCnhonXSZexk/AYsegNRV1lJGgaNZJuKjJSWKyEqo=
|
||||
modernc.org/gc/v3 v3.1.2/go.mod h1:HFK/6AGESC7Ex+EZJhJ2Gni6cTaYpSMmU/cT9RmlfYY=
|
||||
modernc.org/goabi0 v0.2.0 h1:HvEowk7LxcPd0eq6mVOAEMai46V+i7Jrj13t4AzuNks=
|
||||
modernc.org/goabi0 v0.2.0/go.mod h1:CEFRnnJhKvWT1c1JTI3Avm+tgOWbkOu5oPA8eH8LnMI=
|
||||
modernc.org/libc v1.70.0 h1:U58NawXqXbgpZ/dcdS9kMshu08aiA6b7gusEusqzNkw=
|
||||
modernc.org/libc v1.70.0/go.mod h1:OVmxFGP1CI/Z4L3E0Q3Mf1PDE0BucwMkcXjjLntvHJo=
|
||||
modernc.org/mathutil v1.7.1 h1:GCZVGXdaN8gTqB1Mf/usp1Y/hSqgI2vAGGP4jZMCxOU=
|
||||
modernc.org/mathutil v1.7.1/go.mod h1:4p5IwJITfppl0G4sUEDtCr4DthTaT47/N3aT6MhfgJg=
|
||||
modernc.org/memory v1.11.0 h1:o4QC8aMQzmcwCK3t3Ux/ZHmwFPzE6hf2Y5LbkRs+hbI=
|
||||
modernc.org/memory v1.11.0/go.mod h1:/JP4VbVC+K5sU2wZi9bHoq2MAkCnrt2r98UGeSK7Mjw=
|
||||
modernc.org/opt v0.1.4 h1:2kNGMRiUjrp4LcaPuLY2PzUfqM/w9N23quVwhKt5Qm8=
|
||||
modernc.org/opt v0.1.4/go.mod h1:03fq9lsNfvkYSfxrfUhZCWPk1lm4cq4N+Bh//bEtgns=
|
||||
modernc.org/sortutil v1.2.1 h1:+xyoGf15mM3NMlPDnFqrteY07klSFxLElE2PVuWIJ7w=
|
||||
modernc.org/sortutil v1.2.1/go.mod h1:7ZI3a3REbai7gzCLcotuw9AC4VZVpYMjDzETGsSMqJE=
|
||||
modernc.org/sqlite v1.47.0 h1:R1XyaNpoW4Et9yly+I2EeX7pBza/w+pmYee/0HJDyKk=
|
||||
modernc.org/sqlite v1.47.0/go.mod h1:hWjRO6Tj/5Ik8ieqxQybiEOUXy0NJFNp2tpvVpKlvig=
|
||||
modernc.org/strutil v1.2.1 h1:UneZBkQA+DX2Rp35KcM69cSsNES9ly8mQWD71HKlOA0=
|
||||
modernc.org/strutil v1.2.1/go.mod h1:EHkiggD70koQxjVdSBM3JKM7k6L0FbGE5eymy9i3B9A=
|
||||
modernc.org/token v1.1.0 h1:Xl7Ap9dKaEs5kLoOQeQmPWevfnk/DM5qcLcYlA8ys6Y=
|
||||
modernc.org/token v1.1.0/go.mod h1:UGzOrNV1mAFSEB63lOFHIpNRUVMvYTc6yu1SMY/XTDM=
|
||||
@@ -0,0 +1,17 @@
|
||||
package consts
|
||||
|
||||
// 向量与检索参数
|
||||
const (
|
||||
DefaultEmbeddingDim = 1024 // embedding 默认维度（未配置时）
|
||||
VectorTopK = 20 // 向量检索召回数
|
||||
FtsTopK = 20 // 全文检索召回数
|
||||
HybridTopK = 10 // 混合检索融合后返回数
|
||||
RrfK = 60 // RRF 融合常数
|
||||
|
||||
MaxChunkSize = 800 // 分块最大字数
|
||||
ChunkOverlap = 100 // 分块重叠字数
|
||||
|
||||
ParsePollIntervalSeconds = 3 // 解析任务轮询间隔
|
||||
|
||||
EmbedBatchSize = 16 // 单次向量化请求的文本批量
|
||||
)
|
||||
@@ -0,0 +1,14 @@
|
||||
package consts
|
||||
|
||||
// 模型类型
|
||||
const (
|
||||
ModelTypeChat = "chat"
|
||||
ModelTypeEmbedding = "embedding"
|
||||
)
|
||||
|
||||
// system_config 预置键
|
||||
const (
|
||||
CfgKeyAccessToken = "access_token"
|
||||
CfgKeyDefaultChatModel = "default_chat_model"
|
||||
CfgKeyDefaultDataset = "default_dataset"
|
||||
)
|
||||
@@ -0,0 +1,30 @@
|
||||
package consts
|
||||
|
||||
// 文档状态
|
||||
const (
|
||||
DocumentStatusPending = 0 // 待处理
|
||||
DocumentStatusParsing = 1 // 处理中
|
||||
DocumentStatusDone = 2 // 完成
|
||||
DocumentStatusFailed = 3 // 失败
|
||||
)
|
||||
|
||||
// 解析任务状态
|
||||
const (
|
||||
TaskStatusPending = 0 // 待处理
|
||||
TaskStatusRunning = 1 // 处理中
|
||||
TaskStatusDone = 2 // 完成
|
||||
TaskStatusFailed = 3 // 失败
|
||||
)
|
||||
|
||||
// 任务类型
|
||||
const (
|
||||
TaskTypeParse = "parse" // 解析+向量化
|
||||
TaskTypeReembed = "reembed" // 重新向量化（切换 embedding 模型后）
|
||||
)
|
||||
|
||||
// 消息角色
|
||||
const (
|
||||
MsgRoleUser = "user"
|
||||
MsgRoleAssistant = "assistant"
|
||||
MsgRoleSystem = "system"
|
||||
)
|
||||
@@ -0,0 +1,23 @@
|
||||
package consts
|
||||
|
||||
const (
|
||||
TableNameSystemConfig = "system_config"
|
||||
TableNameModelConfig = "model_config"
|
||||
TableNameDataset = "kb_dataset"
|
||||
TableNameDocument = "kb_document"
|
||||
TableNameChunk = "kb_chunk"
|
||||
TableNameChunkVec = "kb_chunk_vec"
|
||||
TableNameChunkFts = "kb_chunk_fts"
|
||||
TableNameParseTask = "kb_parse_task"
|
||||
TableNameConversation = "chat_conversation"
|
||||
TableNameMessage = "chat_message"
|
||||
TableNameKgEntity = "kg_entity"
|
||||
TableNameKgRelation = "kg_relation"
|
||||
)
|
||||
|
||||
// 数据库组：默认组(default)=business.db、system=system.db、chat=chat.db
|
||||
const (
|
||||
DbGroupDefault = "" // 默认组，对应 business.db
|
||||
DbGroupSystem = "system" // system.db
|
||||
DbGroupChat = "chat" // chat.db
|
||||
)
|
||||
@@ -0,0 +1,32 @@
|
||||
package controller
|
||||
|
||||
import (
|
||||
"context"
|
||||
|
||||
"rag-local/kb/model/dto"
|
||||
"rag-local/kb/service"
|
||||
)
|
||||
|
||||
type chunk struct{}
|
||||
|
||||
var Chunk = &chunk{}
|
||||
|
||||
func (c *chunk) List(ctx context.Context, req *dto.ListChunkReq) (*dto.ListChunkRes, error) {
|
||||
list, total, err := service.ChunkService.List(ctx, req.DocumentId, req.Page, req.PageSize)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return &dto.ListChunkRes{
|
||||
List: list,
|
||||
Total: total,
|
||||
Page: req.Page,
|
||||
PageSize: req.PageSize,
|
||||
}, nil
|
||||
}
|
||||
|
||||
func (c *chunk) Update(ctx context.Context, req *dto.UpdateChunkReq) (*dto.UpdateChunkRes, error) {
|
||||
if err := service.ChunkService.Update(ctx, req.Id, req.Content); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return &dto.UpdateChunkRes{}, nil
|
||||
}
|
||||
@@ -0,0 +1,40 @@
|
||||
package controller
|
||||
|
||||
import (
|
||||
"context"
|
||||
|
||||
"rag-local/kb/model/dto"
|
||||
"rag-local/kb/model/entity"
|
||||
"rag-local/kb/service"
|
||||
)
|
||||
|
||||
type conversation struct{}
|
||||
|
||||
var Conversation = &conversation{}
|
||||
|
||||
func (c *conversation) List(ctx context.Context, req *dto.ListConversationReq) (*dto.ListConversationRes, error) {
|
||||
list, err := service.ConversationService.List(ctx)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return &dto.ListConversationRes{List: list}, nil
|
||||
}
|
||||
|
||||
func (c *conversation) Save(ctx context.Context, req *dto.SaveConversationReq) (*dto.SaveConversationRes, error) {
|
||||
id, err := service.ConversationService.Save(ctx, &entity.Conversation{
|
||||
Id: req.Id,
|
||||
DatasetId: req.DatasetId,
|
||||
Title: req.Title,
|
||||
})
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return &dto.SaveConversationRes{Id: id}, nil
|
||||
}
|
||||
|
||||
func (c *conversation) Delete(ctx context.Context, req *dto.DeleteConversationReq) (*dto.DeleteConversationRes, error) {
|
||||
if err := service.ConversationService.Delete(ctx, req.Id); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return &dto.DeleteConversationRes{}, nil
|
||||
}
|
||||
@@ -0,0 +1,42 @@
|
||||
package controller
|
||||
|
||||
import (
|
||||
"context"
|
||||
|
||||
"rag-local/kb/model/dto"
|
||||
"rag-local/kb/model/entity"
|
||||
"rag-local/kb/service"
|
||||
)
|
||||
|
||||
type dataset struct{}
|
||||
|
||||
var Dataset = &dataset{}
|
||||
|
||||
func (c *dataset) List(ctx context.Context, _ *dto.ListDatasetReq) (*dto.ListDatasetRes, error) {
|
||||
list, err := service.DatasetService.List(ctx)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return &dto.ListDatasetRes{List: list}, nil
|
||||
}
|
||||
|
||||
func (c *dataset) Save(ctx context.Context, req *dto.SaveDatasetReq) (*dto.SaveDatasetRes, error) {
|
||||
id, err := service.DatasetService.Save(ctx, &entity.Dataset{
|
||||
Id: req.Id,
|
||||
Name: req.Name,
|
||||
Description: req.Description,
|
||||
EmbeddingCfgId: req.EmbeddingCfgId,
|
||||
Status: 1,
|
||||
})
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return &dto.SaveDatasetRes{Id: id}, nil
|
||||
}
|
||||
|
||||
func (c *dataset) Delete(ctx context.Context, req *dto.DeleteDatasetReq) (*dto.DeleteDatasetRes, error) {
|
||||
if err := service.DatasetService.Delete(ctx, req.Id); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return &dto.DeleteDatasetRes{}, nil
|
||||
}
|
||||
@@ -0,0 +1,62 @@
|
||||
package controller
|
||||
|
||||
import (
|
||||
"context"
|
||||
"io"
|
||||
|
||||
"rag-local/kb/model/dto"
|
||||
"rag-local/kb/service"
|
||||
|
||||
"github.com/gogf/gf/v2/errors/gerror"
|
||||
)
|
||||
|
||||
type document struct{}
|
||||
|
||||
var Document = &document{}
|
||||
|
||||
func (c *document) Upload(ctx context.Context, req *dto.UploadDocumentReq) (*dto.UploadDocumentRes, error) {
|
||||
if req.File == nil {
|
||||
return nil, gerror.New("请选择文件")
|
||||
}
|
||||
f, err := req.File.Open()
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer func() { _ = f.Close() }()
|
||||
data, err := io.ReadAll(f)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
doc, err := service.DocumentService.Upload(ctx, req.DatasetId, req.File.Filename, data)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return &dto.UploadDocumentRes{Id: doc.Id}, nil
|
||||
}
|
||||
|
||||
func (c *document) List(ctx context.Context, req *dto.ListDocumentReq) (*dto.ListDocumentRes, error) {
|
||||
list, total, err := service.DocumentService.List(ctx, req.DatasetId, req.Page, req.PageSize)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return &dto.ListDocumentRes{
|
||||
List: list,
|
||||
Total: total,
|
||||
Page: req.Page,
|
||||
PageSize: req.PageSize,
|
||||
}, nil
|
||||
}
|
||||
|
||||
func (c *document) Delete(ctx context.Context, req *dto.DeleteDocumentReq) (*dto.DeleteDocumentRes, error) {
|
||||
if err := service.DocumentService.Delete(ctx, req.Id); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return &dto.DeleteDocumentRes{}, nil
|
||||
}
|
||||
|
||||
func (c *document) Reembed(ctx context.Context, req *dto.ReembedDocumentReq) (*dto.ReembedDocumentRes, error) {
|
||||
if err := service.DocumentService.Reembed(ctx, req.Id); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return &dto.ReembedDocumentRes{}, nil
|
||||
}
|
||||
@@ -0,0 +1,25 @@
|
||||
package controller
|
||||
|
||||
import (
|
||||
"context"
|
||||
|
||||
"rag-local/kb/model/dto"
|
||||
"rag-local/kb/service"
|
||||
)
|
||||
|
||||
type kgEntity struct{}
|
||||
|
||||
var KgEntity = &kgEntity{}
|
||||
|
||||
func (c *kgEntity) List(ctx context.Context, req *dto.ListKgEntityReq) (*dto.ListKgEntityRes, error) {
|
||||
list, total, err := service.KgEntityService.List(ctx, req.DatasetId, req.Page, req.PageSize)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return &dto.ListKgEntityRes{
|
||||
List: list,
|
||||
Total: total,
|
||||
Page: req.Page,
|
||||
PageSize: req.PageSize,
|
||||
}, nil
|
||||
}
|
||||
@@ -0,0 +1,25 @@
|
||||
package controller
|
||||
|
||||
import (
|
||||
"context"
|
||||
|
||||
"rag-local/kb/model/dto"
|
||||
"rag-local/kb/service"
|
||||
)
|
||||
|
||||
type kgRelation struct{}
|
||||
|
||||
var KgRelation = &kgRelation{}
|
||||
|
||||
func (c *kgRelation) List(ctx context.Context, req *dto.ListKgRelationReq) (*dto.ListKgRelationRes, error) {
|
||||
list, total, err := service.KgRelationService.List(ctx, req.DatasetId, req.Page, req.PageSize)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return &dto.ListKgRelationRes{
|
||||
List: list,
|
||||
Total: total,
|
||||
Page: req.Page,
|
||||
PageSize: req.PageSize,
|
||||
}, nil
|
||||
}
|
||||
@@ -0,0 +1,78 @@
|
||||
package controller
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"time"
|
||||
|
||||
"rag-local/kb/model/domain"
|
||||
"rag-local/kb/model/dto"
|
||||
"rag-local/kb/service"
|
||||
|
||||
"github.com/gogf/gf/v2/frame/g"
|
||||
)
|
||||
|
||||
type message struct{}
|
||||
|
||||
var Message = &message{}
|
||||
|
||||
func (c *message) List(ctx context.Context, req *dto.ListMessageReq) (*dto.ListMessageRes, error) {
|
||||
list, err := service.MessageService.List(ctx, req.ConversationId)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return &dto.ListMessageRes{List: list}, nil
|
||||
}
|
||||
|
||||
// Chat RAG 问答 SSE 流式：citations → delta* → done / error
|
||||
func (c *message) Chat(ctx context.Context, req *dto.ChatReq) (*dto.ChatRes, error) {
|
||||
r := g.RequestFromCtx(ctx)
|
||||
r.Response.Header().Set("Content-Type", "text/event-stream; charset=utf-8")
|
||||
r.Response.Header().Set("Cache-Control", "no-cache")
|
||||
r.Response.Header().Set("X-Accel-Buffering", "no")
|
||||
r.Response.Header().Set("Connection", "keep-alive")
|
||||
|
||||
send := func(event string, data any) bool {
|
||||
buf, err := json.Marshal(data)
|
||||
if err != nil {
|
||||
return false
|
||||
}
|
||||
if event != "" {
|
||||
r.Response.Write("event: " + event + "\n")
|
||||
}
|
||||
r.Response.Write("data: " + string(buf) + "\n\n")
|
||||
r.Response.Flush()
|
||||
return true
|
||||
}
|
||||
|
||||
// 心跳：长回答期间保持连接，避免网关断流
|
||||
stopHeartbeat := make(chan struct{})
|
||||
defer close(stopHeartbeat)
|
||||
go func() {
|
||||
ticker := time.NewTicker(15 * time.Second)
|
||||
defer ticker.Stop()
|
||||
for {
|
||||
select {
|
||||
case <-stopHeartbeat:
|
||||
return
|
||||
case <-ticker.C:
|
||||
r.Response.Write(": ping\n\n")
|
||||
r.Response.Flush()
|
||||
}
|
||||
}
|
||||
}()
|
||||
|
||||
_, _, _, err := service.MessageService.Chat(ctx, req.ConversationId, req.DatasetId, req.Question,
|
||||
func(citations []domain.Citation, conversationId int64) {
|
||||
send("citations", map[string]any{"conversation_id": conversationId, "citations": citations})
|
||||
},
|
||||
func(delta string) {
|
||||
send("delta", map[string]string{"content": delta})
|
||||
})
|
||||
if err != nil {
|
||||
send("error", map[string]string{"message": err.Error()})
|
||||
return nil, nil
|
||||
}
|
||||
send("done", map[string]string{"status": "ok"})
|
||||
return nil, nil
|
||||
}
|
||||
@@ -0,0 +1,49 @@
|
||||
package controller
|
||||
|
||||
import (
|
||||
"context"
|
||||
|
||||
"rag-local/kb/model/dto"
|
||||
"rag-local/kb/model/entity"
|
||||
"rag-local/kb/service"
|
||||
)
|
||||
|
||||
type modelConfig struct{}
|
||||
|
||||
var ModelConfig = new(modelConfig)
|
||||
|
||||
func (c *modelConfig) List(ctx context.Context, req *dto.ListModelConfigReq) (res *dto.ListModelConfigRes, err error) {
|
||||
list, err := service.ModelConfigService.List(ctx, req.ModelType)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return &dto.ListModelConfigRes{List: list}, nil
|
||||
}
|
||||
|
||||
func (c *modelConfig) Save(ctx context.Context, req *dto.SaveModelConfigReq) (res *dto.SaveModelConfigRes, err error) {
|
||||
id, err := service.ModelConfigService.Save(ctx, &entity.ModelConfig{
|
||||
Id: req.Id,
|
||||
Name: req.Name,
|
||||
ModelType: req.ModelType,
|
||||
ModelName: req.ModelName,
|
||||
EndpointUrl: req.EndpointUrl,
|
||||
ApiKey: req.ApiKey,
|
||||
Dimension: req.Dimension,
|
||||
Extra: req.Extra,
|
||||
})
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return &dto.SaveModelConfigRes{Id: id}, nil
|
||||
}
|
||||
|
||||
func (c *modelConfig) Delete(ctx context.Context, req *dto.DeleteModelConfigReq) (res *dto.DeleteModelConfigRes, err error) {
|
||||
return nil, service.ModelConfigService.Delete(ctx, req.Id)
|
||||
}
|
||||
|
||||
func (c *modelConfig) Test(ctx context.Context, req *dto.TestModelConfigReq) (res *dto.TestModelConfigRes, err error) {
|
||||
if err := service.ModelConfigService.Test(ctx, req.Id); err != nil {
|
||||
return &dto.TestModelConfigRes{Ok: false, Msg: err.Error()}, nil
|
||||
}
|
||||
return &dto.TestModelConfigRes{Ok: true, Msg: "连接正常"}, nil
|
||||
}
|
||||
@@ -0,0 +1,32 @@
|
||||
package controller
|
||||
|
||||
import (
|
||||
"context"
|
||||
|
||||
"rag-local/kb/model/dto"
|
||||
"rag-local/kb/service"
|
||||
)
|
||||
|
||||
type parseTask struct{}
|
||||
|
||||
var ParseTask = &parseTask{}
|
||||
|
||||
func (c *parseTask) List(ctx context.Context, req *dto.ListParseTaskReq) (*dto.ListParseTaskRes, error) {
|
||||
list, total, err := service.ParseTaskService.List(ctx, req.Page, req.PageSize)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return &dto.ListParseTaskRes{
|
||||
List: list,
|
||||
Total: total,
|
||||
Page: req.Page,
|
||||
PageSize: req.PageSize,
|
||||
}, nil
|
||||
}
|
||||
|
||||
func (c *parseTask) Retry(ctx context.Context, req *dto.RetryParseTaskReq) (*dto.RetryParseTaskRes, error) {
|
||||
if err := service.ParseTaskService.Retry(ctx, req.Id); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return &dto.RetryParseTaskRes{}, nil
|
||||
}
|
||||
@@ -0,0 +1,53 @@
|
||||
package controller
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
|
||||
"rag-local/kb/model/dto"
|
||||
"rag-local/kb/service"
|
||||
)
|
||||
|
||||
type systemConfig struct{}
|
||||
|
||||
var SystemConfig = new(systemConfig)
|
||||
|
||||
func (c *systemConfig) Login(ctx context.Context, req *dto.LoginReq) (res *dto.LoginRes, err error) {
|
||||
token, err := service.SystemConfigService.Login(ctx, req.Token)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return &dto.LoginRes{Token: token}, nil
|
||||
}
|
||||
|
||||
func (c *systemConfig) Get(ctx context.Context, req *dto.GetSystemConfigReq) (res *dto.GetSystemConfigRes, err error) {
|
||||
chatModel, dataset, err := service.SystemConfigService.GetSettings(ctx)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return &dto.GetSystemConfigRes{DefaultChatModel: chatModel, DefaultDataset: dataset}, nil
|
||||
}
|
||||
|
||||
func (c *systemConfig) Update(ctx context.Context, req *dto.UpdateSystemConfigReq) (res *dto.UpdateSystemConfigRes, err error) {
|
||||
return nil, service.SystemConfigService.UpdateSettings(ctx, req.DefaultChatModel, req.DefaultDataset)
|
||||
}
|
||||
|
||||
func (c *systemConfig) GetToken(ctx context.Context, req *dto.GetTokenReq) (*dto.GetTokenRes, error) {
|
||||
token, err := service.SystemConfigService.GetToken(ctx)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return &dto.GetTokenRes{Token: token}, nil
|
||||
}
|
||||
|
||||
func (c *systemConfig) RegenerateToken(ctx context.Context, req *dto.RegenerateTokenReq) (res *dto.RegenerateTokenRes, err error) {
|
||||
token, err := service.SystemConfigService.RegenerateToken(ctx)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
fmt.Printf("\n============================================\n")
|
||||
fmt.Printf("访问令牌(登录用)已更新: %s\n", token)
|
||||
fmt.Printf("旧令牌已失效，请在登录页重新输入\n")
|
||||
fmt.Printf("============================================\n\n")
|
||||
return &dto.RegenerateTokenRes{Token: token}, nil
|
||||
}
|
||||
@@ -0,0 +1,244 @@
|
||||
package dao
|
||||
|
||||
import (
|
||||
"context"
|
||||
"database/sql"
|
||||
"errors"
|
||||
"strconv"
|
||||
"strings"
|
||||
|
||||
_ "modernc.org/sqlite/vec"
|
||||
|
||||
"rag-local/common"
|
||||
"rag-local/kb/consts"
|
||||
"rag-local/kb/model/domain"
|
||||
"rag-local/kb/model/entity"
|
||||
|
||||
"github.com/gogf/gf/v2/errors/gerror"
|
||||
"github.com/gogf/gf/v2/frame/g"
|
||||
"github.com/gogf/gf/v2/os/gtime"
|
||||
"github.com/gogf/gf/v2/text/gstr"
|
||||
)
|
||||
|
||||
var Chunk = &chunkDao{}
|
||||
|
||||
type chunkDao struct{}
|
||||
|
||||
func init() {
|
||||
ctx := context.Background()
|
||||
_, err := g.DB(consts.DbGroupDefault).Exec(ctx, `CREATE TABLE IF NOT EXISTS `+consts.TableNameChunk+` (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
dataset_id INTEGER NOT NULL DEFAULT 0,
|
||||
document_id INTEGER NOT NULL DEFAULT 0,
|
||||
seq INTEGER NOT NULL DEFAULT 0,
|
||||
content TEXT NOT NULL DEFAULT '',
|
||||
meta TEXT NOT NULL DEFAULT '',
|
||||
created_at DATETIME DEFAULT (datetime('now','localtime'))
|
||||
)`)
|
||||
if err != nil {
|
||||
g.Log().Warningf(ctx, "create kb_chunk table failed: %v", err)
|
||||
}
|
||||
if _, err := g.DB(consts.DbGroupDefault).Exec(ctx, "CREATE INDEX IF NOT EXISTS idx_kb_chunk_document ON "+consts.TableNameChunk+"(document_id)"); err != nil {
|
||||
g.Log().Warningf(ctx, "create index idx_kb_chunk_document failed: %v", err)
|
||||
}
|
||||
// 向量虚拟表（维度取配置 vector.dim，切换维度需删表重建）
|
||||
dim := g.Cfg().MustGet(ctx, "vector.dim", consts.DefaultEmbeddingDim).Int()
|
||||
if dim < 1 {
|
||||
dim = consts.DefaultEmbeddingDim
|
||||
}
|
||||
if _, err := g.DB(consts.DbGroupDefault).Exec(ctx, `CREATE VIRTUAL TABLE IF NOT EXISTS `+consts.TableNameChunkVec+
|
||||
` USING vec0(chunk_id INTEGER PRIMARY KEY, embedding float[`+strconv.Itoa(dim)+`])`); err != nil {
|
||||
g.Log().Warningf(ctx, "create vec0 table failed: %v", err)
|
||||
}
|
||||
// 全文索引虚拟表（默认 unicode61 tokenizer；中文分词在应用层完成，content_tokens 存分词后空格连接文本）
|
||||
if _, err := g.DB(consts.DbGroupDefault).Exec(ctx, `CREATE VIRTUAL TABLE IF NOT EXISTS `+consts.TableNameChunkFts+
|
||||
` USING fts5(chunk_id UNINDEXED, dataset_id UNINDEXED, title, content_tokens)`); err != nil {
|
||||
g.Log().Warningf(ctx, "create fts5 table failed: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func (d *chunkDao) GetOne(ctx context.Context, id int64) (*entity.Chunk, error) {
|
||||
var m entity.Chunk
|
||||
err := g.DB(consts.DbGroupDefault).Model(consts.TableNameChunk).Ctx(ctx).Where("id", id).Scan(&m)
|
||||
if errors.Is(err, sql.ErrNoRows) {
|
||||
return nil, nil
|
||||
}
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return &m, nil
|
||||
}
|
||||
|
||||
func (d *chunkDao) ListByDocument(ctx context.Context, documentId int64, page, pageSize int) ([]*entity.Chunk, int, error) {
|
||||
if page < 1 {
|
||||
page = 1
|
||||
}
|
||||
if pageSize < 1 {
|
||||
pageSize = 20
|
||||
}
|
||||
total, err := g.DB(consts.DbGroupDefault).Model(consts.TableNameChunk).Ctx(ctx).Where("document_id", documentId).Count()
|
||||
if err != nil {
|
||||
return nil, 0, err
|
||||
}
|
||||
var list []*entity.Chunk
|
||||
err = g.DB(consts.DbGroupDefault).Model(consts.TableNameChunk).Ctx(ctx).
|
||||
Where("document_id", documentId).Page(page, pageSize).OrderAsc("seq").Scan(&list)
|
||||
return list, total, err
|
||||
}
|
||||
|
||||
// InsertWithVec 事务内写入 chunk + 向量 + 全文索引，返回 chunk id
|
||||
func (d *chunkDao) InsertWithVec(ctx context.Context, datasetId, documentId int64, seq int, content, meta, title string, vecJson string, dim int) (int64, error) {
|
||||
tx, err := g.DB(consts.DbGroupDefault).Begin(ctx)
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
defer func() { _ = tx.Rollback() }()
|
||||
|
||||
r, err := tx.Model(consts.TableNameChunk).Ctx(ctx).Data(g.Map{
|
||||
"dataset_id": datasetId,
|
||||
"document_id": documentId,
|
||||
"seq": seq,
|
||||
"content": content,
|
||||
"meta": meta,
|
||||
"created_at": gtime.Now().Format("Y-m-d H:i:s"),
|
||||
}).Insert()
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
chunkId, _ := r.LastInsertId()
|
||||
if chunkId == 0 {
|
||||
return 0, gerror.New("chunk insert failed")
|
||||
}
|
||||
if vecJson != "" {
|
||||
if _, err := tx.Exec("INSERT INTO "+consts.TableNameChunkVec+" (chunk_id, embedding) VALUES (?, vec_f32(?))", chunkId, vecJson); err != nil {
|
||||
return 0, err
|
||||
}
|
||||
}
|
||||
if _, err := tx.Exec("INSERT INTO "+consts.TableNameChunkFts+" (chunk_id, dataset_id, title, content_tokens) VALUES (?, ?, ?, ?)",
|
||||
chunkId, datasetId, title, common.Tokenize(content)); err != nil {
|
||||
return 0, err
|
||||
}
|
||||
if err := tx.Commit(); err != nil {
|
||||
return 0, err
|
||||
}
|
||||
return chunkId, nil
|
||||
}
|
||||
|
||||
// DeleteByDocument 事务内删除文档全部分块（chunk + 向量 + 全文索引）
|
||||
func (d *chunkDao) DeleteByDocument(ctx context.Context, documentId int64) error {
|
||||
tx, err := g.DB(consts.DbGroupDefault).Begin(ctx)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer func() { _ = tx.Rollback() }()
|
||||
|
||||
ids := tx.Model(consts.TableNameChunk).Ctx(ctx).Fields("id").Where("document_id", documentId)
|
||||
r, err := ids.Array()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
chunkIds := make([]int64, 0, len(r))
|
||||
for _, v := range r {
|
||||
chunkIds = append(chunkIds, v.Int64())
|
||||
}
|
||||
if len(chunkIds) > 0 {
|
||||
placeholders := make([]string, 0, len(chunkIds))
|
||||
args := make([]interface{}, 0, len(chunkIds))
|
||||
for _, id := range chunkIds {
|
||||
placeholders = append(placeholders, "?")
|
||||
args = append(args, id)
|
||||
}
|
||||
in := gstr.Join(placeholders, ",")
|
||||
if _, err := tx.Exec("DELETE FROM "+consts.TableNameChunkVec+" WHERE chunk_id IN ("+in+")", args...); err != nil {
|
||||
return err
|
||||
}
|
||||
if _, err := tx.Exec("DELETE FROM "+consts.TableNameChunkFts+" WHERE chunk_id IN ("+in+")", args...); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
if _, err := tx.Model(consts.TableNameChunk).Ctx(ctx).Where("document_id", documentId).Delete(); err != nil {
|
||||
return err
|
||||
}
|
||||
return tx.Commit()
|
||||
}
|
||||
|
||||
func (d *chunkDao) UpdateContent(ctx context.Context, id int64, content, vecJson, tokens string) error {
|
||||
tx, err := g.DB(consts.DbGroupDefault).Begin(ctx)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer func() { _ = tx.Rollback() }()
|
||||
|
||||
if _, err := tx.Model(consts.TableNameChunk).Ctx(ctx).Data(g.Map{"content": content}).Where("id", id).Update(); err != nil {
|
||||
return err
|
||||
}
|
||||
if vecJson != "" {
|
||||
if _, err := tx.Exec("UPDATE "+consts.TableNameChunkVec+" SET embedding = vec_f32(?) WHERE chunk_id = ?", vecJson, id); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
if _, err := tx.Exec("UPDATE "+consts.TableNameChunkFts+" SET content_tokens = ? WHERE chunk_id = ?", tokens, id); err != nil {
|
||||
return err
|
||||
}
|
||||
return tx.Commit()
|
||||
}
|
||||
|
||||
// UpdateVec 仅更新向量（重新向量化，不动文本与 FTS）
|
||||
func (d *chunkDao) UpdateVec(ctx context.Context, id int64, vecJson string) error {
|
||||
if vecJson == "" {
|
||||
return nil
|
||||
}
|
||||
_, err := g.DB(consts.DbGroupDefault).Exec(ctx,
|
||||
"UPDATE "+consts.TableNameChunkVec+" SET embedding = vec_f32(?) WHERE chunk_id = ?", vecJson, id)
|
||||
return err
|
||||
}
|
||||
|
||||
func (d *chunkDao) CountByDocument(ctx context.Context, documentId int64) (int, error) {
|
||||
return g.DB(consts.DbGroupDefault).Model(consts.TableNameChunk).Ctx(ctx).Where("document_id", documentId).Count()
|
||||
}
|
||||
|
||||
// VecSearch 向量 KNN 检索：vec0 取最近 topK*4 后按数据集过滤（vec0 无 dataset 列）
|
||||
func (d *chunkDao) VecSearch(ctx context.Context, datasetId int64, vecJson string, topK int) ([]domain.VecHit, error) {
|
||||
r, err := g.DB(consts.DbGroupDefault).Ctx(ctx).Raw(
|
||||
`SELECT v.chunk_id, v.distance FROM (
|
||||
SELECT chunk_id, distance FROM `+consts.TableNameChunkVec+`
|
||||
WHERE embedding MATCH ? ORDER BY distance LIMIT ?
|
||||
) v INNER JOIN `+consts.TableNameChunk+` c ON c.id = v.chunk_id
|
||||
WHERE c.dataset_id = ? ORDER BY v.distance LIMIT ?`,
|
||||
vecJson, topK*4, datasetId, topK,
|
||||
).All()
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
hits := make([]domain.VecHit, 0, len(r))
|
||||
for _, row := range r {
|
||||
hits = append(hits, domain.VecHit{
|
||||
ChunkId: row["chunk_id"].Int64(),
|
||||
Distance: row["distance"].Float64(),
|
||||
})
|
||||
}
|
||||
return hits, nil
|
||||
}
|
||||
|
||||
// FtsSearch 全文检索（BM25）：中文分词已在应用层完成，query 为空格连接的引号词串
|
||||
func (d *chunkDao) FtsSearch(ctx context.Context, datasetId int64, query string, topK int) ([]domain.FtsHit, error) {
|
||||
if strings.TrimSpace(query) == "" {
|
||||
return nil, nil
|
||||
}
|
||||
r, err := g.DB(consts.DbGroupDefault).Ctx(ctx).Raw(
|
||||
`SELECT chunk_id, bm25(`+consts.TableNameChunkFts+`) AS score FROM `+consts.TableNameChunkFts+
|
||||
` WHERE `+consts.TableNameChunkFts+` MATCH ? AND dataset_id = ? ORDER BY score LIMIT ?`,
|
||||
query, datasetId, topK,
|
||||
).All()
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
hits := make([]domain.FtsHit, 0, len(r))
|
||||
for _, row := range r {
|
||||
hits = append(hits, domain.FtsHit{
|
||||
ChunkId: row["chunk_id"].Int64(),
|
||||
Score: row["score"].Float64(),
|
||||
})
|
||||
}
|
||||
return hits, nil
|
||||
}
|
||||
@@ -0,0 +1,94 @@
|
||||
package dao
|
||||
|
||||
import (
|
||||
"context"
|
||||
"database/sql"
|
||||
"errors"
|
||||
|
||||
"rag-local/kb/consts"
|
||||
"rag-local/kb/model/entity"
|
||||
|
||||
"github.com/gogf/gf/v2/frame/g"
|
||||
"github.com/gogf/gf/v2/os/gtime"
|
||||
)
|
||||
|
||||
var Conversation = &conversationDao{}
|
||||
|
||||
type conversationDao struct{}
|
||||
|
||||
func init() {
|
||||
ctx := context.Background()
|
||||
_, err := g.DB(consts.DbGroupChat).Exec(ctx, `CREATE TABLE IF NOT EXISTS `+consts.TableNameConversation+` (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
dataset_id INTEGER NOT NULL DEFAULT 0,
|
||||
title TEXT NOT NULL DEFAULT '',
|
||||
created_at DATETIME DEFAULT (datetime('now','localtime')),
|
||||
updated_at DATETIME DEFAULT (datetime('now','localtime'))
|
||||
)`)
|
||||
if err != nil {
|
||||
g.Log().Warningf(ctx, "create chat_conversation table failed: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func (d *conversationDao) GetOne(ctx context.Context, id int64) (*entity.Conversation, error) {
|
||||
var m entity.Conversation
|
||||
err := g.DB(consts.DbGroupChat).Model(consts.TableNameConversation).Ctx(ctx).Where("id", id).Scan(&m)
|
||||
if errors.Is(err, sql.ErrNoRows) {
|
||||
return nil, nil
|
||||
}
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return &m, nil
|
||||
}
|
||||
|
||||
func (d *conversationDao) List(ctx context.Context) ([]*entity.Conversation, error) {
|
||||
var list []*entity.Conversation
|
||||
err := g.DB(consts.DbGroupChat).Model(consts.TableNameConversation).Ctx(ctx).OrderDesc("id").Scan(&list)
|
||||
return list, err
|
||||
}
|
||||
|
||||
func (d *conversationDao) Insert(ctx context.Context, datasetId int64, title string) (int64, error) {
|
||||
now := gtime.Now().Format("Y-m-d H:i:s")
|
||||
r, err := g.DB(consts.DbGroupChat).Model(consts.TableNameConversation).Ctx(ctx).Data(g.Map{
|
||||
"dataset_id": datasetId,
|
||||
"title": title,
|
||||
"created_at": now,
|
||||
"updated_at": now,
|
||||
}).Insert()
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
return r.LastInsertId()
|
||||
}
|
||||
|
||||
func (d *conversationDao) UpdateTitle(ctx context.Context, id int64, title string) error {
|
||||
_, err := g.DB(consts.DbGroupChat).Model(consts.TableNameConversation).Ctx(ctx).Data(g.Map{
|
||||
"title": title,
|
||||
"updated_at": gtime.Now().Format("Y-m-d H:i:s"),
|
||||
}).Where("id", id).Update()
|
||||
return err
|
||||
}
|
||||
|
||||
func (d *conversationDao) UpdateDataset(ctx context.Context, id, datasetId int64) error {
|
||||
_, err := g.DB(consts.DbGroupChat).Model(consts.TableNameConversation).Ctx(ctx).Data(g.Map{
|
||||
"dataset_id": datasetId,
|
||||
"updated_at": gtime.Now().Format("Y-m-d H:i:s"),
|
||||
}).Where("id", id).Update()
|
||||
return err
|
||||
}
|
||||
|
||||
func (d *conversationDao) Delete(ctx context.Context, id int64) error {
|
||||
tx, err := g.DB(consts.DbGroupChat).Begin(ctx)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer func() { _ = tx.Rollback() }()
|
||||
if _, err := tx.Model(consts.TableNameMessage).Ctx(ctx).Where("conversation_id", id).Delete(); err != nil {
|
||||
return err
|
||||
}
|
||||
if _, err := tx.Model(consts.TableNameConversation).Ctx(ctx).Where("id", id).Delete(); err != nil {
|
||||
return err
|
||||
}
|
||||
return tx.Commit()
|
||||
}
|
||||
@@ -0,0 +1,95 @@
|
||||
package dao
|
||||
|
||||
import (
|
||||
"context"
|
||||
"database/sql"
|
||||
"errors"
|
||||
|
||||
"rag-local/kb/consts"
|
||||
"rag-local/kb/model/entity"
|
||||
|
||||
"github.com/gogf/gf/v2/frame/g"
|
||||
"github.com/gogf/gf/v2/os/gtime"
|
||||
)
|
||||
|
||||
var Dataset = &datasetDao{}
|
||||
|
||||
type datasetDao struct{}
|
||||
|
||||
func init() {
|
||||
ctx := context.Background()
|
||||
_, err := g.DB(consts.DbGroupDefault).Exec(ctx, `CREATE TABLE IF NOT EXISTS `+consts.TableNameDataset+` (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
name TEXT NOT NULL DEFAULT '',
|
||||
description TEXT NOT NULL DEFAULT '',
|
||||
embedding_cfg_id INTEGER NOT NULL DEFAULT 0,
|
||||
status INTEGER NOT NULL DEFAULT 1,
|
||||
created_at DATETIME DEFAULT (datetime('now','localtime')),
|
||||
updated_at DATETIME DEFAULT (datetime('now','localtime'))
|
||||
)`)
|
||||
if err != nil {
|
||||
g.Log().Warningf(ctx, "create kb_dataset table failed: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func (d *datasetDao) GetOne(ctx context.Context, id int64) (*entity.Dataset, error) {
|
||||
var m entity.Dataset
|
||||
err := g.DB(consts.DbGroupDefault).Model(consts.TableNameDataset).Ctx(ctx).Where("id", id).Scan(&m)
|
||||
if errors.Is(err, sql.ErrNoRows) {
|
||||
return nil, nil
|
||||
}
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return &m, nil
|
||||
}
|
||||
|
||||
func (d *datasetDao) List(ctx context.Context) ([]*entity.Dataset, error) {
|
||||
var list []*entity.Dataset
|
||||
err := g.DB(consts.DbGroupDefault).Model(consts.TableNameDataset).Ctx(ctx).OrderAsc("id").Scan(&list)
|
||||
return list, err
|
||||
}
|
||||
|
||||
func (d *datasetDao) Insert(ctx context.Context, data *entity.Dataset) (int64, error) {
|
||||
now := gtime.Now().Format("Y-m-d H:i:s")
|
||||
r, err := g.DB(consts.DbGroupDefault).Model(consts.TableNameDataset).Ctx(ctx).Data(g.Map{
|
||||
"name": data.Name,
|
||||
"description": data.Description,
|
||||
"embedding_cfg_id": data.EmbeddingCfgId,
|
||||
"status": data.Status,
|
||||
"created_at": now,
|
||||
"updated_at": now,
|
||||
}).Insert()
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
return r.LastInsertId()
|
||||
}
|
||||
|
||||
func (d *datasetDao) Update(ctx context.Context, data *entity.Dataset) error {
|
||||
_, err := g.DB(consts.DbGroupDefault).Model(consts.TableNameDataset).Ctx(ctx).Data(g.Map{
|
||||
"name": data.Name,
|
||||
"description": data.Description,
|
||||
"embedding_cfg_id": data.EmbeddingCfgId,
|
||||
"status": data.Status,
|
||||
"updated_at": gtime.Now().Format("Y-m-d H:i:s"),
|
||||
}).Where("id", data.Id).Update()
|
||||
return err
|
||||
}
|
||||
|
||||
func (d *datasetDao) Delete(ctx context.Context, id int64) error {
|
||||
_, err := g.DB(consts.DbGroupDefault).Model(consts.TableNameDataset).Ctx(ctx).Where("id", id).Delete()
|
||||
return err
|
||||
}
|
||||
|
||||
func (d *datasetDao) GetEmbeddingCfgId(ctx context.Context, id int64) (int64, error) {
|
||||
r, err := g.DB(consts.DbGroupDefault).Model(consts.TableNameDataset).Ctx(ctx).
|
||||
Fields("embedding_cfg_id").Where("id", id).One()
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
if r == nil {
|
||||
return 0, nil
|
||||
}
|
||||
return r["embedding_cfg_id"].Int64(), nil
|
||||
}
|
||||
@@ -0,0 +1,98 @@
|
||||
package dao
|
||||
|
||||
import (
|
||||
"context"
|
||||
"database/sql"
|
||||
"errors"
|
||||
|
||||
"rag-local/kb/consts"
|
||||
"rag-local/kb/model/entity"
|
||||
|
||||
"github.com/gogf/gf/v2/frame/g"
|
||||
"github.com/gogf/gf/v2/os/gtime"
|
||||
)
|
||||
|
||||
var Document = &documentDao{}
|
||||
|
||||
type documentDao struct{}
|
||||
|
||||
func init() {
|
||||
ctx := context.Background()
|
||||
_, err := g.DB(consts.DbGroupDefault).Exec(ctx, `CREATE TABLE IF NOT EXISTS `+consts.TableNameDocument+` (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
dataset_id INTEGER NOT NULL DEFAULT 0,
|
||||
filename TEXT NOT NULL DEFAULT '',
|
||||
file_path TEXT NOT NULL DEFAULT '',
|
||||
file_size INTEGER NOT NULL DEFAULT 0,
|
||||
file_type TEXT NOT NULL DEFAULT '',
|
||||
status INTEGER NOT NULL DEFAULT 0,
|
||||
chunk_count INTEGER NOT NULL DEFAULT 0,
|
||||
error_msg TEXT NOT NULL DEFAULT '',
|
||||
created_at DATETIME DEFAULT (datetime('now','localtime')),
|
||||
updated_at DATETIME DEFAULT (datetime('now','localtime'))
|
||||
)`)
|
||||
if err != nil {
|
||||
g.Log().Warningf(ctx, "create kb_document table failed: %v", err)
|
||||
}
|
||||
if _, err := g.DB(consts.DbGroupDefault).Exec(ctx, "CREATE INDEX IF NOT EXISTS idx_kb_document_dataset ON "+consts.TableNameDocument+"(dataset_id)"); err != nil {
|
||||
g.Log().Warningf(ctx, "create index idx_kb_document_dataset failed: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func (d *documentDao) GetOne(ctx context.Context, id int64) (*entity.Document, error) {
|
||||
var m entity.Document
|
||||
err := g.DB(consts.DbGroupDefault).Model(consts.TableNameDocument).Ctx(ctx).Where("id", id).Scan(&m)
|
||||
if errors.Is(err, sql.ErrNoRows) {
|
||||
return nil, nil
|
||||
}
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return &m, nil
|
||||
}
|
||||
|
||||
func (d *documentDao) List(ctx context.Context, datasetId int64, page, pageSize int) ([]*entity.Document, int, error) {
|
||||
if page < 1 {
|
||||
page = 1
|
||||
}
|
||||
if pageSize < 1 {
|
||||
pageSize = 20
|
||||
}
|
||||
total, err := g.DB(consts.DbGroupDefault).Model(consts.TableNameDocument).Ctx(ctx).Where("dataset_id", datasetId).Count()
|
||||
if err != nil {
|
||||
return nil, 0, err
|
||||
}
|
||||
var list []*entity.Document
|
||||
err = g.DB(consts.DbGroupDefault).Model(consts.TableNameDocument).Ctx(ctx).
|
||||
Where("dataset_id", datasetId).Page(page, pageSize).OrderDesc("id").Scan(&list)
|
||||
return list, total, err
|
||||
}
|
||||
|
||||
func (d *documentDao) Insert(ctx context.Context, data *entity.Document) (int64, error) {
|
||||
now := gtime.Now().Format("Y-m-d H:i:s")
|
||||
r, err := g.DB(consts.DbGroupDefault).Model(consts.TableNameDocument).Ctx(ctx).Data(g.Map{
|
||||
"dataset_id": data.DatasetId,
|
||||
"filename": data.Filename,
|
||||
"file_path": data.FilePath,
|
||||
"file_size": data.FileSize,
|
||||
"file_type": data.FileType,
|
||||
"status": data.Status,
|
||||
"created_at": now,
|
||||
"updated_at": now,
|
||||
}).Insert()
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
return r.LastInsertId()
|
||||
}
|
||||
|
||||
func (d *documentDao) UpdateFields(ctx context.Context, id int64, data g.Map) error {
|
||||
data["updated_at"] = gtime.Now().Format("Y-m-d H:i:s")
|
||||
_, err := g.DB(consts.DbGroupDefault).Model(consts.TableNameDocument).Ctx(ctx).Data(data).Where("id", id).Update()
|
||||
return err
|
||||
}
|
||||
|
||||
func (d *documentDao) Delete(ctx context.Context, id int64) error {
|
||||
_, err := g.DB(consts.DbGroupDefault).Model(consts.TableNameDocument).Ctx(ctx).Where("id", id).Delete()
|
||||
return err
|
||||
}
|
||||
@@ -0,0 +1,117 @@
|
||||
package dao
|
||||
|
||||
import (
|
||||
"context"
|
||||
"database/sql"
|
||||
"errors"
|
||||
"strings"
|
||||
|
||||
"rag-local/kb/consts"
|
||||
"rag-local/kb/model/entity"
|
||||
|
||||
"github.com/gogf/gf/v2/frame/g"
|
||||
"github.com/gogf/gf/v2/os/gtime"
|
||||
)
|
||||
|
||||
var KgEntity = &kgEntityDao{}
|
||||
|
||||
type kgEntityDao struct{}
|
||||
|
||||
func init() {
|
||||
ctx := context.Background()
|
||||
_, err := g.DB(consts.DbGroupDefault).Exec(ctx, `CREATE TABLE IF NOT EXISTS `+consts.TableNameKgEntity+` (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
dataset_id INTEGER NOT NULL DEFAULT 0,
|
||||
name TEXT NOT NULL DEFAULT '',
|
||||
entity_type TEXT NOT NULL DEFAULT '',
|
||||
chunk_id INTEGER NOT NULL DEFAULT 0,
|
||||
created_at DATETIME DEFAULT (datetime('now','localtime')),
|
||||
updated_at DATETIME DEFAULT (datetime('now','localtime')),
|
||||
UNIQUE(dataset_id, name)
|
||||
)`)
|
||||
if err != nil {
|
||||
g.Log().Warningf(ctx, "create kg_entity table failed: %v", err)
|
||||
}
|
||||
if _, err := g.DB(consts.DbGroupDefault).Exec(ctx, "CREATE INDEX IF NOT EXISTS idx_kg_entity_dataset ON "+consts.TableNameKgEntity+"(dataset_id)"); err != nil {
|
||||
g.Log().Warningf(ctx, "create index idx_kg_entity_dataset failed: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func (d *kgEntityDao) GetOne(ctx context.Context, id int64) (*entity.KgEntity, error) {
|
||||
var m entity.KgEntity
|
||||
err := g.DB(consts.DbGroupDefault).Model(consts.TableNameKgEntity).Ctx(ctx).Where("id", id).Scan(&m)
|
||||
if errors.Is(err, sql.ErrNoRows) {
|
||||
return nil, nil
|
||||
}
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return &m, nil
|
||||
}
|
||||
|
||||
func (d *kgEntityDao) List(ctx context.Context, datasetId int64, page, pageSize int) ([]*entity.KgEntity, int, error) {
|
||||
if page < 1 {
|
||||
page = 1
|
||||
}
|
||||
if pageSize < 1 {
|
||||
pageSize = 20
|
||||
}
|
||||
m := g.DB(consts.DbGroupDefault).Model(consts.TableNameKgEntity).Ctx(ctx)
|
||||
if datasetId > 0 {
|
||||
m = m.Where("dataset_id", datasetId)
|
||||
}
|
||||
total, err := m.Count()
|
||||
if err != nil {
|
||||
return nil, 0, err
|
||||
}
|
||||
var list []*entity.KgEntity
|
||||
err = m.Page(page, pageSize).OrderDesc("id").Scan(&list)
|
||||
return list, total, err
|
||||
}
|
||||
|
||||
// ListNames 数据集全部实体名（实体链接在内存中打分，本地库规模可控）
|
||||
func (d *kgEntityDao) ListNames(ctx context.Context, datasetId int64) ([]string, error) {
|
||||
var list []*entity.KgEntity
|
||||
err := g.DB(consts.DbGroupDefault).Model(consts.TableNameKgEntity).Ctx(ctx).
|
||||
Fields("name").Where("dataset_id", datasetId).Scan(&list)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
names := make([]string, 0, len(list))
|
||||
for _, e := range list {
|
||||
names = append(names, e.Name)
|
||||
}
|
||||
return names, nil
|
||||
}
|
||||
|
||||
// Upsert 按 (dataset_id, name) 去重，已存在则更新类型与来源
|
||||
func (d *kgEntityDao) Upsert(ctx context.Context, datasetId, chunkId int64, name, entityType string) error {
|
||||
now := gtime.Now().Format("Y-m-d H:i:s")
|
||||
_, err := g.DB(consts.DbGroupDefault).Exec(ctx, `INSERT INTO `+consts.TableNameKgEntity+`
|
||||
(dataset_id, name, entity_type, chunk_id, created_at, updated_at)
|
||||
VALUES (?,?,?,?,?,?)
|
||||
ON CONFLICT(dataset_id, name) DO UPDATE SET
|
||||
entity_type=excluded.entity_type, chunk_id=excluded.chunk_id, updated_at=excluded.updated_at`,
|
||||
datasetId, name, entityType, chunkId, now, now)
|
||||
return err
|
||||
}
|
||||
|
||||
func (d *kgEntityDao) DeleteByChunkIds(ctx context.Context, chunkIds []int64) error {
|
||||
if len(chunkIds) == 0 {
|
||||
return nil
|
||||
}
|
||||
placeholders := strings.TrimSuffix(strings.Repeat("?,", len(chunkIds)), ",")
|
||||
args := make([]any, 0, len(chunkIds))
|
||||
for _, id := range chunkIds {
|
||||
args = append(args, id)
|
||||
}
|
||||
_, err := g.DB(consts.DbGroupDefault).Exec(ctx,
|
||||
"DELETE FROM "+consts.TableNameKgEntity+" WHERE chunk_id IN ("+placeholders+")", args...)
|
||||
return err
|
||||
}
|
||||
|
||||
func (d *kgEntityDao) DeleteByDataset(ctx context.Context, datasetId int64) error {
|
||||
_, err := g.DB(consts.DbGroupDefault).Model(consts.TableNameKgEntity).Ctx(ctx).
|
||||
Where("dataset_id", datasetId).Delete()
|
||||
return err
|
||||
}
|
||||
@@ -0,0 +1,137 @@
|
||||
package dao
|
||||
|
||||
import (
|
||||
"context"
|
||||
"database/sql"
|
||||
"errors"
|
||||
"strings"
|
||||
|
||||
"rag-local/kb/consts"
|
||||
"rag-local/kb/model/entity"
|
||||
|
||||
"github.com/gogf/gf/v2/frame/g"
|
||||
"github.com/gogf/gf/v2/os/gtime"
|
||||
)
|
||||
|
||||
var KgRelation = &kgRelationDao{}
|
||||
|
||||
type kgRelationDao struct{}
|
||||
|
||||
func init() {
|
||||
ctx := context.Background()
|
||||
_, err := g.DB(consts.DbGroupDefault).Exec(ctx, `CREATE TABLE IF NOT EXISTS `+consts.TableNameKgRelation+` (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
dataset_id INTEGER NOT NULL DEFAULT 0,
|
||||
head TEXT NOT NULL DEFAULT '',
|
||||
relation TEXT NOT NULL DEFAULT '',
|
||||
tail TEXT NOT NULL DEFAULT '',
|
||||
chunk_id INTEGER NOT NULL DEFAULT 0,
|
||||
created_at DATETIME DEFAULT (datetime('now','localtime'))
|
||||
)`)
|
||||
if err != nil {
|
||||
g.Log().Warningf(ctx, "create kg_relation table failed: %v", err)
|
||||
}
|
||||
if _, err := g.DB(consts.DbGroupDefault).Exec(ctx, "CREATE INDEX IF NOT EXISTS idx_kg_relation_dataset ON "+consts.TableNameKgRelation+"(dataset_id)"); err != nil {
|
||||
g.Log().Warningf(ctx, "create index idx_kg_relation_dataset failed: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func (d *kgRelationDao) GetOne(ctx context.Context, id int64) (*entity.KgRelation, error) {
|
||||
var m entity.KgRelation
|
||||
err := g.DB(consts.DbGroupDefault).Model(consts.TableNameKgRelation).Ctx(ctx).Where("id", id).Scan(&m)
|
||||
if errors.Is(err, sql.ErrNoRows) {
|
||||
return nil, nil
|
||||
}
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return &m, nil
|
||||
}
|
||||
|
||||
func (d *kgRelationDao) List(ctx context.Context, datasetId int64, page, pageSize int) ([]*entity.KgRelation, int, error) {
|
||||
if page < 1 {
|
||||
page = 1
|
||||
}
|
||||
if pageSize < 1 {
|
||||
pageSize = 20
|
||||
}
|
||||
m := g.DB(consts.DbGroupDefault).Model(consts.TableNameKgRelation).Ctx(ctx)
|
||||
if datasetId > 0 {
|
||||
m = m.Where("dataset_id", datasetId)
|
||||
}
|
||||
total, err := m.Count()
|
||||
if err != nil {
|
||||
return nil, 0, err
|
||||
}
|
||||
var list []*entity.KgRelation
|
||||
err = m.Page(page, pageSize).OrderDesc("id").Scan(&list)
|
||||
return list, total, err
|
||||
}
|
||||
|
||||
func (d *kgRelationDao) Insert(ctx context.Context, datasetId, chunkId int64, head, relation, tail string) error {
|
||||
_, err := g.DB(consts.DbGroupDefault).Model(consts.TableNameKgRelation).Ctx(ctx).Data(g.Map{
|
||||
"dataset_id": datasetId,
|
||||
"head": head,
|
||||
"relation": relation,
|
||||
"tail": tail,
|
||||
"chunk_id": chunkId,
|
||||
"created_at": gtime.Now().Format("Y-m-d H:i:s"),
|
||||
}).Insert()
|
||||
return err
|
||||
}
|
||||
|
||||
// Neighbors 一跳邻居：head 或 tail 命中实体名的三元组（实体链接用）
|
||||
func (d *kgRelationDao) Neighbors(ctx context.Context, datasetId int64, names []string, limit int) ([]*entity.KgRelation, error) {
|
||||
if len(names) == 0 {
|
||||
return nil, nil
|
||||
}
|
||||
if limit < 1 {
|
||||
limit = 20
|
||||
}
|
||||
placeholders := strings.TrimSuffix(strings.Repeat("?,", len(names)), ",")
|
||||
args := make([]any, 0, len(names)*2+2)
|
||||
for _, n := range names {
|
||||
args = append(args, n, n)
|
||||
}
|
||||
args = append(args, datasetId, limit)
|
||||
r, err := g.DB(consts.DbGroupDefault).Ctx(ctx).Raw(`
|
||||
SELECT id, dataset_id, head, relation, tail, chunk_id, created_at
|
||||
FROM `+consts.TableNameKgRelation+`
|
||||
WHERE (head IN (`+placeholders+`) OR tail IN (`+placeholders+`)) AND dataset_id = ?
|
||||
ORDER BY id LIMIT ?`, args...).All()
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
list := make([]*entity.KgRelation, 0, len(r))
|
||||
for _, row := range r {
|
||||
list = append(list, &entity.KgRelation{
|
||||
Id: row["id"].Int64(),
|
||||
DatasetId: row["dataset_id"].Int64(),
|
||||
Head: row["head"].String(),
|
||||
Relation: row["relation"].String(),
|
||||
Tail: row["tail"].String(),
|
||||
ChunkId: row["chunk_id"].Int64(),
|
||||
})
|
||||
}
|
||||
return list, nil
|
||||
}
|
||||
|
||||
func (d *kgRelationDao) DeleteByChunkIds(ctx context.Context, chunkIds []int64) error {
|
||||
if len(chunkIds) == 0 {
|
||||
return nil
|
||||
}
|
||||
placeholders := strings.TrimSuffix(strings.Repeat("?,", len(chunkIds)), ",")
|
||||
args := make([]any, 0, len(chunkIds))
|
||||
for _, id := range chunkIds {
|
||||
args = append(args, id)
|
||||
}
|
||||
_, err := g.DB(consts.DbGroupDefault).Exec(ctx,
|
||||
"DELETE FROM "+consts.TableNameKgRelation+" WHERE chunk_id IN ("+placeholders+")", args...)
|
||||
return err
|
||||
}
|
||||
|
||||
func (d *kgRelationDao) DeleteByDataset(ctx context.Context, datasetId int64) error {
|
||||
_, err := g.DB(consts.DbGroupDefault).Model(consts.TableNameKgRelation).Ctx(ctx).
|
||||
Where("dataset_id", datasetId).Delete()
|
||||
return err
|
||||
}
|
||||
@@ -0,0 +1,54 @@
|
||||
package dao
|
||||
|
||||
import (
|
||||
"context"
|
||||
|
||||
"rag-local/kb/consts"
|
||||
"rag-local/kb/model/entity"
|
||||
|
||||
"github.com/gogf/gf/v2/frame/g"
|
||||
"github.com/gogf/gf/v2/os/gtime"
|
||||
)
|
||||
|
||||
var Message = &messageDao{}
|
||||
|
||||
type messageDao struct{}
|
||||
|
||||
func init() {
|
||||
ctx := context.Background()
|
||||
_, err := g.DB(consts.DbGroupChat).Exec(ctx, `CREATE TABLE IF NOT EXISTS `+consts.TableNameMessage+` (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
conversation_id INTEGER NOT NULL DEFAULT 0,
|
||||
role TEXT NOT NULL DEFAULT 'user',
|
||||
content TEXT NOT NULL DEFAULT '',
|
||||
citations TEXT NOT NULL DEFAULT '',
|
||||
created_at DATETIME DEFAULT (datetime('now','localtime'))
|
||||
)`)
|
||||
if err != nil {
|
||||
g.Log().Warningf(ctx, "create chat_message table failed: %v", err)
|
||||
}
|
||||
if _, err := g.DB(consts.DbGroupChat).Exec(ctx, "CREATE INDEX IF NOT EXISTS idx_chat_message_conversation ON "+consts.TableNameMessage+"(conversation_id)"); err != nil {
|
||||
g.Log().Warningf(ctx, "create index idx_chat_message_conversation failed: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func (d *messageDao) List(ctx context.Context, conversationId int64) ([]*entity.Message, error) {
|
||||
var list []*entity.Message
|
||||
err := g.DB(consts.DbGroupChat).Model(consts.TableNameMessage).Ctx(ctx).
|
||||
Where("conversation_id", conversationId).OrderAsc("id").Scan(&list)
|
||||
return list, err
|
||||
}
|
||||
|
||||
func (d *messageDao) Insert(ctx context.Context, conversationId int64, role, content, citations string) (int64, error) {
|
||||
r, err := g.DB(consts.DbGroupChat).Model(consts.TableNameMessage).Ctx(ctx).Data(g.Map{
|
||||
"conversation_id": conversationId,
|
||||
"role": role,
|
||||
"content": content,
|
||||
"citations": citations,
|
||||
"created_at": gtime.Now().Format("Y-m-d H:i:s"),
|
||||
}).Insert()
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
return r.LastInsertId()
|
||||
}
|
||||
@@ -0,0 +1,110 @@
|
||||
package dao
|
||||
|
||||
import (
|
||||
"context"
|
||||
"database/sql"
|
||||
"errors"
|
||||
|
||||
"rag-local/common"
|
||||
"rag-local/kb/consts"
|
||||
"rag-local/kb/model/entity"
|
||||
|
||||
"github.com/gogf/gf/v2/database/gdb"
|
||||
"github.com/gogf/gf/v2/frame/g"
|
||||
"github.com/gogf/gf/v2/os/gcache"
|
||||
"github.com/gogf/gf/v2/os/gtime"
|
||||
"github.com/gogf/gf/v2/util/gconv"
|
||||
)
|
||||
|
||||
var ModelConfig = &modelConfigDao{}
|
||||
|
||||
type modelConfigDao struct{}
|
||||
|
||||
func init() {
|
||||
ctx := context.Background()
|
||||
_, err := g.DB(consts.DbGroupSystem).Exec(ctx, `CREATE TABLE IF NOT EXISTS `+consts.TableNameModelConfig+` (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
name TEXT NOT NULL DEFAULT '',
|
||||
model_type TEXT NOT NULL DEFAULT 'chat',
|
||||
model_name TEXT NOT NULL DEFAULT '',
|
||||
endpoint_url TEXT NOT NULL DEFAULT '',
|
||||
api_key TEXT NOT NULL DEFAULT '',
|
||||
dimension INTEGER NOT NULL DEFAULT 1024,
|
||||
extra TEXT NOT NULL DEFAULT '',
|
||||
created_at DATETIME DEFAULT (datetime('now','localtime')),
|
||||
updated_at DATETIME DEFAULT (datetime('now','localtime'))
|
||||
)`)
|
||||
if err != nil {
|
||||
g.Log().Warningf(ctx, "create model_config table failed: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func (d *modelConfigDao) GetOne(ctx context.Context, id int64) (*entity.ModelConfig, error) {
|
||||
var m entity.ModelConfig
|
||||
err := g.DB(consts.DbGroupSystem).Model(consts.TableNameModelConfig).Ctx(ctx).
|
||||
Cache(gdb.CacheOption{Duration: common.CacheTTL(), Name: "model_config_GetOne_" + gconv.String(id)}).
|
||||
Where("id", id).Scan(&m)
|
||||
if errors.Is(err, sql.ErrNoRows) {
|
||||
return nil, nil
|
||||
}
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return &m, nil
|
||||
}
|
||||
|
||||
func (d *modelConfigDao) List(ctx context.Context, modelType string) ([]*entity.ModelConfig, error) {
|
||||
var list []*entity.ModelConfig
|
||||
m := g.DB(consts.DbGroupSystem).Model(consts.TableNameModelConfig).Ctx(ctx).OrderAsc("id")
|
||||
if modelType != "" {
|
||||
m = m.Where("model_type", modelType)
|
||||
}
|
||||
err := m.Scan(&list)
|
||||
return list, err
|
||||
}
|
||||
|
||||
func (d *modelConfigDao) Insert(ctx context.Context, data *entity.ModelConfig) (int64, error) {
|
||||
now := gtime.Now().Format("Y-m-d H:i:s")
|
||||
r, err := g.DB(consts.DbGroupSystem).Model(consts.TableNameModelConfig).Ctx(ctx).Data(g.Map{
|
||||
"name": data.Name,
|
||||
"model_type": data.ModelType,
|
||||
"model_name": data.ModelName,
|
||||
"endpoint_url": data.EndpointUrl,
|
||||
"api_key": data.ApiKey,
|
||||
"dimension": data.Dimension,
|
||||
"extra": data.Extra,
|
||||
"created_at": now,
|
||||
"updated_at": now,
|
||||
}).Insert()
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
return r.LastInsertId()
|
||||
}
|
||||
|
||||
func (d *modelConfigDao) Update(ctx context.Context, data *entity.ModelConfig) error {
|
||||
_, err := g.DB(consts.DbGroupSystem).Model(consts.TableNameModelConfig).Ctx(ctx).Data(g.Map{
|
||||
"name": data.Name,
|
||||
"model_type": data.ModelType,
|
||||
"model_name": data.ModelName,
|
||||
"endpoint_url": data.EndpointUrl,
|
||||
"api_key": data.ApiKey,
|
||||
"dimension": data.Dimension,
|
||||
"extra": data.Extra,
|
||||
"updated_at": gtime.Now().Format("Y-m-d H:i:s"),
|
||||
}).Where("id", data.Id).Update()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
_, _ = gcache.Remove(ctx, "model_config_GetOne_"+gconv.String(data.Id))
|
||||
return nil
|
||||
}
|
||||
|
||||
func (d *modelConfigDao) Delete(ctx context.Context, id int64) error {
|
||||
_, err := g.DB(consts.DbGroupSystem).Model(consts.TableNameModelConfig).Ctx(ctx).Where("id", id).Delete()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
_, _ = gcache.Remove(ctx, "model_config_GetOne_"+gconv.String(id))
|
||||
return nil
|
||||
}
|
||||
@@ -0,0 +1,110 @@
|
||||
package dao
|
||||
|
||||
import (
|
||||
"context"
|
||||
"database/sql"
|
||||
"errors"
|
||||
|
||||
"rag-local/kb/consts"
|
||||
"rag-local/kb/model/entity"
|
||||
|
||||
"github.com/gogf/gf/v2/frame/g"
|
||||
"github.com/gogf/gf/v2/os/gtime"
|
||||
)
|
||||
|
||||
var ParseTask = &parseTaskDao{}
|
||||
|
||||
type parseTaskDao struct{}
|
||||
|
||||
func init() {
|
||||
ctx := context.Background()
|
||||
_, err := g.DB(consts.DbGroupDefault).Exec(ctx, `CREATE TABLE IF NOT EXISTS `+consts.TableNameParseTask+` (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
document_id INTEGER NOT NULL DEFAULT 0,
|
||||
dataset_id INTEGER NOT NULL DEFAULT 0,
|
||||
task_type TEXT NOT NULL DEFAULT 'parse',
|
||||
status INTEGER NOT NULL DEFAULT 0,
|
||||
error_msg TEXT NOT NULL DEFAULT '',
|
||||
created_at DATETIME DEFAULT (datetime('now','localtime')),
|
||||
updated_at DATETIME DEFAULT (datetime('now','localtime'))
|
||||
)`)
|
||||
if err != nil {
|
||||
g.Log().Warningf(ctx, "create kb_parse_task table failed: %v", err)
|
||||
}
|
||||
if _, err := g.DB(consts.DbGroupDefault).Exec(ctx, "CREATE INDEX IF NOT EXISTS idx_kb_parse_task_status ON "+consts.TableNameParseTask+"(status)"); err != nil {
|
||||
g.Log().Warningf(ctx, "create index idx_kb_parse_task_status failed: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func (d *parseTaskDao) GetOne(ctx context.Context, id int64) (*entity.ParseTask, error) {
|
||||
var m entity.ParseTask
|
||||
err := g.DB(consts.DbGroupDefault).Model(consts.TableNameParseTask).Ctx(ctx).Where("id", id).Scan(&m)
|
||||
if errors.Is(err, sql.ErrNoRows) {
|
||||
return nil, nil
|
||||
}
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return &m, nil
|
||||
}
|
||||
|
||||
func (d *parseTaskDao) List(ctx context.Context, page, pageSize int) ([]*entity.ParseTask, int, error) {
|
||||
if page < 1 {
|
||||
page = 1
|
||||
}
|
||||
if pageSize < 1 {
|
||||
pageSize = 20
|
||||
}
|
||||
total, err := g.DB(consts.DbGroupDefault).Model(consts.TableNameParseTask).Ctx(ctx).Count()
|
||||
if err != nil {
|
||||
return nil, 0, err
|
||||
}
|
||||
var list []*entity.ParseTask
|
||||
err = g.DB(consts.DbGroupDefault).Model(consts.TableNameParseTask).Ctx(ctx).
|
||||
Page(page, pageSize).OrderDesc("id").Scan(&list)
|
||||
return list, total, err
|
||||
}
|
||||
|
||||
func (d *parseTaskDao) Insert(ctx context.Context, documentId, datasetId int64, taskType string) (int64, error) {
|
||||
now := gtime.Now().Format("Y-m-d H:i:s")
|
||||
r, err := g.DB(consts.DbGroupDefault).Model(consts.TableNameParseTask).Ctx(ctx).Data(g.Map{
|
||||
"document_id": documentId,
|
||||
"dataset_id": datasetId,
|
||||
"task_type": taskType,
|
||||
"status": consts.TaskStatusPending,
|
||||
"created_at": now,
|
||||
"updated_at": now,
|
||||
}).Insert()
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
return r.LastInsertId()
|
||||
}
|
||||
|
||||
func (d *parseTaskDao) UpdateStatus(ctx context.Context, id int64, status int, errorMsg string) error {
|
||||
_, err := g.DB(consts.DbGroupDefault).Model(consts.TableNameParseTask).Ctx(ctx).Data(g.Map{
|
||||
"status": status,
|
||||
"error_msg": errorMsg,
|
||||
"updated_at": gtime.Now().Format("Y-m-d H:i:s"),
|
||||
}).Where("id", id).Update()
|
||||
return err
|
||||
}
|
||||
|
||||
func (d *parseTaskDao) DeleteByDocument(ctx context.Context, documentId int64) error {
|
||||
_, err := g.DB(consts.DbGroupDefault).Model(consts.TableNameParseTask).Ctx(ctx).
|
||||
Where("document_id", documentId).Delete()
|
||||
return err
|
||||
}
|
||||
|
||||
func (d *parseTaskDao) NextPending(ctx context.Context) (*entity.ParseTask, error) {
|
||||
var m entity.ParseTask
|
||||
err := g.DB(consts.DbGroupDefault).Model(consts.TableNameParseTask).Ctx(ctx).
|
||||
Where("status", consts.TaskStatusPending).OrderAsc("id").Scan(&m)
|
||||
if errors.Is(err, sql.ErrNoRows) {
|
||||
return nil, nil
|
||||
}
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return &m, nil
|
||||
}
|
||||
@@ -0,0 +1,57 @@
|
||||
package dao
|
||||
|
||||
import (
|
||||
"context"
|
||||
|
||||
"rag-local/common"
|
||||
"rag-local/kb/consts"
|
||||
|
||||
"github.com/gogf/gf/v2/database/gdb"
|
||||
"github.com/gogf/gf/v2/frame/g"
|
||||
"github.com/gogf/gf/v2/os/gcache"
|
||||
)
|
||||
|
||||
var SystemConfig = &systemConfigDao{}
|
||||
|
||||
type systemConfigDao struct{}
|
||||
|
||||
func init() {
|
||||
ctx := context.Background()
|
||||
_, err := g.DB(consts.DbGroupSystem).Exec(ctx, `CREATE TABLE IF NOT EXISTS `+consts.TableNameSystemConfig+` (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
cfg_key TEXT NOT NULL DEFAULT '',
|
||||
cfg_value TEXT NOT NULL DEFAULT '',
|
||||
updated_at DATETIME DEFAULT (datetime('now','localtime'))
|
||||
)`)
|
||||
if err != nil {
|
||||
g.Log().Warningf(ctx, "create system_config table failed: %v", err)
|
||||
}
|
||||
if _, err := g.DB(consts.DbGroupSystem).Exec(ctx, "CREATE UNIQUE INDEX IF NOT EXISTS idx_system_config_key ON "+consts.TableNameSystemConfig+"(cfg_key)"); err != nil {
|
||||
g.Log().Warningf(ctx, "create index idx_system_config_key failed: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func (d *systemConfigDao) Get(ctx context.Context, key string) (string, error) {
|
||||
r, err := g.DB(consts.DbGroupSystem).Model(consts.TableNameSystemConfig).Ctx(ctx).
|
||||
Cache(gdb.CacheOption{Duration: common.CacheTTL(), Name: "system_config_Get_" + key}).
|
||||
Fields("cfg_value").Where("cfg_key", key).One()
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
if r == nil {
|
||||
return "", nil
|
||||
}
|
||||
return r["cfg_value"].String(), nil
|
||||
}
|
||||
|
||||
func (d *systemConfigDao) Set(ctx context.Context, key, value string) error {
|
||||
_, err := g.DB(consts.DbGroupSystem).Exec(ctx,
|
||||
"INSERT INTO "+consts.TableNameSystemConfig+" (cfg_key, cfg_value, updated_at) VALUES (?, ?, datetime('now','localtime')) "+
|
||||
"ON CONFLICT(cfg_key) DO UPDATE SET cfg_value=excluded.cfg_value, updated_at=datetime('now','localtime')",
|
||||
key, value)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
_, _ = gcache.Remove(ctx, "system_config_Get_"+key)
|
||||
return nil
|
||||
}
|
||||
@@ -0,0 +1,66 @@
|
||||
package domain
|
||||
|
||||
import (
|
||||
"strconv"
|
||||
"strings"
|
||||
|
||||
"rag-local/kb/model/entity"
|
||||
)
|
||||
|
||||
// VecHit 向量 KNN 命中（vec0 默认 L2 距离）
|
||||
type VecHit struct {
|
||||
ChunkId int64
|
||||
Distance float64
|
||||
}
|
||||
|
||||
// FtsHit 全文检索命中（bm25 负分，越小越相关）
|
||||
type FtsHit struct {
|
||||
ChunkId int64
|
||||
Score float64
|
||||
}
|
||||
|
||||
// RetrievedChunk 混合检索融合后的结果
|
||||
type RetrievedChunk struct {
|
||||
Chunk *entity.Chunk
|
||||
Score float64
|
||||
// Sources 命中来源："vector" / "fts" / "hybrid"
|
||||
Sources []string
|
||||
}
|
||||
|
||||
// HybridResult 混合检索整体结果（chunk + 知识图谱三元组，M5 扩展）
|
||||
type HybridResult struct {
|
||||
Chunks []*RetrievedChunk
|
||||
}
|
||||
|
||||
// Citation 回答引用来源（与回答文本中 [1][2] 编号对应）
|
||||
type Citation struct {
|
||||
Index int `json:"index"`
|
||||
DocumentId int64 `json:"document_id"`
|
||||
ChunkId int64 `json:"chunk_id"`
|
||||
Content string `json:"content"`
|
||||
Score float64 `json:"score"`
|
||||
Sources []string `json:"sources"`
|
||||
}
|
||||
|
||||
// VecJson 向量 JSON 序列化（[0.1,0.2,...]）
|
||||
func VecJson(vec []float32) string {
|
||||
return vecJsonFloat(vec)
|
||||
}
|
||||
|
||||
// VecJsonF64 float64 向量 JSON 序列化（embedding 接口返回 float64）
|
||||
func VecJsonF64(vec []float64) string {
|
||||
return vecJsonFloat(vec)
|
||||
}
|
||||
|
||||
func vecJsonFloat[T float32 | float64](vec []T) string {
|
||||
var sb strings.Builder
|
||||
sb.WriteByte('[')
|
||||
for i, v := range vec {
|
||||
if i > 0 {
|
||||
sb.WriteByte(',')
|
||||
}
|
||||
sb.WriteString(strconv.FormatFloat(float64(v), 'f', -1, 32))
|
||||
}
|
||||
sb.WriteByte(']')
|
||||
return sb.String()
|
||||
}
|
||||
@@ -0,0 +1,29 @@
|
||||
package dto
|
||||
|
||||
import (
|
||||
"rag-local/kb/model/entity"
|
||||
|
||||
"github.com/gogf/gf/v2/frame/g"
|
||||
)
|
||||
|
||||
type ListChunkReq struct {
|
||||
g.Meta `path:"/list" method:"get" tags:"分块" summary:"分块列表"`
|
||||
DocumentId int64 `v:"required" json:"document_id"`
|
||||
Page int `json:"page"`
|
||||
PageSize int `json:"page_size"`
|
||||
}
|
||||
|
||||
type ListChunkRes struct {
|
||||
List []*entity.Chunk `json:"list"`
|
||||
Total int `json:"total"`
|
||||
Page int `json:"page"`
|
||||
PageSize int `json:"page_size"`
|
||||
}
|
||||
|
||||
type UpdateChunkReq struct {
|
||||
g.Meta `path:"/update" method:"post" tags:"分块" summary:"编辑分块（改文本后重新向量化）"`
|
||||
Id int64 `v:"required" json:"id"`
|
||||
Content string `v:"required" json:"content"`
|
||||
}
|
||||
|
||||
type UpdateChunkRes struct{}
|
||||
@@ -0,0 +1,33 @@
|
||||
package dto
|
||||
|
||||
import (
|
||||
"rag-local/kb/model/entity"
|
||||
|
||||
"github.com/gogf/gf/v2/frame/g"
|
||||
)
|
||||
|
||||
type ListConversationReq struct {
|
||||
g.Meta `path:"/list" method:"get" tags:"会话" summary:"会话列表"`
|
||||
}
|
||||
|
||||
type ListConversationRes struct {
|
||||
List []*entity.Conversation `json:"list"`
|
||||
}
|
||||
|
||||
type SaveConversationReq struct {
|
||||
g.Meta `path:"/save" method:"post" tags:"会话" summary:"保存会话"`
|
||||
Id int64 `json:"id"`
|
||||
DatasetId int64 `json:"dataset_id"`
|
||||
Title string `json:"title"`
|
||||
}
|
||||
|
||||
type SaveConversationRes struct {
|
||||
Id int64 `json:"id"`
|
||||
}
|
||||
|
||||
type DeleteConversationReq struct {
|
||||
g.Meta `path:"/delete" method:"post" tags:"会话" summary:"删除会话"`
|
||||
Id int64 `v:"required" json:"id"`
|
||||
}
|
||||
|
||||
type DeleteConversationRes struct{}
|
||||
@@ -0,0 +1,34 @@
|
||||
package dto
|
||||
|
||||
import (
|
||||
"rag-local/kb/model/entity"
|
||||
|
||||
"github.com/gogf/gf/v2/frame/g"
|
||||
)
|
||||
|
||||
type ListDatasetReq struct {
|
||||
g.Meta `path:"/list" method:"get" tags:"数据集" summary:"数据集列表"`
|
||||
}
|
||||
|
||||
type ListDatasetRes struct {
|
||||
List []*entity.Dataset `json:"list"`
|
||||
}
|
||||
|
||||
type SaveDatasetReq struct {
|
||||
g.Meta `path:"/save" method:"post" tags:"数据集" summary:"保存数据集"`
|
||||
Id int64 `json:"id"`
|
||||
Name string `v:"required" json:"name"`
|
||||
Description string `json:"description"`
|
||||
EmbeddingCfgId int64 `json:"embedding_cfg_id"`
|
||||
}
|
||||
|
||||
type SaveDatasetRes struct {
|
||||
Id int64 `json:"id"`
|
||||
}
|
||||
|
||||
type DeleteDatasetReq struct {
|
||||
g.Meta `path:"/delete" method:"post" tags:"数据集" summary:"删除数据集"`
|
||||
Id int64 `v:"required" json:"id"`
|
||||
}
|
||||
|
||||
type DeleteDatasetRes struct{}
|
||||
@@ -0,0 +1,46 @@
|
||||
package dto
|
||||
|
||||
import (
|
||||
"rag-local/kb/model/entity"
|
||||
|
||||
"github.com/gogf/gf/v2/frame/g"
|
||||
"github.com/gogf/gf/v2/net/ghttp"
|
||||
)
|
||||
|
||||
type UploadDocumentReq struct {
|
||||
g.Meta `path:"/upload" method:"post" tags:"文档" summary:"上传文档"`
|
||||
DatasetId int64 `json:"dataset_id"`
|
||||
File *ghttp.UploadFile `json:"file" dc:"文档文件"`
|
||||
}
|
||||
|
||||
type UploadDocumentRes struct {
|
||||
Id int64 `json:"id"`
|
||||
}
|
||||
|
||||
type ListDocumentReq struct {
|
||||
g.Meta `path:"/list" method:"get" tags:"文档" summary:"文档列表"`
|
||||
DatasetId int64 `v:"required" json:"dataset_id"`
|
||||
Page int `json:"page"`
|
||||
PageSize int `json:"page_size"`
|
||||
}
|
||||
|
||||
type ListDocumentRes struct {
|
||||
List []*entity.Document `json:"list"`
|
||||
Total int `json:"total"`
|
||||
Page int `json:"page"`
|
||||
PageSize int `json:"page_size"`
|
||||
}
|
||||
|
||||
type DeleteDocumentReq struct {
|
||||
g.Meta `path:"/delete" method:"post" tags:"文档" summary:"删除文档"`
|
||||
Id int64 `v:"required" json:"id"`
|
||||
}
|
||||
|
||||
type DeleteDocumentRes struct{}
|
||||
|
||||
type ReembedDocumentReq struct {
|
||||
g.Meta `path:"/reembed" method:"post" tags:"文档" summary:"重新向量化"`
|
||||
Id int64 `v:"required" json:"id"`
|
||||
}
|
||||
|
||||
type ReembedDocumentRes struct{}
|
||||
@@ -0,0 +1,21 @@
|
||||
package dto
|
||||
|
||||
import (
|
||||
"rag-local/kb/model/entity"
|
||||
|
||||
"github.com/gogf/gf/v2/frame/g"
|
||||
)
|
||||
|
||||
type ListKgEntityReq struct {
|
||||
g.Meta `path:"/list" method:"get" tags:"知识图谱" summary:"实体列表"`
|
||||
DatasetId int64 `json:"dataset_id"`
|
||||
Page int `json:"page"`
|
||||
PageSize int `json:"page_size"`
|
||||
}
|
||||
|
||||
type ListKgEntityRes struct {
|
||||
List []*entity.KgEntity `json:"list"`
|
||||
Total int `json:"total"`
|
||||
Page int `json:"page"`
|
||||
PageSize int `json:"page_size"`
|
||||
}
|
||||
@@ -0,0 +1,21 @@
|
||||
package dto
|
||||
|
||||
import (
|
||||
"rag-local/kb/model/entity"
|
||||
|
||||
"github.com/gogf/gf/v2/frame/g"
|
||||
)
|
||||
|
||||
type ListKgRelationReq struct {
|
||||
g.Meta `path:"/list" method:"get" tags:"知识图谱" summary:"关系列表"`
|
||||
DatasetId int64 `json:"dataset_id"`
|
||||
Page int `json:"page"`
|
||||
PageSize int `json:"page_size"`
|
||||
}
|
||||
|
||||
type ListKgRelationRes struct {
|
||||
List []*entity.KgRelation `json:"list"`
|
||||
Total int `json:"total"`
|
||||
Page int `json:"page"`
|
||||
PageSize int `json:"page_size"`
|
||||
}
|
||||
@@ -0,0 +1,26 @@
|
||||
package dto
|
||||
|
||||
import (
|
||||
"rag-local/kb/model/entity"
|
||||
|
||||
"github.com/gogf/gf/v2/frame/g"
|
||||
)
|
||||
|
||||
type ListMessageReq struct {
|
||||
g.Meta `path:"/list" method:"get" tags:"消息" summary:"历史消息"`
|
||||
ConversationId int64 `v:"required" json:"conversation_id"`
|
||||
}
|
||||
|
||||
type ListMessageRes struct {
|
||||
List []*entity.Message `json:"list"`
|
||||
}
|
||||
|
||||
// ChatReq RAG 问答（SSE 流式响应，非标准 JSON 包装）
|
||||
type ChatReq struct {
|
||||
g.Meta `path:"/chat" method:"post" tags:"消息" summary:"RAG 问答（SSE 流式）"`
|
||||
ConversationId int64 `json:"conversation_id"`
|
||||
DatasetId int64 `json:"dataset_id"`
|
||||
Question string `v:"required" json:"question"`
|
||||
}
|
||||
|
||||
type ChatRes struct{}
|
||||
@@ -0,0 +1,49 @@
|
||||
package dto
|
||||
|
||||
import (
|
||||
"rag-local/kb/model/entity"
|
||||
|
||||
"github.com/gogf/gf/v2/frame/g"
|
||||
)
|
||||
|
||||
type ListModelConfigReq struct {
|
||||
g.Meta `path:"/list" method:"get" tags:"模型配置" summary:"模型配置列表"`
|
||||
ModelType string `json:"model_type"`
|
||||
}
|
||||
|
||||
type ListModelConfigRes struct {
|
||||
List []*entity.ModelConfig `json:"list"`
|
||||
}
|
||||
|
||||
type SaveModelConfigReq struct {
|
||||
g.Meta `path:"/save" method:"post" tags:"模型配置" summary:"保存模型配置"`
|
||||
Id int64 `json:"id"`
|
||||
Name string `v:"required" json:"name"`
|
||||
ModelType string `v:"required|in:chat,embedding" json:"model_type"`
|
||||
ModelName string `v:"required" json:"model_name"`
|
||||
EndpointUrl string `json:"endpoint_url"`
|
||||
ApiKey string `json:"api_key"`
|
||||
Dimension int `json:"dimension"`
|
||||
Extra string `json:"extra"`
|
||||
}
|
||||
|
||||
type SaveModelConfigRes struct {
|
||||
Id int64 `json:"id"`
|
||||
}
|
||||
|
||||
type DeleteModelConfigReq struct {
|
||||
g.Meta `path:"/delete" method:"post" tags:"模型配置" summary:"删除模型配置"`
|
||||
Id int64 `v:"required" json:"id"`
|
||||
}
|
||||
|
||||
type DeleteModelConfigRes struct{}
|
||||
|
||||
type TestModelConfigReq struct {
|
||||
g.Meta `path:"/test" method:"post" tags:"模型配置" summary:"连通性测试"`
|
||||
Id int64 `v:"required" json:"id"`
|
||||
}
|
||||
|
||||
type TestModelConfigRes struct {
|
||||
Ok bool `json:"ok"`
|
||||
Msg string `json:"msg"`
|
||||
}
|
||||
@@ -0,0 +1,27 @@
|
||||
package dto
|
||||
|
||||
import (
|
||||
"rag-local/kb/model/entity"
|
||||
|
||||
"github.com/gogf/gf/v2/frame/g"
|
||||
)
|
||||
|
||||
type ListParseTaskReq struct {
|
||||
g.Meta `path:"/list" method:"get" tags:"解析任务" summary:"解析任务列表"`
|
||||
Page int `json:"page"`
|
||||
PageSize int `json:"page_size"`
|
||||
}
|
||||
|
||||
type ListParseTaskRes struct {
|
||||
List []*entity.ParseTask `json:"list"`
|
||||
Total int `json:"total"`
|
||||
Page int `json:"page"`
|
||||
PageSize int `json:"page_size"`
|
||||
}
|
||||
|
||||
type RetryParseTaskReq struct {
|
||||
g.Meta `path:"/retry" method:"post" tags:"解析任务" summary:"重试失败任务"`
|
||||
Id int64 `v:"required" json:"id"`
|
||||
}
|
||||
|
||||
type RetryParseTaskRes struct{}
|
||||
@@ -0,0 +1,45 @@
|
||||
package dto
|
||||
|
||||
import "github.com/gogf/gf/v2/frame/g"
|
||||
|
||||
type LoginReq struct {
|
||||
g.Meta `path:"/login" method:"post" tags:"系统配置" summary:"访问令牌登录"`
|
||||
Token string `v:"required" json:"token"`
|
||||
}
|
||||
|
||||
type LoginRes struct {
|
||||
Token string `json:"token"`
|
||||
}
|
||||
|
||||
type GetSystemConfigReq struct {
|
||||
g.Meta `path:"/" method:"get" tags:"系统配置" summary:"获取系统设置"`
|
||||
}
|
||||
|
||||
type GetSystemConfigRes struct {
|
||||
DefaultChatModel int64 `json:"default_chat_model"`
|
||||
DefaultDataset int64 `json:"default_dataset"`
|
||||
}
|
||||
|
||||
type UpdateSystemConfigReq struct {
|
||||
g.Meta `path:"/" method:"put" tags:"系统配置" summary:"更新系统设置"`
|
||||
DefaultChatModel int64 `json:"default_chat_model"`
|
||||
DefaultDataset int64 `json:"default_dataset"`
|
||||
}
|
||||
|
||||
type UpdateSystemConfigRes struct{}
|
||||
|
||||
type RegenerateTokenReq struct {
|
||||
g.Meta `path:"/regenerate-token" method:"post" tags:"系统配置" summary:"重新生成访问令牌"`
|
||||
}
|
||||
|
||||
type RegenerateTokenRes struct {
|
||||
Token string `json:"token"`
|
||||
}
|
||||
|
||||
type GetTokenReq struct {
|
||||
g.Meta `path:"/token" method:"get" tags:"系统配置" summary:"获取当前访问令牌"`
|
||||
}
|
||||
|
||||
type GetTokenRes struct {
|
||||
Token string `json:"token"`
|
||||
}
|
||||
@@ -0,0 +1,13 @@
|
||||
package entity
|
||||
|
||||
import "github.com/gogf/gf/v2/os/gtime"
|
||||
|
||||
type Chunk struct {
|
||||
Id int64 `orm:"id" json:"id"`
|
||||
DatasetId int64 `orm:"dataset_id" json:"dataset_id"`
|
||||
DocumentId int64 `orm:"document_id" json:"document_id"`
|
||||
Seq int `orm:"seq" json:"seq"`
|
||||
Content string `orm:"content" json:"content"`
|
||||
Meta string `orm:"meta" json:"meta"`
|
||||
CreatedAt *gtime.Time `orm:"created_at" json:"created_at"`
|
||||
}
|
||||
@@ -0,0 +1,11 @@
|
||||
package entity
|
||||
|
||||
import "github.com/gogf/gf/v2/os/gtime"
|
||||
|
||||
type Conversation struct {
|
||||
Id int64 `orm:"id" json:"id"`
|
||||
DatasetId int64 `orm:"dataset_id" json:"dataset_id"`
|
||||
Title string `orm:"title" json:"title"`
|
||||
CreatedAt *gtime.Time `orm:"created_at" json:"created_at"`
|
||||
UpdatedAt *gtime.Time `orm:"updated_at" json:"updated_at"`
|
||||
}
|
||||
@@ -0,0 +1,13 @@
|
||||
package entity
|
||||
|
||||
import "github.com/gogf/gf/v2/os/gtime"
|
||||
|
||||
type Dataset struct {
|
||||
Id int64 `orm:"id" json:"id"`
|
||||
Name string `orm:"name" json:"name"`
|
||||
Description string `orm:"description" json:"description"`
|
||||
EmbeddingCfgId int64 `orm:"embedding_cfg_id" json:"embedding_cfg_id"`
|
||||
Status int `orm:"status" json:"status"`
|
||||
CreatedAt *gtime.Time `orm:"created_at" json:"created_at"`
|
||||
UpdatedAt *gtime.Time `orm:"updated_at" json:"updated_at"`
|
||||
}
|
||||
@@ -0,0 +1,17 @@
|
||||
package entity
|
||||
|
||||
import "github.com/gogf/gf/v2/os/gtime"
|
||||
|
||||
type Document struct {
|
||||
Id int64 `orm:"id" json:"id"`
|
||||
DatasetId int64 `orm:"dataset_id" json:"dataset_id"`
|
||||
Filename string `orm:"filename" json:"filename"`
|
||||
FilePath string `orm:"file_path" json:"file_path"`
|
||||
FileSize int64 `orm:"file_size" json:"file_size"`
|
||||
FileType string `orm:"file_type" json:"file_type"`
|
||||
Status int `orm:"status" json:"status"`
|
||||
ChunkCount int `orm:"chunk_count" json:"chunk_count"`
|
||||
ErrorMsg string `orm:"error_msg" json:"error_msg"`
|
||||
CreatedAt *gtime.Time `orm:"created_at" json:"created_at"`
|
||||
UpdatedAt *gtime.Time `orm:"updated_at" json:"updated_at"`
|
||||
}
|
||||
@@ -0,0 +1,13 @@
|
||||
package entity
|
||||
|
||||
import "github.com/gogf/gf/v2/os/gtime"
|
||||
|
||||
type KgEntity struct {
|
||||
Id int64 `orm:"id" json:"id"`
|
||||
DatasetId int64 `orm:"dataset_id" json:"dataset_id"`
|
||||
Name string `orm:"name" json:"name"`
|
||||
EntityType string `orm:"entity_type" json:"entity_type"`
|
||||
ChunkId int64 `orm:"chunk_id" json:"chunk_id"`
|
||||
CreatedAt *gtime.Time `orm:"created_at" json:"created_at"`
|
||||
UpdatedAt *gtime.Time `orm:"updated_at" json:"updated_at"`
|
||||
}
|
||||
@@ -0,0 +1,13 @@
|
||||
package entity
|
||||
|
||||
import "github.com/gogf/gf/v2/os/gtime"
|
||||
|
||||
type KgRelation struct {
|
||||
Id int64 `orm:"id" json:"id"`
|
||||
DatasetId int64 `orm:"dataset_id" json:"dataset_id"`
|
||||
Head string `orm:"head" json:"head"`
|
||||
Relation string `orm:"relation" json:"relation"`
|
||||
Tail string `orm:"tail" json:"tail"`
|
||||
ChunkId int64 `orm:"chunk_id" json:"chunk_id"`
|
||||
CreatedAt *gtime.Time `orm:"created_at" json:"created_at"`
|
||||
}
|
||||
@@ -0,0 +1,12 @@
|
||||
package entity
|
||||
|
||||
import "github.com/gogf/gf/v2/os/gtime"
|
||||
|
||||
type Message struct {
|
||||
Id int64 `orm:"id" json:"id"`
|
||||
ConversationId int64 `orm:"conversation_id" json:"conversation_id"`
|
||||
Role string `orm:"role" json:"role"`
|
||||
Content string `orm:"content" json:"content"`
|
||||
Citations string `orm:"citations" json:"citations"`
|
||||
CreatedAt *gtime.Time `orm:"created_at" json:"created_at"`
|
||||
}
|
||||
@@ -0,0 +1,16 @@
|
||||
package entity
|
||||
|
||||
import "github.com/gogf/gf/v2/os/gtime"
|
||||
|
||||
type ModelConfig struct {
|
||||
Id int64 `orm:"id" json:"id"`
|
||||
Name string `orm:"name" json:"name"`
|
||||
ModelType string `orm:"model_type" json:"model_type"`
|
||||
ModelName string `orm:"model_name" json:"model_name"`
|
||||
EndpointUrl string `orm:"endpoint_url" json:"endpoint_url"`
|
||||
ApiKey string `orm:"api_key" json:"api_key"`
|
||||
Dimension int `orm:"dimension" json:"dimension"`
|
||||
Extra string `orm:"extra" json:"extra"`
|
||||
CreatedAt *gtime.Time `orm:"created_at" json:"created_at"`
|
||||
UpdatedAt *gtime.Time `orm:"updated_at" json:"updated_at"`
|
||||
}
|
||||
@@ -0,0 +1,14 @@
|
||||
package entity
|
||||
|
||||
import "github.com/gogf/gf/v2/os/gtime"
|
||||
|
||||
type ParseTask struct {
|
||||
Id int64 `orm:"id" json:"id"`
|
||||
DocumentId int64 `orm:"document_id" json:"document_id"`
|
||||
DatasetId int64 `orm:"dataset_id" json:"dataset_id"`
|
||||
TaskType string `orm:"task_type" json:"task_type"`
|
||||
Status int `orm:"status" json:"status"`
|
||||
ErrorMsg string `orm:"error_msg" json:"error_msg"`
|
||||
CreatedAt *gtime.Time `orm:"created_at" json:"created_at"`
|
||||
UpdatedAt *gtime.Time `orm:"updated_at" json:"updated_at"`
|
||||
}
|
||||
@@ -0,0 +1,10 @@
|
||||
package entity
|
||||
|
||||
import "github.com/gogf/gf/v2/os/gtime"
|
||||
|
||||
type SystemConfig struct {
|
||||
Id int64 `orm:"id" json:"id"`
|
||||
CfgKey string `orm:"cfg_key" json:"cfg_key"`
|
||||
CfgValue string `orm:"cfg_value" json:"cfg_value"`
|
||||
UpdatedAt *gtime.Time `orm:"updated_at" json:"updated_at"`
|
||||
}
|
||||
@@ -0,0 +1,476 @@
|
||||
package service
|
||||
|
||||
import (
|
||||
"bufio"
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"net/http"
|
||||
"sort"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"rag-local/common"
|
||||
"rag-local/kb/consts"
|
||||
"rag-local/kb/dao"
|
||||
"rag-local/kb/model/domain"
|
||||
"rag-local/kb/model/entity"
|
||||
|
||||
eembedding "github.com/cloudwego/eino/components/embedding"
|
||||
emodel "github.com/cloudwego/eino/components/model"
|
||||
eretriever "github.com/cloudwego/eino/components/retriever"
|
||||
"github.com/cloudwego/eino/schema"
|
||||
"github.com/gogf/gf/v2/errors/gerror"
|
||||
"github.com/gogf/gf/v2/frame/g"
|
||||
)
|
||||
|
||||
var httpClient = &http.Client{Timeout: 2 * time.Minute}
|
||||
|
||||
// ---------- OpenAI 兼容 HTTP 组件 ----------
|
||||
|
||||
type openAIMessage struct {
|
||||
Role string `json:"role"`
|
||||
Content string `json:"content"`
|
||||
}
|
||||
|
||||
type openAIChatResponse struct {
|
||||
Choices []struct {
|
||||
Message openAIMessage `json:"message"`
|
||||
FinishReason string `json:"finish_reason"`
|
||||
} `json:"choices"`
|
||||
Usage *schema.TokenUsage `json:"usage"`
|
||||
}
|
||||
|
||||
type openAIStreamChunk struct {
|
||||
Choices []struct {
|
||||
Delta openAIMessage `json:"delta"`
|
||||
FinishReason string `json:"finish_reason"`
|
||||
} `json:"choices"`
|
||||
}
|
||||
|
||||
// OpenAIChatModel 基于 OpenAI 兼容 /chat/completions 接口的对话模型，实现 eino model.ChatModel
|
||||
type OpenAIChatModel struct {
|
||||
cfg *entity.ModelConfig
|
||||
}
|
||||
|
||||
func NewOpenAIChatModel(cfg *entity.ModelConfig) *OpenAIChatModel {
|
||||
return &OpenAIChatModel{cfg: cfg}
|
||||
}
|
||||
|
||||
func (m *OpenAIChatModel) Generate(ctx context.Context, input []*schema.Message, opts ...emodel.Option) (*schema.Message, error) {
|
||||
payload := map[string]any{
|
||||
"model": m.cfg.ModelName,
|
||||
"messages": buildOpenAIMessages(input),
|
||||
"stream": false,
|
||||
}
|
||||
body, err := postOpenAI(ctx, m.cfg, m.endpoint("/chat/completions"), payload)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
var resp openAIChatResponse
|
||||
if err := json.Unmarshal(body, &resp); err != nil {
|
||||
return nil, gerror.Wrap(err, "解析模型响应失败")
|
||||
}
|
||||
if len(resp.Choices) == 0 {
|
||||
return nil, gerror.New("模型返回空响应")
|
||||
}
|
||||
choice := resp.Choices[0]
|
||||
msg := &schema.Message{Role: schema.Assistant, Content: choice.Message.Content}
|
||||
if choice.FinishReason != "" {
|
||||
msg.ResponseMeta = &schema.ResponseMeta{FinishReason: choice.FinishReason, Usage: resp.Usage}
|
||||
}
|
||||
return msg, nil
|
||||
}
|
||||
|
||||
func (m *OpenAIChatModel) Stream(ctx context.Context, input []*schema.Message, opts ...emodel.Option) (*schema.StreamReader[*schema.Message], error) {
|
||||
payload := map[string]any{
|
||||
"model": m.cfg.ModelName,
|
||||
"messages": buildOpenAIMessages(input),
|
||||
"stream": true,
|
||||
}
|
||||
reader, writer := schema.Pipe[*schema.Message](16)
|
||||
go func() {
|
||||
defer writer.Close()
|
||||
body, err := postOpenAIStream(ctx, m.cfg, m.endpoint("/chat/completions"), payload)
|
||||
if err != nil {
|
||||
writer.Send(nil, err)
|
||||
return
|
||||
}
|
||||
defer body.Close()
|
||||
br := bufio.NewReader(body)
|
||||
for {
|
||||
line, err := br.ReadBytes('\n')
|
||||
text := strings.TrimSpace(string(line))
|
||||
if strings.HasPrefix(text, "data:") {
|
||||
data := strings.TrimSpace(strings.TrimPrefix(text, "data:"))
|
||||
if data == "[DONE]" {
|
||||
break
|
||||
}
|
||||
var chunk openAIStreamChunk
|
||||
if json.Unmarshal([]byte(data), &chunk) == nil && len(chunk.Choices) > 0 {
|
||||
if delta := chunk.Choices[0].Delta.Content; delta != "" {
|
||||
if closed := writer.Send(&schema.Message{Role: schema.Assistant, Content: delta}, nil); closed {
|
||||
return
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
if err != nil {
|
||||
break
|
||||
}
|
||||
}
|
||||
}()
|
||||
return reader, nil
|
||||
}
|
||||
|
||||
func (m *OpenAIChatModel) BindTools(tools []*schema.ToolInfo) error {
|
||||
return nil
|
||||
}
|
||||
|
||||
func (m *OpenAIChatModel) endpoint(path string) string {
|
||||
return strings.TrimRight(m.cfg.EndpointUrl, "/") + path
|
||||
}
|
||||
|
||||
// OpenAIEmbedder 基于 OpenAI 兼容 /embeddings 接口的向量模型，实现 eino embedding.Embedder
|
||||
type OpenAIEmbedder struct {
|
||||
cfg *entity.ModelConfig
|
||||
}
|
||||
|
||||
func NewOpenAIEmbedder(cfg *entity.ModelConfig) *OpenAIEmbedder {
|
||||
return &OpenAIEmbedder{cfg: cfg}
|
||||
}
|
||||
|
||||
// Dim 配置的向量维度（vec0 表建表维度需一致）
|
||||
func (e *OpenAIEmbedder) Dim() int {
|
||||
if e.cfg.Dimension > 0 {
|
||||
return e.cfg.Dimension
|
||||
}
|
||||
return consts.DefaultEmbeddingDim
|
||||
}
|
||||
|
||||
func (e *OpenAIEmbedder) EmbedStrings(ctx context.Context, texts []string, opts ...eembedding.Option) ([][]float64, error) {
|
||||
payload := map[string]any{"model": e.cfg.ModelName, "input": texts}
|
||||
body, err := postOpenAI(ctx, e.cfg, strings.TrimRight(e.cfg.EndpointUrl, "/")+"/embeddings", payload)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
var resp struct {
|
||||
Data []struct {
|
||||
Embedding []float64 `json:"embedding"`
|
||||
Index int `json:"index"`
|
||||
} `json:"data"`
|
||||
}
|
||||
if err := json.Unmarshal(body, &resp); err != nil {
|
||||
return nil, gerror.Wrap(err, "解析 embedding 响应失败")
|
||||
}
|
||||
if len(resp.Data) == 0 {
|
||||
return nil, gerror.New("embedding 接口返回空数据")
|
||||
}
|
||||
out := make([][]float64, len(texts))
|
||||
for _, d := range resp.Data {
|
||||
if d.Index >= 0 && d.Index < len(out) {
|
||||
out[d.Index] = d.Embedding
|
||||
}
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// BuildChatModel 按配置 id 构建对话模型
|
||||
func BuildChatModel(ctx context.Context, cfgId int64) (*OpenAIChatModel, error) {
|
||||
cfg, err := dao.ModelConfig.GetOne(ctx, cfgId)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if cfg == nil {
|
||||
return nil, gerror.New("模型配置不存在")
|
||||
}
|
||||
if cfg.ModelType != consts.ModelTypeChat {
|
||||
return nil, gerror.New("该模型配置不是对话模型（model_type=chat）")
|
||||
}
|
||||
if cfg.EndpointUrl == "" || cfg.ModelName == "" {
|
||||
return nil, gerror.New("模型配置缺少 endpoint_url 或 model_name")
|
||||
}
|
||||
return NewOpenAIChatModel(cfg), nil
|
||||
}
|
||||
|
||||
// BuildEmbedder 按配置 id 构建向量模型
|
||||
func BuildEmbedder(ctx context.Context, cfgId int64) (*OpenAIEmbedder, error) {
|
||||
cfg, err := dao.ModelConfig.GetOne(ctx, cfgId)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if cfg == nil {
|
||||
return nil, gerror.New("模型配置不存在")
|
||||
}
|
||||
if cfg.ModelType != consts.ModelTypeEmbedding {
|
||||
return nil, gerror.New("该模型配置不是向量模型（model_type=embedding）")
|
||||
}
|
||||
if cfg.EndpointUrl == "" || cfg.ModelName == "" {
|
||||
return nil, gerror.New("模型配置缺少 endpoint_url 或 model_name")
|
||||
}
|
||||
return NewOpenAIEmbedder(cfg), nil
|
||||
}
|
||||
|
||||
// HybridRetriever 混合检索器：向量 KNN + FTS5 BM25，RRF 融合，实现 eino retriever.Retriever
|
||||
type HybridRetriever struct {
|
||||
embedder eembedding.Embedder
|
||||
datasetId int64
|
||||
}
|
||||
|
||||
func NewHybridRetriever(embedder eembedding.Embedder, datasetId int64) *HybridRetriever {
|
||||
return &HybridRetriever{embedder: embedder, datasetId: datasetId}
|
||||
}
|
||||
|
||||
func (r *HybridRetriever) Retrieve(ctx context.Context, query string, opts ...eretriever.Option) ([]*schema.Document, error) {
|
||||
o := eretriever.GetCommonOptions(nil, opts...)
|
||||
topK := consts.HybridTopK
|
||||
if o.TopK != nil && *o.TopK > 0 {
|
||||
topK = *o.TopK
|
||||
}
|
||||
|
||||
scores := make(map[int64]float64)
|
||||
srcs := make(map[int64][]string)
|
||||
|
||||
if r.embedder != nil {
|
||||
vecs, err := r.embedder.EmbedStrings(ctx, []string{query})
|
||||
if err != nil {
|
||||
g.Log().Warningf(ctx, "query embed failed: %v", err)
|
||||
} else if len(vecs) > 0 {
|
||||
hits, err := dao.Chunk.VecSearch(ctx, r.datasetId, domain.VecJsonF64(vecs[0]), consts.VectorTopK)
|
||||
if err != nil {
|
||||
g.Log().Warningf(ctx, "vec search failed: %v", err)
|
||||
} else {
|
||||
for i, h := range hits {
|
||||
scores[h.ChunkId] += 1 / (float64(consts.RrfK) + float64(i) + 1)
|
||||
srcs[h.ChunkId] = append(srcs[h.ChunkId], "vector")
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
ftsHits, err := dao.Chunk.FtsSearch(ctx, r.datasetId, common.TokenizeQuery(query), consts.FtsTopK)
|
||||
if err != nil {
|
||||
g.Log().Warningf(ctx, "fts search failed: %v", err)
|
||||
} else {
|
||||
for i, h := range ftsHits {
|
||||
scores[h.ChunkId] += 1 / (float64(consts.RrfK) + float64(i) + 1)
|
||||
srcs[h.ChunkId] = append(srcs[h.ChunkId], "fts")
|
||||
}
|
||||
}
|
||||
|
||||
type scoredChunk struct {
|
||||
id int64
|
||||
score float64
|
||||
sources []string
|
||||
}
|
||||
items := make([]scoredChunk, 0, len(scores))
|
||||
for id, s := range scores {
|
||||
items = append(items, scoredChunk{id: id, score: s, sources: srcs[id]})
|
||||
}
|
||||
sort.Slice(items, func(i, j int) bool { return items[i].score > items[j].score })
|
||||
if len(items) > topK {
|
||||
items = items[:topK]
|
||||
}
|
||||
|
||||
docs := make([]*schema.Document, 0, len(items))
|
||||
for _, it := range items {
|
||||
chunk, err := dao.Chunk.GetOne(ctx, it.id)
|
||||
if err != nil || chunk == nil {
|
||||
continue
|
||||
}
|
||||
docs = append(docs, &schema.Document{
|
||||
ID: strconv.FormatInt(chunk.Id, 10),
|
||||
Content: chunk.Content,
|
||||
MetaData: map[string]any{
|
||||
"chunk_id": chunk.Id,
|
||||
"document_id": chunk.DocumentId,
|
||||
"seq": chunk.Seq,
|
||||
"score": it.score,
|
||||
"sources": it.sources,
|
||||
},
|
||||
})
|
||||
}
|
||||
return docs, nil
|
||||
}
|
||||
|
||||
// ---------- RAG 问答工作流 ----------
|
||||
|
||||
var ChatService = &chatService{}
|
||||
|
||||
type chatService struct{}
|
||||
|
||||
// MaxHistoryRounds 携带进模型的历史对话轮数（每条消息算一条，含用户与助手）
|
||||
const MaxHistoryRounds = 10
|
||||
|
||||
// Ask RAG 问答工作流：混合检索 → 组装提示（含引用编号）→ 对话模型流式生成。
|
||||
// history 需已包含最新一条用户问题；onCitations 在检索完成后先于流式输出回调；onDelta 接收增量文本，均可为 nil。
|
||||
func (s *chatService) Ask(ctx context.Context, datasetId int64, question string, history []*schema.Message, onCitations func([]domain.Citation), onDelta func(string)) (string, []domain.Citation, error) {
|
||||
docs, err := s.retrieve(ctx, datasetId, question)
|
||||
if err != nil {
|
||||
return "", nil, err
|
||||
}
|
||||
citations := buildCitations(docs)
|
||||
if onCitations != nil {
|
||||
onCitations(citations)
|
||||
}
|
||||
|
||||
triples, err := KgRelationService.GraphEnhance(ctx, datasetId, question)
|
||||
if err != nil {
|
||||
g.Log().Warningf(ctx, "graph enhance failed: %v", err)
|
||||
}
|
||||
|
||||
defaultChatModel, _, err := SystemConfigService.GetSettings(ctx)
|
||||
if err != nil {
|
||||
return "", nil, err
|
||||
}
|
||||
if defaultChatModel <= 0 {
|
||||
return "", nil, gerror.New("请先在设置中选择默认对话模型")
|
||||
}
|
||||
model, err := BuildChatModel(ctx, defaultChatModel)
|
||||
if err != nil {
|
||||
return "", nil, err
|
||||
}
|
||||
|
||||
msgs := make([]*schema.Message, 0, len(history)+1)
|
||||
msgs = append(msgs, &schema.Message{Role: schema.System, Content: buildSystemPrompt(citations, triples)})
|
||||
if start := len(history) - MaxHistoryRounds*2; start > 0 {
|
||||
history = history[start:]
|
||||
}
|
||||
msgs = append(msgs, history...)
|
||||
|
||||
sr, err := model.Stream(ctx, msgs)
|
||||
if err != nil {
|
||||
return "", nil, gerror.Wrap(err, "调用对话模型失败")
|
||||
}
|
||||
defer sr.Close()
|
||||
var full strings.Builder
|
||||
for {
|
||||
m, err := sr.Recv()
|
||||
if errors.Is(err, io.EOF) {
|
||||
break
|
||||
}
|
||||
if err != nil {
|
||||
return "", nil, gerror.Wrap(err, "流式输出中断")
|
||||
}
|
||||
full.WriteString(m.Content)
|
||||
if onDelta != nil {
|
||||
onDelta(m.Content)
|
||||
}
|
||||
}
|
||||
return full.String(), citations, nil
|
||||
}
|
||||
|
||||
// retrieve 构建数据集绑定的混合检索器并执行检索（无 embedding 配置时仅全文）
|
||||
func (s *chatService) retrieve(ctx context.Context, datasetId int64, question string) ([]*schema.Document, error) {
|
||||
var emb eembedding.Embedder
|
||||
if cfgId, err := dao.Dataset.GetEmbeddingCfgId(ctx, datasetId); err == nil && cfgId > 0 {
|
||||
if em, err := BuildEmbedder(ctx, cfgId); err == nil {
|
||||
emb = em
|
||||
} else {
|
||||
g.Log().Warningf(ctx, "build embedder failed, retrieve fts only: %v", err)
|
||||
}
|
||||
}
|
||||
return NewHybridRetriever(emb, datasetId).Retrieve(ctx, question)
|
||||
}
|
||||
|
||||
// buildCitations 从检索结果生成引用列表（编号从 1 开始，与提示词 [编号] 对应）
|
||||
func buildCitations(docs []*schema.Document) []domain.Citation {
|
||||
cits := make([]domain.Citation, 0, len(docs))
|
||||
for i, d := range docs {
|
||||
c := domain.Citation{Index: i + 1, Content: d.Content}
|
||||
if id, ok := d.MetaData["chunk_id"].(int64); ok {
|
||||
c.ChunkId = id
|
||||
}
|
||||
if id, ok := d.MetaData["document_id"].(int64); ok {
|
||||
c.DocumentId = id
|
||||
}
|
||||
if s, ok := d.MetaData["score"].(float64); ok {
|
||||
c.Score = s
|
||||
}
|
||||
if srcs, ok := d.MetaData["sources"].([]string); ok {
|
||||
c.Sources = srcs
|
||||
}
|
||||
cits = append(cits, c)
|
||||
}
|
||||
return cits
|
||||
}
|
||||
|
||||
// buildSystemPrompt 系统提示词：引用资料编号 + 检索片段 + 知识图谱三元组（M5 图增强）
|
||||
func buildSystemPrompt(citations []domain.Citation, triples []string) string {
|
||||
var sb strings.Builder
|
||||
sb.WriteString("你是一个本地知识库助手。请仅根据以下资料回答用户问题；若资料不足以回答，请明确说明。")
|
||||
sb.WriteString("回答引用资料时，在对应位置标注 [编号]。\n\n【资料】\n")
|
||||
for _, c := range citations {
|
||||
sb.WriteString(fmt.Sprintf("[%d] %s\n", c.Index, c.Content))
|
||||
}
|
||||
if len(triples) > 0 {
|
||||
sb.WriteString("\n【知识图谱】以下为与问题相关的实体关系，可辅助回答关系类问题：\n")
|
||||
for _, t := range triples {
|
||||
sb.WriteString(t + "\n")
|
||||
}
|
||||
}
|
||||
return sb.String()
|
||||
}
|
||||
|
||||
// ---------- HTTP 辅助 ----------
|
||||
|
||||
func buildOpenAIMessages(input []*schema.Message) []openAIMessage {
|
||||
out := make([]openAIMessage, 0, len(input))
|
||||
for _, m := range input {
|
||||
if m == nil {
|
||||
continue
|
||||
}
|
||||
role := string(m.Role)
|
||||
if role == "" {
|
||||
role = string(schema.User)
|
||||
}
|
||||
out = append(out, openAIMessage{Role: role, Content: m.Content})
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func postOpenAI(ctx context.Context, cfg *entity.ModelConfig, url string, payload any) ([]byte, error) {
|
||||
body, err := doOpenAIRequest(ctx, cfg, url, payload)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer body.Close()
|
||||
resp, err := io.ReadAll(body)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return resp, nil
|
||||
}
|
||||
|
||||
func postOpenAIStream(ctx context.Context, cfg *entity.ModelConfig, url string, payload any) (io.ReadCloser, error) {
|
||||
return doOpenAIRequest(ctx, cfg, url, payload)
|
||||
}
|
||||
|
||||
func doOpenAIRequest(ctx context.Context, cfg *entity.ModelConfig, url string, payload any) (io.ReadCloser, error) {
|
||||
buf, err := json.Marshal(payload)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
req, err := http.NewRequestWithContext(ctx, http.MethodPost, url, bytes.NewReader(buf))
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
req.Header.Set("Content-Type", "application/json")
|
||||
if cfg.ApiKey != "" {
|
||||
req.Header.Set("Authorization", "Bearer "+cfg.ApiKey)
|
||||
}
|
||||
resp, err := httpClient.Do(req)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if resp.StatusCode >= 400 {
|
||||
msg, _ := io.ReadAll(resp.Body)
|
||||
resp.Body.Close()
|
||||
return nil, gerror.Newf("模型接口 %s 返回 %d: %s", url, resp.StatusCode, strings.TrimSpace(string(msg)))
|
||||
}
|
||||
return resp.Body, nil
|
||||
}
|
||||
@@ -0,0 +1,182 @@
|
||||
package service
|
||||
|
||||
import (
|
||||
"context"
|
||||
"strings"
|
||||
"unicode/utf8"
|
||||
|
||||
"rag-local/common"
|
||||
"rag-local/kb/consts"
|
||||
"rag-local/kb/dao"
|
||||
"rag-local/kb/model/domain"
|
||||
"rag-local/kb/model/entity"
|
||||
|
||||
eembedding "github.com/cloudwego/eino/components/embedding"
|
||||
"github.com/gogf/gf/v2/errors/gerror"
|
||||
"github.com/gogf/gf/v2/frame/g"
|
||||
)
|
||||
|
||||
var ChunkService = &chunkService{}
|
||||
|
||||
type chunkService struct{}
|
||||
|
||||
// SplitText 文本分块：标题感知 + 固定大小回退，超长段落按句号/换行切分并保留重叠
|
||||
func (s *chunkService) SplitText(text string) []string {
|
||||
text = strings.ReplaceAll(text, "\r\n", "\n")
|
||||
text = strings.ReplaceAll(text, "\r", "\n")
|
||||
|
||||
paragraphs := splitParagraphs(text)
|
||||
var merged []string
|
||||
var cur strings.Builder
|
||||
curRunes := 0
|
||||
for _, p := range paragraphs {
|
||||
pRunes := utf8.RuneCountInString(p)
|
||||
if cur.Len() > 0 && curRunes+pRunes > consts.MaxChunkSize {
|
||||
merged = append(merged, cur.String())
|
||||
cur.Reset()
|
||||
curRunes = 0
|
||||
}
|
||||
cur.WriteString(p)
|
||||
cur.WriteString("\n\n")
|
||||
curRunes += pRunes + 2
|
||||
}
|
||||
if cur.Len() > 0 {
|
||||
merged = append(merged, cur.String())
|
||||
}
|
||||
|
||||
var chunks []string
|
||||
for _, c := range merged {
|
||||
if len(c) <= consts.MaxChunkSize {
|
||||
chunks = append(chunks, strings.TrimSpace(c))
|
||||
continue
|
||||
}
|
||||
chunks = append(chunks, forceSplit(c)...)
|
||||
}
|
||||
return chunks
|
||||
}
|
||||
|
||||
// splitParagraphs 按空行/标题切段；# 标题行并入其后的段落（标题保留在段首，供检索回显）
|
||||
func splitParagraphs(text string) []string {
|
||||
lines := strings.Split(text, "\n")
|
||||
var paras []string
|
||||
var cur []string
|
||||
flush := func() {
|
||||
if len(cur) > 0 {
|
||||
paras = append(paras, strings.Join(cur, "\n"))
|
||||
cur = nil
|
||||
}
|
||||
}
|
||||
for _, line := range lines {
|
||||
trimmed := strings.TrimSpace(line)
|
||||
if trimmed == "" {
|
||||
flush()
|
||||
continue
|
||||
}
|
||||
if isHeading(trimmed) && len(cur) > 0 {
|
||||
flush()
|
||||
}
|
||||
cur = append(cur, trimmed)
|
||||
}
|
||||
flush()
|
||||
return paras
|
||||
}
|
||||
|
||||
func isHeading(line string) bool {
|
||||
return strings.HasPrefix(line, "#") ||
|
||||
strings.HasPrefix(line, "标题") || strings.HasPrefix(line, "第") && strings.Contains(line, "章")
|
||||
}
|
||||
|
||||
// forceSplit 超长文本按可读位置切分（rune 安全，避免切在 UTF-8 中间），重叠 overlap 字
|
||||
func forceSplit(text string) []string {
|
||||
runes := []rune(text)
|
||||
var result []string
|
||||
for len(runes) > consts.MaxChunkSize {
|
||||
limit := min(consts.MaxChunkSize, len(runes))
|
||||
cut := lastCutPoint(runes[:limit])
|
||||
if cut < consts.MaxChunkSize/2 {
|
||||
cut = consts.MaxChunkSize
|
||||
}
|
||||
if chunk := strings.TrimSpace(string(runes[:cut])); chunk != "" {
|
||||
result = append(result, chunk)
|
||||
}
|
||||
runes = runes[max(0, cut-consts.ChunkOverlap):]
|
||||
}
|
||||
if chunk := strings.TrimSpace(string(runes)); chunk != "" {
|
||||
result = append(result, chunk)
|
||||
}
|
||||
return result
|
||||
}
|
||||
|
||||
// lastCutPoint 在 limit 内找最后一个可读切点（句号/问号/感叹号/分号/换行/英文标点），找不到返回 -1
|
||||
func lastCutPoint(runes []rune) int {
|
||||
cut := -1
|
||||
for i, r := range runes {
|
||||
switch r {
|
||||
case '。', '！', '？', '；', '\n', '.', '!', '?', ';':
|
||||
cut = i + 1
|
||||
}
|
||||
}
|
||||
return cut
|
||||
}
|
||||
|
||||
// List 分块列表
|
||||
func (s *chunkService) List(ctx context.Context, documentId int64, page, pageSize int) ([]*entity.Chunk, int, error) {
|
||||
return dao.Chunk.ListByDocument(ctx, documentId, page, pageSize)
|
||||
}
|
||||
|
||||
// InsertAll 写入文档的全部分块；embedder 非空时批量向量化写入 vec0，否则仅写 FTS5
|
||||
func (s *chunkService) InsertAll(ctx context.Context, datasetId, documentId int64, chunks []string, embedder eembedding.Embedder) error {
|
||||
for start := 0; start < len(chunks); start += consts.EmbedBatchSize {
|
||||
end := min(start+consts.EmbedBatchSize, len(chunks))
|
||||
var vecs [][]float64
|
||||
if embedder != nil {
|
||||
v, err := embedder.EmbedStrings(ctx, chunks[start:end])
|
||||
if err != nil {
|
||||
return gerror.Wrap(err, "分块向量化失败")
|
||||
}
|
||||
vecs = v
|
||||
}
|
||||
for j, content := range chunks[start:end] {
|
||||
vecJson := ""
|
||||
if len(vecs) > 0 && j < len(vecs) {
|
||||
vecJson = domain.VecJsonF64(vecs[j])
|
||||
}
|
||||
if _, err := dao.Chunk.InsertWithVec(ctx, datasetId, documentId, start+j+1, content, "", "", vecJson, 0); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
}
|
||||
return dao.Document.UpdateFields(ctx, documentId, g.Map{
|
||||
"chunk_count": len(chunks),
|
||||
"status": consts.DocumentStatusDone,
|
||||
})
|
||||
}
|
||||
|
||||
// Update 编辑分块文本：重新分词 FTS 索引，数据集绑定 embedding 配置时同步重新向量化
|
||||
func (s *chunkService) Update(ctx context.Context, id int64, content string) error {
|
||||
chunk, err := dao.Chunk.GetOne(ctx, id)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if chunk == nil {
|
||||
return gerror.New("分块不存在")
|
||||
}
|
||||
vecJson := ""
|
||||
if cfgId, err := dao.Dataset.GetEmbeddingCfgId(ctx, chunk.DatasetId); err == nil && cfgId > 0 {
|
||||
if em, err := BuildEmbedder(ctx, cfgId); err == nil {
|
||||
if vecs, err := em.EmbedStrings(ctx, []string{content}); err != nil {
|
||||
g.Log().Warningf(ctx, "re-embed chunk %d failed: %v", id, err)
|
||||
} else if len(vecs) > 0 {
|
||||
vecJson = domain.VecJsonF64(vecs[0])
|
||||
}
|
||||
} else {
|
||||
g.Log().Warningf(ctx, "build embedder failed: %v", err)
|
||||
}
|
||||
}
|
||||
return dao.Chunk.UpdateContent(ctx, id, content, vecJson, common.Tokenize(content))
|
||||
}
|
||||
|
||||
// DeleteByDocument 删除文档全部数据（chunk + vec + fts）
|
||||
func (s *chunkService) DeleteByDocument(ctx context.Context, documentId int64) error {
|
||||
return dao.Chunk.DeleteByDocument(ctx, documentId)
|
||||
}
|
||||
@@ -0,0 +1,49 @@
|
||||
package service
|
||||
|
||||
import (
|
||||
"context"
|
||||
|
||||
"rag-local/kb/dao"
|
||||
"rag-local/kb/model/entity"
|
||||
|
||||
"github.com/gogf/gf/v2/errors/gerror"
|
||||
)
|
||||
|
||||
var ConversationService = &conversationService{}
|
||||
|
||||
type conversationService struct{}
|
||||
|
||||
func (s *conversationService) List(ctx context.Context) ([]*entity.Conversation, error) {
|
||||
return dao.Conversation.List(ctx)
|
||||
}
|
||||
|
||||
func (s *conversationService) Save(ctx context.Context, c *entity.Conversation) (int64, error) {
|
||||
if c.Id > 0 {
|
||||
conv, err := dao.Conversation.GetOne(ctx, c.Id)
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
if conv == nil {
|
||||
return 0, gerror.New("会话不存在")
|
||||
}
|
||||
if c.Title != "" && c.Title != conv.Title {
|
||||
if err := dao.Conversation.UpdateTitle(ctx, c.Id, c.Title); err != nil {
|
||||
return 0, err
|
||||
}
|
||||
}
|
||||
if c.DatasetId > 0 && c.DatasetId != conv.DatasetId {
|
||||
if err := dao.Conversation.UpdateDataset(ctx, c.Id, c.DatasetId); err != nil {
|
||||
return 0, err
|
||||
}
|
||||
}
|
||||
return c.Id, nil
|
||||
}
|
||||
if c.DatasetId <= 0 {
|
||||
return 0, gerror.New("请选择知识库数据集")
|
||||
}
|
||||
return dao.Conversation.Insert(ctx, c.DatasetId, c.Title)
|
||||
}
|
||||
|
||||
func (s *conversationService) Delete(ctx context.Context, id int64) error {
|
||||
return dao.Conversation.Delete(ctx, id)
|
||||
}
|
||||
@@ -0,0 +1,46 @@
|
||||
package service
|
||||
|
||||
import (
|
||||
"context"
|
||||
|
||||
"rag-local/kb/consts"
|
||||
"rag-local/kb/dao"
|
||||
"rag-local/kb/model/entity"
|
||||
|
||||
"github.com/gogf/gf/v2/errors/gerror"
|
||||
"github.com/gogf/gf/v2/frame/g"
|
||||
)
|
||||
|
||||
var DatasetService = &datasetService{}
|
||||
|
||||
type datasetService struct{}
|
||||
|
||||
func (s *datasetService) List(ctx context.Context) ([]*entity.Dataset, error) {
|
||||
return dao.Dataset.List(ctx)
|
||||
}
|
||||
|
||||
func (s *datasetService) Save(ctx context.Context, m *entity.Dataset) (int64, error) {
|
||||
if m.Status == 0 {
|
||||
m.Status = 1
|
||||
}
|
||||
if m.Id > 0 {
|
||||
if err := dao.Dataset.Update(ctx, m); err != nil {
|
||||
return 0, err
|
||||
}
|
||||
return m.Id, nil
|
||||
}
|
||||
return dao.Dataset.Insert(ctx, m)
|
||||
}
|
||||
|
||||
func (s *datasetService) Delete(ctx context.Context, id int64) error {
|
||||
// 有文档的数据集不允许删除
|
||||
count, err := g.DB(consts.DbGroupDefault).Model(consts.TableNameDocument).Ctx(ctx).
|
||||
Where("dataset_id", id).Count()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if count > 0 {
|
||||
return gerror.New("数据集下存在文档，无法删除")
|
||||
}
|
||||
return dao.Dataset.Delete(ctx, id)
|
||||
}
|
||||
@@ -0,0 +1,161 @@
|
||||
package service
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"rag-local/common"
|
||||
"rag-local/kb/consts"
|
||||
"rag-local/kb/dao"
|
||||
"rag-local/kb/model/domain"
|
||||
"rag-local/kb/model/entity"
|
||||
|
||||
"github.com/gogf/gf/v2/errors/gerror"
|
||||
)
|
||||
|
||||
var DocumentService = &documentService{}
|
||||
|
||||
type documentService struct{}
|
||||
|
||||
// Upload 保存上传文件到 workspace/{datasetId}/{yyyymmdd}/{uuid}.ext，落库并提交解析任务
|
||||
func (s *documentService) Upload(ctx context.Context, datasetId int64, filename string, data []byte) (*entity.Document, error) {
|
||||
dataset, err := dao.Dataset.GetOne(ctx, datasetId)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if dataset == nil {
|
||||
return nil, gerror.New("数据集不存在")
|
||||
}
|
||||
ext := strings.TrimPrefix(strings.ToLower(filepath.Ext(filename)), ".")
|
||||
if !isSupportedExt(ext) {
|
||||
return nil, gerror.Newf("不支持的文档类型: %s（支持 %s）", ext, strings.Join(common.SupportedExts(), "/"))
|
||||
}
|
||||
if len(data) == 0 {
|
||||
return nil, gerror.New("文件内容为空")
|
||||
}
|
||||
|
||||
relDir := filepath.Join(fmt.Sprintf("%d", datasetId), time.Now().Format("20060102"))
|
||||
relPath := filepath.Join(relDir, fmt.Sprintf("%s.%s", common.RandomToken(16), ext))
|
||||
absPath := filepath.Join("workspace", relPath)
|
||||
if err := os.MkdirAll(filepath.Dir(absPath), 0o755); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if err := os.WriteFile(absPath, data, 0o644); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
docId, err := dao.Document.Insert(ctx, &entity.Document{
|
||||
DatasetId: datasetId,
|
||||
Filename: filename,
|
||||
FilePath: relPath,
|
||||
FileSize: int64(len(data)),
|
||||
FileType: ext,
|
||||
Status: consts.DocumentStatusPending,
|
||||
})
|
||||
if err != nil {
|
||||
_ = os.Remove(absPath)
|
||||
return nil, err
|
||||
}
|
||||
if _, err := dao.ParseTask.Insert(ctx, docId, datasetId, consts.TaskTypeParse); err != nil {
|
||||
_ = os.Remove(absPath)
|
||||
return nil, err
|
||||
}
|
||||
return dao.Document.GetOne(ctx, docId)
|
||||
}
|
||||
|
||||
func isSupportedExt(ext string) bool {
|
||||
for _, e := range common.SupportedExts() {
|
||||
if e == ext {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// List 文档列表
|
||||
func (s *documentService) List(ctx context.Context, datasetId int64, page, pageSize int) ([]*entity.Document, int, error) {
|
||||
return dao.Document.List(ctx, datasetId, page, pageSize)
|
||||
}
|
||||
|
||||
// Reembed 文档全部分块重新向量化（分块文本不变，仅重算向量）
|
||||
func (s *documentService) Reembed(ctx context.Context, id int64) error {
|
||||
doc, err := dao.Document.GetOne(ctx, id)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if doc == nil {
|
||||
return gerror.New("文档不存在")
|
||||
}
|
||||
cfgId, err := dao.Dataset.GetEmbeddingCfgId(ctx, doc.DatasetId)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if cfgId <= 0 {
|
||||
return gerror.New("数据集未绑定向量模型，无法向量化")
|
||||
}
|
||||
em, err := BuildEmbedder(ctx, cfgId)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
chunks, _, err := dao.Chunk.ListByDocument(ctx, id, 1, 100000)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
for start := 0; start < len(chunks); start += consts.EmbedBatchSize {
|
||||
end := min(start+consts.EmbedBatchSize, len(chunks))
|
||||
texts := make([]string, 0, end-start)
|
||||
for _, c := range chunks[start:end] {
|
||||
texts = append(texts, c.Content)
|
||||
}
|
||||
vecs, err := em.EmbedStrings(ctx, texts)
|
||||
if err != nil {
|
||||
return gerror.Wrap(err, "向量化失败")
|
||||
}
|
||||
for j, c := range chunks[start:end] {
|
||||
if err := dao.Chunk.UpdateVec(ctx, c.Id, domain.VecJsonF64(vecs[j])); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// Delete 删除文档：先删文件与索引数据，再删记录
|
||||
func (s *documentService) Delete(ctx context.Context, id int64) error {
|
||||
doc, err := dao.Document.GetOne(ctx, id)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if doc == nil {
|
||||
return nil
|
||||
}
|
||||
// 先取分块 id 清理知识图谱数据（分块删除后无法再映射）
|
||||
chunks, _, err := dao.Chunk.ListByDocument(ctx, id, 1, 100000)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
chunkIds := make([]int64, 0, len(chunks))
|
||||
for _, c := range chunks {
|
||||
chunkIds = append(chunkIds, c.Id)
|
||||
}
|
||||
if err := dao.KgRelation.DeleteByChunkIds(ctx, chunkIds); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := dao.KgEntity.DeleteByChunkIds(ctx, chunkIds); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := ChunkService.DeleteByDocument(ctx, id); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := dao.ParseTask.DeleteByDocument(ctx, id); err != nil {
|
||||
return err
|
||||
}
|
||||
if doc.FilePath != "" {
|
||||
_ = os.Remove(filepath.Join("workspace", doc.FilePath))
|
||||
}
|
||||
return dao.Document.Delete(ctx, id)
|
||||
}
|
||||
@@ -0,0 +1,117 @@
|
||||
package service
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"strings"
|
||||
|
||||
"rag-local/kb/dao"
|
||||
"rag-local/kb/model/entity"
|
||||
|
||||
"github.com/cloudwego/eino/schema"
|
||||
"github.com/gogf/gf/v2/errors/gerror"
|
||||
"github.com/gogf/gf/v2/frame/g"
|
||||
)
|
||||
|
||||
var KgEntityService = &kgEntityService{}
|
||||
|
||||
type kgEntityService struct{}
|
||||
|
||||
type kgEntityItem struct {
|
||||
Name string `json:"name"`
|
||||
Type string `json:"type"`
|
||||
}
|
||||
|
||||
type kgRelationItem struct {
|
||||
Head string `json:"head"`
|
||||
Relation string `json:"relation"`
|
||||
Tail string `json:"tail"`
|
||||
}
|
||||
|
||||
type kgExtractResult struct {
|
||||
Entities []kgEntityItem `json:"entities"`
|
||||
Relations []kgRelationItem `json:"relations"`
|
||||
}
|
||||
|
||||
// ExtractDocument 对文档全部新分块做 LLM 抽取（挂在解析流水线分块落库之后）。
|
||||
// 每个分块一次调用，任何失败只记日志，不阻断解析流水线；未配置默认对话模型时直接跳过。
|
||||
func (s *kgEntityService) ExtractDocument(ctx context.Context, datasetId, documentId int64) error {
|
||||
chunks, _, err := dao.Chunk.ListByDocument(ctx, documentId, 1, 100000)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
model, err := s.buildModel(ctx)
|
||||
if err != nil {
|
||||
g.Log().Infof(ctx, "kg extract skipped: %v", err)
|
||||
return nil
|
||||
}
|
||||
for _, c := range chunks {
|
||||
if err := s.extractChunk(ctx, model, datasetId, c); err != nil {
|
||||
g.Log().Warningf(ctx, "kg extract chunk %d failed: %v", c.Id, err)
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (s *kgEntityService) buildModel(ctx context.Context) (*OpenAIChatModel, error) {
|
||||
defaultChatModel, _, err := SystemConfigService.GetSettings(ctx)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if defaultChatModel <= 0 {
|
||||
return nil, gerror.New("未配置默认对话模型")
|
||||
}
|
||||
return BuildChatModel(ctx, defaultChatModel)
|
||||
}
|
||||
|
||||
func (s *kgEntityService) extractChunk(ctx context.Context, model *OpenAIChatModel, datasetId int64, chunk *entity.Chunk) error {
|
||||
msgs := []*schema.Message{
|
||||
{Role: schema.System, Content: "你是知识抽取助手。从文档片段中抽取实体（人名、组织、地名、产品等专有名词）及实体间的关系（动词或介词短语）。只输出 JSON，不要 markdown 代码块或任何解释，格式：{\"entities\":[{\"name\":\"实体名\",\"type\":\"类型\"}],\"relations\":[{\"head\":\"主体\",\"relation\":\"关系\",\"tail\":\"客体\"}]}"},
|
||||
{Role: schema.User, Content: "文档片段：\n" + chunk.Content},
|
||||
}
|
||||
resp, err := model.Generate(ctx, msgs)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
data, err := parseKgJSON(resp.Content)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
for _, e := range data.Entities {
|
||||
name := strings.TrimSpace(e.Name)
|
||||
if name == "" {
|
||||
continue
|
||||
}
|
||||
if err := dao.KgEntity.Upsert(ctx, datasetId, chunk.Id, name, strings.TrimSpace(e.Type)); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
for _, r := range data.Relations {
|
||||
head, relation, tail := strings.TrimSpace(r.Head), strings.TrimSpace(r.Relation), strings.TrimSpace(r.Tail)
|
||||
if head == "" || relation == "" || tail == "" || head == tail {
|
||||
continue
|
||||
}
|
||||
if err := dao.KgRelation.Insert(ctx, datasetId, chunk.Id, head, relation, tail); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// parseKgJSON 解析模型输出的 JSON，容忍 ```json 代码块包裹
|
||||
func parseKgJSON(content string) (*kgExtractResult, error) {
|
||||
content = strings.TrimSpace(content)
|
||||
content = strings.TrimPrefix(content, "```json")
|
||||
content = strings.TrimPrefix(content, "```")
|
||||
content = strings.TrimSuffix(content, "```")
|
||||
content = strings.TrimSpace(content)
|
||||
var out kgExtractResult
|
||||
if err := json.Unmarshal([]byte(content), &out); err != nil {
|
||||
return nil, gerror.Wrap(err, "解析抽取 JSON 失败")
|
||||
}
|
||||
return &out, nil
|
||||
}
|
||||
|
||||
func (s *kgEntityService) List(ctx context.Context, datasetId int64, page, pageSize int) ([]*entity.KgEntity, int, error) {
|
||||
return dao.KgEntity.List(ctx, datasetId, page, pageSize)
|
||||
}
|
||||
@@ -0,0 +1,90 @@
|
||||
package service
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"sort"
|
||||
"strings"
|
||||
|
||||
"rag-local/common"
|
||||
"rag-local/kb/dao"
|
||||
"rag-local/kb/model/entity"
|
||||
)
|
||||
|
||||
var KgRelationService = &kgRelationService{}
|
||||
|
||||
type kgRelationService struct{}
|
||||
|
||||
const (
|
||||
kgLinkTopN = 3 // 实体链接命中前 N 个实体
|
||||
kgNeighborLimit = 20 // 一跳邻居三元组上限
|
||||
)
|
||||
|
||||
// GraphEnhance 图增强检索：问题分词与实体名匹配（Top3）→ 一跳邻居三元组 → 格式化文本（供注入提示词）
|
||||
func (s *kgRelationService) GraphEnhance(ctx context.Context, datasetId int64, question string) ([]string, error) {
|
||||
names, err := dao.KgEntity.ListNames(ctx, datasetId)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
linked := linkEntities(question, names, kgLinkTopN)
|
||||
if len(linked) == 0 {
|
||||
return nil, nil
|
||||
}
|
||||
triples, err := dao.KgRelation.Neighbors(ctx, datasetId, linked, kgNeighborLimit)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
out := make([]string, 0, len(triples))
|
||||
for _, t := range triples {
|
||||
out = append(out, fmt.Sprintf("%s -%s-> %s", t.Head, t.Relation, t.Tail))
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// linkEntities 问题分词后按 token 命中实体名的个数打分，取前 topN；同分按名称长度优先
|
||||
func linkEntities(question string, names []string, topN int) []string {
|
||||
tokens := strings.Fields(common.Tokenize(question))
|
||||
if len(tokens) == 0 {
|
||||
return nil
|
||||
}
|
||||
type scored struct {
|
||||
name string
|
||||
score int
|
||||
}
|
||||
var hits []scored
|
||||
for _, n := range names {
|
||||
if n == "" {
|
||||
continue
|
||||
}
|
||||
score := 0
|
||||
if strings.Contains(question, n) {
|
||||
score += 5 // 问题中出现完整实体名，强相关
|
||||
}
|
||||
for _, t := range tokens {
|
||||
if strings.Contains(n, t) {
|
||||
score += len([]rune(t)) // 命中 token 越长相关性越高
|
||||
}
|
||||
}
|
||||
if score > 0 {
|
||||
hits = append(hits, scored{name: n, score: score})
|
||||
}
|
||||
}
|
||||
sort.Slice(hits, func(i, j int) bool {
|
||||
if hits[i].score != hits[j].score {
|
||||
return hits[i].score > hits[j].score
|
||||
}
|
||||
return len(hits[i].name) > len(hits[j].name)
|
||||
})
|
||||
if len(hits) > topN {
|
||||
hits = hits[:topN]
|
||||
}
|
||||
out := make([]string, 0, len(hits))
|
||||
for _, h := range hits {
|
||||
out = append(out, h.name)
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func (s *kgRelationService) List(ctx context.Context, datasetId int64, page, pageSize int) ([]*entity.KgRelation, int, error) {
|
||||
return dao.KgRelation.List(ctx, datasetId, page, pageSize)
|
||||
}
|
||||
@@ -0,0 +1,83 @@
|
||||
package service
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
|
||||
"rag-local/kb/dao"
|
||||
"rag-local/kb/model/domain"
|
||||
"rag-local/kb/model/entity"
|
||||
|
||||
"github.com/cloudwego/eino/schema"
|
||||
"github.com/gogf/gf/v2/errors/gerror"
|
||||
)
|
||||
|
||||
var MessageService = &messageService{}
|
||||
|
||||
type messageService struct{}
|
||||
|
||||
func (s *messageService) List(ctx context.Context, conversationId int64) ([]*entity.Message, error) {
|
||||
return dao.Message.List(ctx, conversationId)
|
||||
}
|
||||
|
||||
// Chat RAG 问答：会话解析 → 用户消息落库 → 工作流流式生成 → 助手消息+引用落库。
|
||||
// onCitations 在检索完成后回调（先于流式输出）；onDelta 接收模型增量文本。
|
||||
func (s *messageService) Chat(ctx context.Context, conversationId, datasetId int64, question string, onCitations func([]domain.Citation, int64), onDelta func(string)) (string, []domain.Citation, int64, error) {
|
||||
if conversationId <= 0 {
|
||||
title := question
|
||||
if r := []rune(title); len(r) > 20 {
|
||||
title = string(r[:20])
|
||||
}
|
||||
id, err := dao.Conversation.Insert(ctx, datasetId, title)
|
||||
if err != nil {
|
||||
return "", nil, 0, err
|
||||
}
|
||||
conversationId = id
|
||||
}
|
||||
conv, err := dao.Conversation.GetOne(ctx, conversationId)
|
||||
if err != nil {
|
||||
return "", nil, 0, err
|
||||
}
|
||||
if conv == nil {
|
||||
return "", nil, 0, gerror.New("会话不存在")
|
||||
}
|
||||
if conv.DatasetId == 0 && datasetId > 0 {
|
||||
_ = dao.Conversation.UpdateDataset(ctx, conversationId, datasetId)
|
||||
conv.DatasetId = datasetId
|
||||
}
|
||||
if conv.DatasetId > 0 {
|
||||
datasetId = conv.DatasetId
|
||||
}
|
||||
if datasetId <= 0 {
|
||||
return "", nil, 0, gerror.New("请选择知识库数据集")
|
||||
}
|
||||
|
||||
if _, err := dao.Message.Insert(ctx, conversationId, "user", question, ""); err != nil {
|
||||
return "", nil, 0, err
|
||||
}
|
||||
|
||||
history, err := dao.Message.List(ctx, conversationId)
|
||||
if err != nil {
|
||||
return "", nil, 0, err
|
||||
}
|
||||
messages := make([]*schema.Message, 0, len(history))
|
||||
for _, m := range history {
|
||||
messages = append(messages, &schema.Message{Role: schema.RoleType(m.Role), Content: m.Content})
|
||||
}
|
||||
|
||||
answer, citations, err := ChatService.Ask(ctx, datasetId, question, messages,
|
||||
func(c []domain.Citation) {
|
||||
if onCitations != nil {
|
||||
onCitations(c, conversationId)
|
||||
}
|
||||
}, onDelta)
|
||||
if err != nil {
|
||||
return "", nil, 0, err
|
||||
}
|
||||
|
||||
citJson, _ := json.Marshal(citations)
|
||||
if _, err := dao.Message.Insert(ctx, conversationId, "assistant", answer, string(citJson)); err != nil {
|
||||
return "", nil, 0, err
|
||||
}
|
||||
return answer, citations, conversationId, nil
|
||||
}
|
||||
@@ -0,0 +1,87 @@
|
||||
package service
|
||||
|
||||
import (
|
||||
"context"
|
||||
|
||||
"rag-local/kb/consts"
|
||||
"rag-local/kb/dao"
|
||||
"rag-local/kb/model/entity"
|
||||
|
||||
"github.com/cloudwego/eino/schema"
|
||||
"github.com/gogf/gf/v2/errors/gerror"
|
||||
"github.com/gogf/gf/v2/frame/g"
|
||||
)
|
||||
|
||||
var ModelConfigService = &modelConfigService{}
|
||||
|
||||
type modelConfigService struct{}
|
||||
|
||||
func (s *modelConfigService) List(ctx context.Context, modelType string) ([]*entity.ModelConfig, error) {
|
||||
return dao.ModelConfig.List(ctx, modelType)
|
||||
}
|
||||
|
||||
func (s *modelConfigService) Save(ctx context.Context, m *entity.ModelConfig) (int64, error) {
|
||||
if m.Dimension <= 0 {
|
||||
m.Dimension = consts.DefaultEmbeddingDim
|
||||
}
|
||||
if m.Id > 0 {
|
||||
if err := dao.ModelConfig.Update(ctx, m); err != nil {
|
||||
return 0, err
|
||||
}
|
||||
return m.Id, nil
|
||||
}
|
||||
return dao.ModelConfig.Insert(ctx, m)
|
||||
}
|
||||
|
||||
// Test 连通性测试：对话模型发一次 ping，向量模型嵌入一次，任一异常即失败
|
||||
func (s *modelConfigService) Test(ctx context.Context, id int64) error {
|
||||
cfg, err := dao.ModelConfig.GetOne(ctx, id)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if cfg == nil {
|
||||
return gerror.New("模型配置不存在")
|
||||
}
|
||||
switch cfg.ModelType {
|
||||
case consts.ModelTypeChat:
|
||||
model, err := BuildChatModel(ctx, id)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
msg, err := model.Generate(ctx, []*schema.Message{{Role: schema.User, Content: "ping"}})
|
||||
if err != nil {
|
||||
return gerror.Wrap(err, "对话接口调用失败")
|
||||
}
|
||||
if msg == nil {
|
||||
return gerror.New("对话接口返回空")
|
||||
}
|
||||
case consts.ModelTypeEmbedding:
|
||||
em, err := BuildEmbedder(ctx, id)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
vecs, err := em.EmbedStrings(ctx, []string{"ping"})
|
||||
if err != nil {
|
||||
return gerror.Wrap(err, "向量接口调用失败")
|
||||
}
|
||||
if len(vecs) == 0 || len(vecs[0]) == 0 {
|
||||
return gerror.New("向量接口返回空数据")
|
||||
}
|
||||
default:
|
||||
return gerror.New("未知模型类型")
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (s *modelConfigService) Delete(ctx context.Context, id int64) error {
|
||||
// 被数据集绑定的 embedding 配置不允许删除
|
||||
count, err := g.DB(consts.DbGroupDefault).Model(consts.TableNameDataset).Ctx(ctx).
|
||||
Where("embedding_cfg_id", id).Count()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if count > 0 {
|
||||
return gerror.New("该模型配置正被数据集使用，无法删除")
|
||||
}
|
||||
return dao.ModelConfig.Delete(ctx, id)
|
||||
}
|
||||
@@ -0,0 +1,130 @@
|
||||
package service
|
||||
|
||||
import (
|
||||
"context"
|
||||
"path/filepath"
|
||||
"time"
|
||||
|
||||
"rag-local/common"
|
||||
"rag-local/kb/consts"
|
||||
"rag-local/kb/dao"
|
||||
"rag-local/kb/model/entity"
|
||||
|
||||
eembedding "github.com/cloudwego/eino/components/embedding"
|
||||
"github.com/gogf/gf/v2/errors/gerror"
|
||||
"github.com/gogf/gf/v2/frame/g"
|
||||
)
|
||||
|
||||
var ParseTaskService = &parseTaskService{}
|
||||
|
||||
type parseTaskService struct{}
|
||||
|
||||
// StartParsePoller 启动任务轮询：单 goroutine 串行消费待处理任务（与 video-factory StartVideoPoller 同模式）
|
||||
func (s *parseTaskService) StartParsePoller(ctx context.Context) {
|
||||
go func() {
|
||||
g.Log().Info(ctx, "parse task poller started")
|
||||
for {
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return
|
||||
case <-time.After(consts.ParsePollIntervalSeconds * time.Second):
|
||||
s.processOne(ctx)
|
||||
}
|
||||
}
|
||||
}()
|
||||
}
|
||||
|
||||
// processOne 处理一个待处理任务：解析 → 分块 → 落库（向量化在 M3 接入）
|
||||
func (s *parseTaskService) processOne(ctx context.Context) {
|
||||
task, err := dao.ParseTask.NextPending(ctx)
|
||||
if err != nil {
|
||||
g.Log().Errorf(ctx, "next parse task failed: %v", err)
|
||||
return
|
||||
}
|
||||
if task == nil {
|
||||
return
|
||||
}
|
||||
if err := dao.ParseTask.UpdateStatus(ctx, task.Id, consts.TaskStatusRunning, ""); err != nil {
|
||||
g.Log().Errorf(ctx, "mark task running failed: %v", err)
|
||||
return
|
||||
}
|
||||
|
||||
doc, err := dao.Document.GetOne(ctx, task.DocumentId)
|
||||
if err != nil {
|
||||
s.fail(ctx, task, "读取文档失败: "+err.Error())
|
||||
return
|
||||
}
|
||||
if doc == nil {
|
||||
s.fail(ctx, task, "文档不存在")
|
||||
return
|
||||
}
|
||||
if err := dao.Document.UpdateFields(ctx, doc.Id, g.Map{"status": consts.DocumentStatusParsing}); err != nil {
|
||||
s.fail(ctx, task, "更新文档状态失败: "+err.Error())
|
||||
return
|
||||
}
|
||||
|
||||
text, err := common.ParseFile(filepath.Join("workspace", doc.FilePath))
|
||||
if err != nil {
|
||||
s.fail(ctx, task, "解析文件失败: "+err.Error())
|
||||
return
|
||||
}
|
||||
chunks := ChunkService.SplitText(text)
|
||||
// 数据集绑定 embedding 配置时构建向量模型，无配置降级为仅全文索引
|
||||
var embedder eembedding.Embedder
|
||||
if cfgId, err := dao.Dataset.GetEmbeddingCfgId(ctx, task.DatasetId); err == nil && cfgId > 0 {
|
||||
if em, err := BuildEmbedder(ctx, cfgId); err == nil {
|
||||
embedder = em
|
||||
if dim := em.Dim(); dim != g.Cfg().MustGet(ctx, "vector.dim", consts.DefaultEmbeddingDim).Int() {
|
||||
g.Log().Warningf(ctx, "embedding 维度 %d 与 vec0 表维度不一致，请确认 vector.dim 配置", dim)
|
||||
}
|
||||
} else {
|
||||
g.Log().Warningf(ctx, "build embedder failed, fallback to fts only: %v", err)
|
||||
}
|
||||
}
|
||||
if err := ChunkService.InsertAll(ctx, task.DatasetId, doc.Id, chunks, embedder); err != nil {
|
||||
s.fail(ctx, task, "写入分块失败: "+err.Error())
|
||||
return
|
||||
}
|
||||
// 第 3.5 步：知识图谱抽取（失败不阻断流水线）
|
||||
if err := KgEntityService.ExtractDocument(ctx, task.DatasetId, doc.Id); err != nil {
|
||||
g.Log().Warningf(ctx, "kg extract skipped for doc %d: %v", doc.Id, err)
|
||||
}
|
||||
if err := dao.ParseTask.UpdateStatus(ctx, task.Id, consts.TaskStatusDone, ""); err != nil {
|
||||
g.Log().Errorf(ctx, "mark task done failed: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func (s *parseTaskService) fail(ctx context.Context, task *entity.ParseTask, msg string) {
|
||||
_ = dao.ParseTask.UpdateStatus(ctx, task.Id, consts.TaskStatusFailed, msg)
|
||||
_ = dao.Document.UpdateFields(ctx, task.DocumentId, g.Map{
|
||||
"status": consts.DocumentStatusFailed,
|
||||
"error_msg": msg,
|
||||
})
|
||||
g.Log().Errorf(ctx, "parse task %d failed: %s", task.Id, msg)
|
||||
}
|
||||
|
||||
// List 任务列表
|
||||
func (s *parseTaskService) List(ctx context.Context, page, pageSize int) ([]*entity.ParseTask, int, error) {
|
||||
return dao.ParseTask.List(ctx, page, pageSize)
|
||||
}
|
||||
|
||||
// Retry 失败任务重置为待处理（文档状态同步重置）
|
||||
func (s *parseTaskService) Retry(ctx context.Context, id int64) error {
|
||||
task, err := dao.ParseTask.GetOne(ctx, id)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if task == nil {
|
||||
return gerror.New("任务不存在")
|
||||
}
|
||||
if task.Status != consts.TaskStatusFailed {
|
||||
return gerror.New("仅失败任务可重试")
|
||||
}
|
||||
if err := dao.ParseTask.UpdateStatus(ctx, id, consts.TaskStatusPending, ""); err != nil {
|
||||
return err
|
||||
}
|
||||
return dao.Document.UpdateFields(ctx, task.DocumentId, g.Map{
|
||||
"status": consts.DocumentStatusPending,
|
||||
"error_msg": "",
|
||||
})
|
||||
}
|
||||
@@ -0,0 +1,100 @@
|
||||
package service
|
||||
|
||||
import (
|
||||
"context"
|
||||
"time"
|
||||
|
||||
"rag-local/common"
|
||||
"rag-local/kb/consts"
|
||||
"rag-local/kb/dao"
|
||||
|
||||
"github.com/gogf/gf/v2/errors/gerror"
|
||||
"github.com/gogf/gf/v2/os/gcache"
|
||||
"github.com/gogf/gf/v2/util/gconv"
|
||||
)
|
||||
|
||||
var SystemConfigService = &systemConfigService{}
|
||||
|
||||
type systemConfigService struct{}
|
||||
|
||||
func init() {
|
||||
// 注入指纹校验函数，避免 common → service 循环依赖
|
||||
common.CheckTokenFingerprint = SystemConfigService.CheckTokenFingerprint
|
||||
}
|
||||
|
||||
// EnsureAccessToken 首次启动生成访问令牌并写入配置；已有则复用。返回当前令牌
|
||||
func (s *systemConfigService) EnsureAccessToken(ctx context.Context) (string, error) {
|
||||
token, err := dao.SystemConfig.Get(ctx, consts.CfgKeyAccessToken)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
if token == "" {
|
||||
token = common.RandomToken(16)
|
||||
if err := dao.SystemConfig.Set(ctx, consts.CfgKeyAccessToken, token); err != nil {
|
||||
return "", err
|
||||
}
|
||||
}
|
||||
return token, nil
|
||||
}
|
||||
|
||||
func (s *systemConfigService) Login(ctx context.Context, token string) (string, error) {
|
||||
cur, err := dao.SystemConfig.Get(ctx, consts.CfgKeyAccessToken)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
if cur == "" || token != cur {
|
||||
return "", gerror.New("访问令牌错误")
|
||||
}
|
||||
return common.SignToken("owner", common.TokenFingerprint(cur), common.TokenExpireSeconds)
|
||||
}
|
||||
|
||||
// GetToken 当前访问令牌（设置页展示用）
|
||||
func (s *systemConfigService) GetToken(ctx context.Context) (string, error) {
|
||||
return dao.SystemConfig.Get(ctx, consts.CfgKeyAccessToken)
|
||||
}
|
||||
|
||||
// TokenFingerprint 当前令牌指纹（带短缓存，鉴权路径避免频繁查库）
|
||||
func (s *systemConfigService) TokenFingerprint(ctx context.Context) string {
|
||||
if v, err := gcache.Get(ctx, "access_token_fp"); err == nil && !v.IsNil() {
|
||||
return v.String()
|
||||
}
|
||||
token, err := dao.SystemConfig.Get(ctx, consts.CfgKeyAccessToken)
|
||||
if err != nil || token == "" {
|
||||
return ""
|
||||
}
|
||||
fp := common.TokenFingerprint(token)
|
||||
_ = gcache.Set(ctx, "access_token_fp", fp, 10*time.Second)
|
||||
return fp
|
||||
}
|
||||
|
||||
func (s *systemConfigService) CheckTokenFingerprint(ctx context.Context, fp string) bool {
|
||||
return fp != "" && fp == s.TokenFingerprint(ctx)
|
||||
}
|
||||
|
||||
func (s *systemConfigService) RegenerateToken(ctx context.Context) (string, error) {
|
||||
token := common.RandomToken(16)
|
||||
if err := dao.SystemConfig.Set(ctx, consts.CfgKeyAccessToken, token); err != nil {
|
||||
return "", err
|
||||
}
|
||||
_, _ = gcache.Remove(ctx, "access_token_fp")
|
||||
return token, nil
|
||||
}
|
||||
|
||||
func (s *systemConfigService) GetSettings(ctx context.Context) (defaultChatModel, defaultDataset int64, err error) {
|
||||
v1, err := dao.SystemConfig.Get(ctx, consts.CfgKeyDefaultChatModel)
|
||||
if err != nil {
|
||||
return
|
||||
}
|
||||
v2, err := dao.SystemConfig.Get(ctx, consts.CfgKeyDefaultDataset)
|
||||
if err != nil {
|
||||
return
|
||||
}
|
||||
return gconv.Int64(v1), gconv.Int64(v2), nil
|
||||
}
|
||||
|
||||
func (s *systemConfigService) UpdateSettings(ctx context.Context, defaultChatModel, defaultDataset int64) error {
|
||||
if err := dao.SystemConfig.Set(ctx, consts.CfgKeyDefaultChatModel, gconv.String(defaultChatModel)); err != nil {
|
||||
return err
|
||||
}
|
||||
return dao.SystemConfig.Set(ctx, consts.CfgKeyDefaultDataset, gconv.String(defaultDataset))
|
||||
}
|
||||
@@ -0,0 +1,95 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"os"
|
||||
"os/signal"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"syscall"
|
||||
"time"
|
||||
|
||||
commonHttp "rag-local/common"
|
||||
"rag-local/kb/controller"
|
||||
"rag-local/kb/service"
|
||||
|
||||
"github.com/gogf/gf/v2/frame/g"
|
||||
"github.com/gogf/gf/v2/net/ghttp"
|
||||
|
||||
_ "github.com/gogf/gf/contrib/drivers/sqlite/v2"
|
||||
_ "modernc.org/sqlite/vec"
|
||||
)
|
||||
|
||||
func main() {
|
||||
// ==================== API 路由（通过 RouteRegister 自动注册） ====================
|
||||
commonHttp.RouteRegister([]interface{}{
|
||||
controller.SystemConfig,
|
||||
controller.ModelConfig,
|
||||
controller.Dataset,
|
||||
controller.Document,
|
||||
controller.Chunk,
|
||||
controller.ParseTask,
|
||||
controller.Conversation,
|
||||
controller.Message,
|
||||
controller.KgEntity,
|
||||
controller.KgRelation,
|
||||
})
|
||||
|
||||
// ==================== Workspace 文件服务（源文件访问，路径穿越防护） ====================
|
||||
commonHttp.Httpserver.BindHandler("/workspace/*", func(r *ghttp.Request) {
|
||||
relPath := strings.TrimPrefix(r.URL.Path, "/workspace/")
|
||||
if relPath == "" || strings.Contains(relPath, "..") {
|
||||
r.Response.WriteStatus(403)
|
||||
return
|
||||
}
|
||||
filePath := filepath.Join("workspace", relPath)
|
||||
if _, err := os.Stat(filePath); os.IsNotExist(err) {
|
||||
r.Response.WriteStatus(404)
|
||||
return
|
||||
}
|
||||
r.Response.ServeFile(filePath)
|
||||
})
|
||||
|
||||
// ==================== 前端静态资源服务（SPA，前后端合并部署） ====================
|
||||
if st, err := os.Stat("ui-src/dist"); err == nil && st.IsDir() {
|
||||
commonHttp.Httpserver.BindHandler("/*", func(r *ghttp.Request) {
|
||||
path := strings.TrimPrefix(r.URL.Path, "/")
|
||||
if strings.Contains(path, "..") {
|
||||
r.Response.WriteStatus(404)
|
||||
return
|
||||
}
|
||||
if path == "" || path == "index.html" {
|
||||
path = "index.html"
|
||||
}
|
||||
filePath := filepath.Join("ui-src/dist", path)
|
||||
if st, err := os.Stat(filePath); err == nil && !st.IsDir() {
|
||||
r.Response.ServeFile(filePath)
|
||||
return
|
||||
}
|
||||
r.Response.WriteStatus(404)
|
||||
})
|
||||
}
|
||||
|
||||
ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM)
|
||||
defer stop()
|
||||
|
||||
// ==================== 解析任务轮询（文档流水线消费端） ====================
|
||||
service.ParseTaskService.StartParsePoller(ctx)
|
||||
|
||||
// ==================== 访问令牌（首次启动生成，打印到控制台） ====================
|
||||
token, err := service.SystemConfigService.EnsureAccessToken(ctx)
|
||||
if err != nil {
|
||||
g.Log().Fatal(ctx, "ensure access token failed: %v", err)
|
||||
}
|
||||
g.Log().Infof(ctx, "============================================")
|
||||
g.Log().Infof(ctx, "访问令牌(登录用): %s", token)
|
||||
g.Log().Infof(ctx, "请在登录页输入上述令牌")
|
||||
g.Log().Infof(ctx, "============================================")
|
||||
|
||||
g.Log().Info(ctx, "service started on :8080")
|
||||
|
||||
<-ctx.Done()
|
||||
g.Log().Info(ctx, "shutting down...")
|
||||
time.Sleep(3 * time.Second)
|
||||
g.Log().Info(ctx, "bye")
|
||||
}
|
||||
@@ -0,0 +1,3 @@
|
||||
node_modules/
|
||||
dist/
|
||||
.env.local
|
||||
@@ -0,0 +1,12 @@
|
||||
<!DOCTYPE html>
|
||||
<html lang="zh-CN">
|
||||
<head>
|
||||
<meta charset="utf-8">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1.0">
|
||||
<title>本地知识库</title>
|
||||
</head>
|
||||
<body>
|
||||
<div id="app"></div>
|
||||
<script type="module" src="/src/main.js"></script>
|
||||
</body>
|
||||
</html>
|
||||
Generated
+1862
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,23 @@
|
||||
{
|
||||
"name": "rag-local-ui",
|
||||
"version": "1.0.0",
|
||||
"private": true,
|
||||
"type": "module",
|
||||
"scripts": {
|
||||
"dev": "vite",
|
||||
"build": "vite build",
|
||||
"preview": "vite preview"
|
||||
},
|
||||
"dependencies": {
|
||||
"@element-plus/icons-vue": "^2.3.0",
|
||||
"axios": "^1.6.0",
|
||||
"element-plus": "^2.5.0",
|
||||
"pinia": "^2.1.0",
|
||||
"vue": "^3.4.0",
|
||||
"vue-router": "^4.3.0"
|
||||
},
|
||||
"devDependencies": {
|
||||
"@vitejs/plugin-vue": "^5.0.0",
|
||||
"vite": "^5.0.0"
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,3 @@
|
||||
<template>
|
||||
<router-view />
|
||||
</template>
|
||||
@@ -0,0 +1,21 @@
|
||||
import request from './request.js'
|
||||
|
||||
export function login(data) {
|
||||
return request.post('/system-config/login', data)
|
||||
}
|
||||
|
||||
export function getSystemConfig() {
|
||||
return request.get('/system-config')
|
||||
}
|
||||
|
||||
export function updateSystemConfig(data) {
|
||||
return request.put('/system-config', data)
|
||||
}
|
||||
|
||||
export function regenerateToken() {
|
||||
return request.post('/system-config/regenerate-token')
|
||||
}
|
||||
|
||||
export function getToken() {
|
||||
return request.get('/system-config/token')
|
||||
}
|
||||
@@ -0,0 +1,73 @@
|
||||
import request from './request.js'
|
||||
|
||||
export function listConversations() {
|
||||
return request.get('/conversation/list')
|
||||
}
|
||||
|
||||
export function saveConversation(data) {
|
||||
return request.post('/conversation/save', data)
|
||||
}
|
||||
|
||||
export function deleteConversation(id) {
|
||||
return request.post('/conversation/delete', { id })
|
||||
}
|
||||
|
||||
export function listMessages(conversationId) {
|
||||
return request.get('/message/list', { params: { conversation_id: conversationId } })
|
||||
}
|
||||
|
||||
// streamChat RAG 问答 SSE：handlers.onCitations / onDelta / onDone / onError
|
||||
export async function streamChat(payload, handlers) {
|
||||
const token = localStorage.getItem('token')
|
||||
let resp
|
||||
try {
|
||||
resp = await fetch('/message/chat', {
|
||||
method: 'POST',
|
||||
headers: {
|
||||
'Content-Type': 'application/json',
|
||||
'Authorization': token ? 'Bearer ' + token : ''
|
||||
},
|
||||
body: JSON.stringify(payload)
|
||||
})
|
||||
} catch (e) {
|
||||
handlers.onError(new Error('网络错误，请检查服务是否启动'))
|
||||
return
|
||||
}
|
||||
if (!resp.ok || !resp.body) {
|
||||
handlers.onError(new Error('请求失败：' + resp.status))
|
||||
return
|
||||
}
|
||||
const reader = resp.body.getReader()
|
||||
const decoder = new TextDecoder()
|
||||
let buf = ''
|
||||
try {
|
||||
while (true) {
|
||||
const { done, value } = await reader.read()
|
||||
if (done) break
|
||||
buf += decoder.decode(value, { stream: true })
|
||||
let idx
|
||||
while ((idx = buf.indexOf('\n\n')) >= 0) {
|
||||
const block = buf.slice(0, idx)
|
||||
buf = buf.slice(idx + 2)
|
||||
for (const line of block.split('\n')) {
|
||||
if (!line.startsWith('data: ')) continue
|
||||
const data = line.slice(6).trim()
|
||||
if (!data || data === '[DONE]') continue
|
||||
let obj
|
||||
try {
|
||||
obj = JSON.parse(data)
|
||||
} catch {
|
||||
continue
|
||||
}
|
||||
if (obj.citations !== undefined) handlers.onCitations(obj)
|
||||
else if (obj.content !== undefined) handlers.onDelta(obj.content)
|
||||
else if (obj.status === 'ok') handlers.onDone()
|
||||
else if (obj.message) handlers.onError(new Error(obj.message))
|
||||
}
|
||||
}
|
||||
}
|
||||
handlers.onDone()
|
||||
} catch (e) {
|
||||
handlers.onError(e)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,9 @@
|
||||
import request from './request.js'
|
||||
|
||||
export function listChunks(params) {
|
||||
return request.get('/chunk/list', { params })
|
||||
}
|
||||
|
||||
export function updateChunk(data) {
|
||||
return request.post('/chunk/update', data)
|
||||
}
|
||||
@@ -0,0 +1,13 @@
|
||||
import request from './request.js'
|
||||
|
||||
export function listDatasets() {
|
||||
return request.get('/dataset/list')
|
||||
}
|
||||
|
||||
export function saveDataset(data) {
|
||||
return request.post('/dataset/save', data)
|
||||
}
|
||||
|
||||
export function deleteDataset(id) {
|
||||
return request.post('/dataset/delete', { id })
|
||||
}
|
||||
@@ -0,0 +1,21 @@
|
||||
import request from './request.js'
|
||||
|
||||
export function listDocuments(params) {
|
||||
return request.get('/document/list', { params })
|
||||
}
|
||||
|
||||
export function uploadDocument(formData) {
|
||||
return request.post('/document/upload', formData, { headers: { 'Content-Type': 'multipart/form-data' } })
|
||||
}
|
||||
|
||||
export function deleteDocument(id) {
|
||||
return request.post('/document/delete', { id })
|
||||
}
|
||||
|
||||
export function reembedDocument(id) {
|
||||
return request.post('/document/reembed', { id })
|
||||
}
|
||||
|
||||
export function retryParseTask(id) {
|
||||
return request.post('/parse-task/retry', { id })
|
||||
}
|
||||
@@ -0,0 +1,9 @@
|
||||
import request from './request.js'
|
||||
|
||||
export function listEntities(params) {
|
||||
return request.get('/kg-entity/list', { params })
|
||||
}
|
||||
|
||||
export function listRelations(params) {
|
||||
return request.get('/kg-relation/list', { params })
|
||||
}
|
||||
@@ -0,0 +1,17 @@
|
||||
import request from './request.js'
|
||||
|
||||
export function listModelConfigs(modelType) {
|
||||
return request.get('/model-config/list', { params: { model_type: modelType } })
|
||||
}
|
||||
|
||||
export function saveModelConfig(data) {
|
||||
return request.post('/model-config/save', data)
|
||||
}
|
||||
|
||||
export function deleteModelConfig(id) {
|
||||
return request.post('/model-config/delete', { id })
|
||||
}
|
||||
|
||||
export function testModelConfig(id) {
|
||||
return request.post('/model-config/test', { id })
|
||||
}
|
||||
@@ -0,0 +1,43 @@
|
||||
import axios from 'axios'
|
||||
import { ElMessage } from 'element-plus'
|
||||
|
||||
const request = axios.create({
|
||||
timeout: 300000
|
||||
})
|
||||
|
||||
request.interceptors.request.use(config => {
|
||||
const token = localStorage.getItem('token')
|
||||
if (token) {
|
||||
config.headers.Authorization = 'Bearer ' + token
|
||||
}
|
||||
return config
|
||||
})
|
||||
|
||||
function redirectLogin() {
|
||||
localStorage.removeItem('token')
|
||||
window.location.href = '/#/login'
|
||||
}
|
||||
|
||||
request.interceptors.response.use(
|
||||
response => {
|
||||
const data = response.data
|
||||
if (data.code !== 0) {
|
||||
if (data.code === 401) {
|
||||
redirectLogin()
|
||||
return Promise.reject(new Error(data.message))
|
||||
}
|
||||
ElMessage.error(data.message || '请求失败')
|
||||
return Promise.reject(new Error(data.message))
|
||||
}
|
||||
return data.data
|
||||
},
|
||||
error => {
|
||||
if (error.response?.status === 401) {
|
||||
redirectLogin()
|
||||
}
|
||||
ElMessage.error(error.message || '网络错误')
|
||||
return Promise.reject(error)
|
||||
}
|
||||
)
|
||||
|
||||
export default request
|
||||
@@ -0,0 +1,13 @@
|
||||
import { createApp } from 'vue'
|
||||
import { createPinia } from 'pinia'
|
||||
import ElementPlus from 'element-plus'
|
||||
import 'element-plus/dist/index.css'
|
||||
import zhCn from 'element-plus/dist/locale/zh-cn.mjs'
|
||||
import App from './App.vue'
|
||||
import router from './router'
|
||||
|
||||
const app = createApp(App)
|
||||
app.use(createPinia())
|
||||
app.use(ElementPlus, { locale: zhCn })
|
||||
app.use(router)
|
||||
app.mount('#app')
|
||||
@@ -0,0 +1,69 @@
|
||||
import { createRouter, createWebHashHistory } from 'vue-router'
|
||||
import { useAuthStore } from '../stores/auth.js'
|
||||
|
||||
const routes = [
|
||||
{
|
||||
path: '/login',
|
||||
name: 'Login',
|
||||
component: () => import('../views/Login.vue')
|
||||
},
|
||||
{
|
||||
path: '/',
|
||||
component: () => import('../views/Layout.vue'),
|
||||
redirect: '/datasets',
|
||||
children: [
|
||||
{
|
||||
path: 'datasets',
|
||||
name: 'DatasetList',
|
||||
meta: { title: '数据集' },
|
||||
component: () => import('../views/DatasetList.vue')
|
||||
},
|
||||
{
|
||||
path: 'datasets/:id',
|
||||
name: 'DatasetDetail',
|
||||
meta: { title: '数据集详情' },
|
||||
component: () => import('../views/DatasetDetail.vue')
|
||||
},
|
||||
{
|
||||
path: 'chat',
|
||||
name: 'Chat',
|
||||
meta: { title: '问答' },
|
||||
component: () => import('../views/Chat.vue')
|
||||
},
|
||||
{
|
||||
path: 'kg',
|
||||
name: 'KgGraph',
|
||||
meta: { title: '知识图谱' },
|
||||
component: () => import('../views/KgGraph.vue')
|
||||
},
|
||||
{
|
||||
path: 'settings',
|
||||
name: 'Settings',
|
||||
meta: { title: '设置' },
|
||||
component: () => import('../views/Settings.vue')
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
|
||||
const router = createRouter({
|
||||
// hash 路由：前后端合并部署后 SPA 路由与后端 API 前缀重叠，
|
||||
// history 模式刷新会命中 API 路由；hash 模式页面始终请求 /，由后端静态服务返回 index.html
|
||||
history: createWebHashHistory(),
|
||||
routes
|
||||
})
|
||||
|
||||
router.beforeEach((to, from, next) => {
|
||||
const auth = useAuthStore()
|
||||
if (to.path !== '/login' && !auth.isLoggedIn) {
|
||||
next('/login')
|
||||
return
|
||||
}
|
||||
if (to.path === '/login' && auth.isLoggedIn) {
|
||||
next('/')
|
||||
return
|
||||
}
|
||||
next()
|
||||
})
|
||||
|
||||
export default router
|
||||
@@ -0,0 +1,23 @@
|
||||
import { defineStore } from 'pinia'
|
||||
import { ref, computed } from 'vue'
|
||||
import { login } from '../api/auth.js'
|
||||
|
||||
export const useAuthStore = defineStore('auth', () => {
|
||||
const token = ref(localStorage.getItem('token') || '')
|
||||
|
||||
const isLoggedIn = computed(() => !!token.value)
|
||||
|
||||
async function doLogin(accessToken) {
|
||||
const res = await login({ token: accessToken })
|
||||
token.value = res.token
|
||||
localStorage.setItem('token', res.token)
|
||||
return res
|
||||
}
|
||||
|
||||
function logout() {
|
||||
token.value = ''
|
||||
localStorage.removeItem('token')
|
||||
}
|
||||
|
||||
return { token, isLoggedIn, doLogin, logout }
|
||||
})
|
||||
@@ -0,0 +1,302 @@
|
||||
<template>
|
||||
<div class="chat-page">
|
||||
<!-- 左侧：会话列表 -->
|
||||
<div class="chat-side">
|
||||
<div class="side-head">
|
||||
<el-button type="primary" style="width: 100%" @click="newConversation">新建会话</el-button>
|
||||
<el-select v-model="datasetId" placeholder="选择知识库" style="width: 100%; margin-top: 10px"
|
||||
@change="onDatasetChange">
|
||||
<el-option v-for="d in datasets" :key="d.id" :label="d.name" :value="d.id" />
|
||||
</el-select>
|
||||
</div>
|
||||
<div class="conv-list">
|
||||
<div v-for="c in conversations" :key="c.id" class="conv-item"
|
||||
:class="{ active: c.id === currentConvId }" @click="openConversation(c)">
|
||||
<span class="conv-title">{{ c.title || '新会话' }}</span>
|
||||
<el-icon class="conv-del" title="删除" @click.stop="removeConversation(c)"><Delete /></el-icon>
|
||||
</div>
|
||||
<el-empty v-if="!conversations.length" description="暂无会话" :image-size="60" />
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<!-- 右侧：对话区 -->
|
||||
<div class="chat-main">
|
||||
<div ref="msgBox" class="msg-area">
|
||||
<template v-for="(m, i) in messages" :key="i">
|
||||
<div class="msg-row" :class="m.role">
|
||||
<div class="msg-bubble">
|
||||
<div class="msg-content">{{ m.content || (m.streaming ? '…' : '') }}</div>
|
||||
<div v-if="m.citations && m.citations.length" class="msg-citations">
|
||||
<el-collapse>
|
||||
<el-collapse-item :title="'引用来源（' + m.citations.length + '）'">
|
||||
<div v-for="c in m.citations" :key="c.index" class="citation-item">
|
||||
<div class="citation-head">
|
||||
<span class="citation-idx">[{{ c.index }}]</span>
|
||||
<el-tag v-for="s in c.sources" :key="s" size="small" type="info">{{ s }}</el-tag>
|
||||
<span class="citation-score">得分 {{ c.score.toFixed(4) }}</span>
|
||||
</div>
|
||||
<div class="citation-content">{{ c.content }}</div>
|
||||
</div>
|
||||
</el-collapse-item>
|
||||
</el-collapse>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</template>
|
||||
</div>
|
||||
<div class="msg-input">
|
||||
<el-input v-model="input" type="textarea" :rows="3" resize="none" placeholder="输入问题，回车发送（Shift+Enter 换行）"
|
||||
:disabled="streaming" @keydown.enter.exact.prevent="send" />
|
||||
<el-button type="primary" :loading="streaming" :disabled="!input.trim()" @click="send" style="margin-left: 10px">
|
||||
{{ streaming ? '生成中' : '发送' }}
|
||||
</el-button>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</template>
|
||||
|
||||
<script setup>
|
||||
import { onMounted, nextTick, ref } from 'vue'
|
||||
import { ElMessage, ElMessageBox } from 'element-plus'
|
||||
import { Delete } from '@element-plus/icons-vue'
|
||||
import { listDatasets } from '../api/dataset.js'
|
||||
import { listConversations, saveConversation, deleteConversation, listMessages, streamChat } from '../api/chat.js'
|
||||
|
||||
const conversations = ref([])
|
||||
const datasets = ref([])
|
||||
const currentConvId = ref(0)
|
||||
const datasetId = ref(null)
|
||||
const messages = ref([])
|
||||
const input = ref('')
|
||||
const streaming = ref(false)
|
||||
const msgBox = ref(null)
|
||||
|
||||
onMounted(async () => {
|
||||
try {
|
||||
datasets.value = await listDatasets()
|
||||
if (datasets.value.length && !datasetId.value) {
|
||||
datasetId.value = datasets.value[0].id
|
||||
}
|
||||
} catch { /* 忽略 */ }
|
||||
await refreshConversations()
|
||||
})
|
||||
|
||||
function scrollBottom() {
|
||||
nextTick(() => {
|
||||
if (msgBox.value) msgBox.value.scrollTop = msgBox.value.scrollHeight
|
||||
})
|
||||
}
|
||||
|
||||
async function refreshConversations() {
|
||||
conversations.value = await listConversations()
|
||||
}
|
||||
|
||||
function newConversation() {
|
||||
currentConvId.value = 0
|
||||
messages.value = []
|
||||
}
|
||||
|
||||
async function openConversation(c) {
|
||||
currentConvId.value = c.id
|
||||
if (c.dataset_id) datasetId.value = c.dataset_id
|
||||
const list = await listMessages(c.id)
|
||||
messages.value = list.map(m => {
|
||||
let citations = []
|
||||
if (m.citations) {
|
||||
try {
|
||||
citations = JSON.parse(m.citations) || []
|
||||
} catch { /* 忽略格式异常 */ }
|
||||
}
|
||||
return { role: m.role, content: m.content, citations }
|
||||
})
|
||||
scrollBottom()
|
||||
}
|
||||
|
||||
async function removeConversation(c) {
|
||||
try {
|
||||
await ElMessageBox.confirm('删除会话将同时删除历史消息，确认？', '删除会话', { type: 'warning' })
|
||||
} catch {
|
||||
return
|
||||
}
|
||||
await deleteConversation(c.id)
|
||||
if (currentConvId.value === c.id) newConversation()
|
||||
await refreshConversations()
|
||||
}
|
||||
|
||||
function onDatasetChange() {
|
||||
// 切换知识库后，正在进行的会话归零（会话绑定数据集）
|
||||
newConversation()
|
||||
}
|
||||
|
||||
async function send() {
|
||||
const question = input.value.trim()
|
||||
if (!question || streaming.value) return
|
||||
if (!datasetId.value) {
|
||||
ElMessage.warning('请先选择知识库')
|
||||
return
|
||||
}
|
||||
input.value = ''
|
||||
messages.value.push({ role: 'user', content: question })
|
||||
const aiMsg = ref({ role: 'assistant', content: '', citations: [], streaming: true })
|
||||
messages.value.push(aiMsg.value)
|
||||
scrollBottom()
|
||||
streaming.value = true
|
||||
const payload = { conversation_id: currentConvId.value, dataset_id: datasetId.value, question }
|
||||
await streamChat(payload, {
|
||||
onCitations(data) {
|
||||
aiMsg.value.citations = data.citations || []
|
||||
if (data.conversation_id && data.conversation_id !== currentConvId.value) {
|
||||
currentConvId.value = data.conversation_id
|
||||
refreshConversations()
|
||||
}
|
||||
scrollBottom()
|
||||
},
|
||||
onDelta(delta) {
|
||||
aiMsg.value.content += delta
|
||||
scrollBottom()
|
||||
},
|
||||
onDone() {
|
||||
aiMsg.value.streaming = false
|
||||
streaming.value = false
|
||||
scrollBottom()
|
||||
},
|
||||
onError(err) {
|
||||
aiMsg.value.streaming = false
|
||||
streaming.value = false
|
||||
aiMsg.value.content = aiMsg.value.content || ''
|
||||
ElMessage.error(err.message || '生成失败')
|
||||
}
|
||||
})
|
||||
}
|
||||
</script>
|
||||
|
||||
<style scoped>
|
||||
.chat-page {
|
||||
display: flex;
|
||||
height: calc(100vh - 110px);
|
||||
gap: 12px;
|
||||
}
|
||||
.chat-side {
|
||||
width: 260px;
|
||||
flex-shrink: 0;
|
||||
display: flex;
|
||||
flex-direction: column;
|
||||
border: 1px solid #e4e7ed;
|
||||
border-radius: 8px;
|
||||
background: #fff;
|
||||
overflow: hidden;
|
||||
}
|
||||
.side-head {
|
||||
padding: 12px;
|
||||
border-bottom: 1px solid #f0f0f0;
|
||||
}
|
||||
.conv-list {
|
||||
flex: 1;
|
||||
overflow-y: auto;
|
||||
padding: 8px;
|
||||
}
|
||||
.conv-item {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
justify-content: space-between;
|
||||
padding: 8px 10px;
|
||||
border-radius: 6px;
|
||||
cursor: pointer;
|
||||
color: #606266;
|
||||
}
|
||||
.conv-item:hover, .conv-item.active {
|
||||
background: #ecf5ff;
|
||||
color: #409eff;
|
||||
}
|
||||
.conv-title {
|
||||
flex: 1;
|
||||
overflow: hidden;
|
||||
text-overflow: ellipsis;
|
||||
white-space: nowrap;
|
||||
}
|
||||
.conv-del {
|
||||
visibility: hidden;
|
||||
color: #909399;
|
||||
}
|
||||
.conv-item:hover .conv-del {
|
||||
visibility: visible;
|
||||
}
|
||||
.chat-main {
|
||||
flex: 1;
|
||||
display: flex;
|
||||
flex-direction: column;
|
||||
border: 1px solid #e4e7ed;
|
||||
border-radius: 8px;
|
||||
background: #fff;
|
||||
overflow: hidden;
|
||||
}
|
||||
.msg-area {
|
||||
flex: 1;
|
||||
overflow-y: auto;
|
||||
padding: 16px;
|
||||
background: #fafafa;
|
||||
}
|
||||
.msg-row {
|
||||
display: flex;
|
||||
margin-bottom: 14px;
|
||||
}
|
||||
.msg-row.user {
|
||||
justify-content: flex-end;
|
||||
}
|
||||
.msg-bubble {
|
||||
max-width: 72%;
|
||||
padding: 10px 14px;
|
||||
border-radius: 8px;
|
||||
font-size: 14px;
|
||||
line-height: 1.7;
|
||||
white-space: pre-wrap;
|
||||
word-break: break-word;
|
||||
}
|
||||
.msg-row.user .msg-bubble {
|
||||
background: #409eff;
|
||||
color: #fff;
|
||||
border-top-right-radius: 2px;
|
||||
}
|
||||
.msg-row.assistant .msg-bubble {
|
||||
background: #fff;
|
||||
border: 1px solid #e4e7ed;
|
||||
border-top-left-radius: 2px;
|
||||
}
|
||||
.msg-citations {
|
||||
margin-top: 8px;
|
||||
}
|
||||
.citation-item {
|
||||
padding: 4px 0;
|
||||
border-bottom: 1px dashed #ebeef5;
|
||||
}
|
||||
.citation-item:last-child {
|
||||
border-bottom: none;
|
||||
}
|
||||
.citation-head {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
gap: 6px;
|
||||
}
|
||||
.citation-idx {
|
||||
font-weight: 600;
|
||||
color: #409eff;
|
||||
}
|
||||
.citation-score {
|
||||
font-size: 12px;
|
||||
color: #909399;
|
||||
}
|
||||
.citation-content {
|
||||
margin-top: 4px;
|
||||
font-size: 13px;
|
||||
color: #606266;
|
||||
line-height: 1.6;
|
||||
}
|
||||
.msg-input {
|
||||
display: flex;
|
||||
padding: 12px;
|
||||
border-top: 1px solid #f0f0f0;
|
||||
}
|
||||
.msg-input .el-textarea {
|
||||
flex: 1;
|
||||
}
|
||||
</style>
|
||||
@@ -0,0 +1,264 @@
|
||||
<template>
|
||||
<div class="dd-page">
|
||||
<div class="dd-head">
|
||||
<el-button link @click="$router.push('/datasets')">
|
||||
<el-icon><ArrowLeft /></el-icon> 返回
|
||||
</el-button>
|
||||
<span class="dd-title">{{ datasetName }}</span>
|
||||
</div>
|
||||
|
||||
<el-upload class="dd-upload" drag :show-file-list="false" :http-request="doUpload" :disabled="uploading"
|
||||
accept=".txt,.md,.pdf,.docx,.doc,.html,.htm">
|
||||
<el-icon class="el-icon--upload"><UploadFilled /></el-icon>
|
||||
<div class="el-upload__text">拖拽文件到此处，或 <em>点击上传</em>（txt/md/pdf/docx/html）</div>
|
||||
</el-upload>
|
||||
|
||||
<el-table :data="documents" v-loading="loading" empty-text="暂无文档">
|
||||
<el-table-column prop="filename" label="文件名" min-width="200" show-overflow-tooltip />
|
||||
<el-table-column label="大小" width="100">
|
||||
<template #default="{ row }">{{ formatSize(row.file_size) }}</template>
|
||||
</el-table-column>
|
||||
<el-table-column label="状态" width="120">
|
||||
<template #default="{ row }">
|
||||
<el-tooltip :content="row.error_msg || ''" placement="top">
|
||||
<el-tag :type="statusType(row.status)">{{ statusText(row.status) }}</el-tag>
|
||||
</el-tooltip>
|
||||
</template>
|
||||
</el-table-column>
|
||||
<el-table-column prop="chunk_count" label="分块数" width="90" />
|
||||
<el-table-column prop="created_at" label="上传时间" width="170" />
|
||||
<el-table-column label="操作" width="240" fixed="right">
|
||||
<template #default="{ row }">
|
||||
<el-button link type="primary" @click="openChunks(row)">分块</el-button>
|
||||
<el-button v-if="row.status === 3" link type="warning" @click="retry(row)">重试</el-button>
|
||||
<el-button link type="primary" @click="reembed(row)">重新向量化</el-button>
|
||||
<el-button link type="danger" @click="remove(row)">删除</el-button>
|
||||
</template>
|
||||
</el-table-column>
|
||||
</el-table>
|
||||
<el-pagination class="dd-page-bar" layout="total, prev, pager, next" :total="total"
|
||||
:page-size="pageSize" v-model:current-page="page" @current-change="load" />
|
||||
|
||||
<el-drawer v-model="chunkDrawer" :title="chunkDrawerTitle" size="55%">
|
||||
<div class="chunk-toolbar">
|
||||
<el-input v-model="chunkKeyword" placeholder="搜索分块内容" clearable style="width: 240px" @change="loadChunks" />
|
||||
</div>
|
||||
<el-table :data="chunks" v-loading="chunkLoading" size="small">
|
||||
<el-table-column type="index" label="#" width="50" />
|
||||
<el-table-column label="内容" min-width="300">
|
||||
<template #default="{ row }">
|
||||
<div class="chunk-content">{{ row.content }}</div>
|
||||
</template>
|
||||
</el-table-column>
|
||||
<el-table-column label="操作" width="110" fixed="right">
|
||||
<template #default="{ row }">
|
||||
<el-button link type="primary" @click="editChunk(row)">编辑</el-button>
|
||||
</template>
|
||||
</el-table-column>
|
||||
</el-table>
|
||||
<el-pagination class="dd-page-bar" layout="total, prev, pager, next" :total="chunkTotal"
|
||||
:page-size="chunkPageSize" v-model:current-page="chunkPage" @current-change="loadChunks" />
|
||||
</el-drawer>
|
||||
|
||||
<el-dialog v-model="editDialog" title="编辑分块" width="640px">
|
||||
<el-input v-model="editContent" type="textarea" :rows="10" />
|
||||
<div class="edit-tip">保存后重新分词并向量化，向量模型未绑定时仅更新全文索引</div>
|
||||
<template #footer>
|
||||
<el-button @click="editDialog = false">取消</el-button>
|
||||
<el-button type="primary" :loading="editSaving" @click="saveChunk">保存</el-button>
|
||||
</template>
|
||||
</el-dialog>
|
||||
</div>
|
||||
</template>
|
||||
|
||||
<script setup>
|
||||
import { computed, onMounted, ref } from 'vue'
|
||||
import { useRoute } from 'vue-router'
|
||||
import { ElMessage, ElMessageBox } from 'element-plus'
|
||||
import { ArrowLeft, UploadFilled } from '@element-plus/icons-vue'
|
||||
import { listDatasets } from '../api/dataset.js'
|
||||
import { listDocuments, uploadDocument, deleteDocument, reembedDocument, retryParseTask } from '../api/document.js'
|
||||
import { listChunks, updateChunk } from '../api/chunk.js'
|
||||
|
||||
const route = useRoute()
|
||||
const datasetId = computed(() => Number(route.params.id))
|
||||
const datasetName = ref('')
|
||||
|
||||
const documents = ref([])
|
||||
const total = ref(0)
|
||||
const page = ref(1)
|
||||
const pageSize = 20
|
||||
const loading = ref(false)
|
||||
const uploading = ref(false)
|
||||
|
||||
const chunkDrawer = ref(false)
|
||||
const chunkDrawerTitle = ref('')
|
||||
const chunks = ref([])
|
||||
const chunkTotal = ref(0)
|
||||
const chunkPage = ref(1)
|
||||
const chunkPageSize = 20
|
||||
const chunkLoading = ref(false)
|
||||
const chunkKeyword = ref('')
|
||||
let currentDoc = null
|
||||
|
||||
const editDialog = ref(false)
|
||||
const editContent = ref('')
|
||||
const editSaving = ref(false)
|
||||
let editingChunk = null
|
||||
|
||||
const statusMap = {
|
||||
0: { text: '待处理', type: 'info' },
|
||||
1: { text: '解析中', type: 'warning' },
|
||||
2: { text: '已完成', type: 'success' },
|
||||
3: { text: '失败', type: 'danger' }
|
||||
}
|
||||
|
||||
function statusText(s) { return (statusMap[s] || {}).text || s }
|
||||
function statusType(s) { return (statusMap[s] || {}).type || 'info' }
|
||||
|
||||
function formatSize(bytes) {
|
||||
if (!bytes) return '-'
|
||||
if (bytes < 1024) return bytes + ' B'
|
||||
if (bytes < 1024 * 1024) return (bytes / 1024).toFixed(1) + ' KB'
|
||||
return (bytes / 1024 / 1024).toFixed(1) + ' MB'
|
||||
}
|
||||
|
||||
onMounted(async () => {
|
||||
try {
|
||||
const ds = await listDatasets()
|
||||
const cur = ds.find(x => x.id === datasetId.value)
|
||||
datasetName.value = cur ? cur.name : '数据集'
|
||||
} catch { /* 忽略 */ }
|
||||
await load()
|
||||
})
|
||||
|
||||
async function load() {
|
||||
loading.value = true
|
||||
try {
|
||||
const d = await listDocuments({ dataset_id: datasetId.value, page: page.value, page_size: pageSize })
|
||||
documents.value = d.list
|
||||
total.value = d.total
|
||||
} finally {
|
||||
loading.value = false
|
||||
}
|
||||
}
|
||||
|
||||
async function doUpload({ file }) {
|
||||
uploading.value = true
|
||||
try {
|
||||
const formData = new FormData()
|
||||
formData.append('dataset_id', datasetId.value)
|
||||
formData.append('file', file)
|
||||
await uploadDocument(formData)
|
||||
ElMessage.success('上传成功，正在解析')
|
||||
await load()
|
||||
} finally {
|
||||
uploading.value = false
|
||||
}
|
||||
}
|
||||
|
||||
async function remove(row) {
|
||||
try {
|
||||
await ElMessageBox.confirm(`删除文档「${row.filename}」？将同时删除分块、向量与全文索引。`, '删除确认', { type: 'warning' })
|
||||
} catch {
|
||||
return
|
||||
}
|
||||
await deleteDocument(row.id)
|
||||
ElMessage.success('已删除')
|
||||
await load()
|
||||
}
|
||||
|
||||
async function retry(row) {
|
||||
await retryParseTask(row.id)
|
||||
ElMessage.success('已重新入队')
|
||||
await load()
|
||||
}
|
||||
|
||||
async function reembed(row) {
|
||||
try {
|
||||
await ElMessageBox.confirm(`重新向量化「${row.filename}」的全部分块？`, '确认', { type: 'info' })
|
||||
} catch {
|
||||
return
|
||||
}
|
||||
await reembedDocument(row.id)
|
||||
ElMessage.success('重新向量化完成')
|
||||
}
|
||||
|
||||
async function openChunks(row) {
|
||||
currentDoc = row
|
||||
chunkDrawerTitle.value = `分块列表 - ${row.filename}`
|
||||
chunkDrawer.value = true
|
||||
chunkPage.value = 1
|
||||
chunkKeyword.value = ''
|
||||
await loadChunks()
|
||||
}
|
||||
|
||||
async function loadChunks() {
|
||||
if (!currentDoc) return
|
||||
chunkLoading.value = true
|
||||
try {
|
||||
const d = await listChunks({ document_id: currentDoc.id, page: chunkPage.value, page_size: chunkPageSize })
|
||||
chunks.value = d.list
|
||||
chunkTotal.value = d.total
|
||||
} finally {
|
||||
chunkLoading.value = false
|
||||
}
|
||||
}
|
||||
|
||||
function editChunk(row) {
|
||||
editingChunk = row
|
||||
editContent.value = row.content
|
||||
editDialog.value = true
|
||||
}
|
||||
|
||||
async function saveChunk() {
|
||||
if (!editContent.value.trim()) {
|
||||
ElMessage.warning('内容不能为空')
|
||||
return
|
||||
}
|
||||
editSaving.value = true
|
||||
try {
|
||||
await updateChunk({ id: editingChunk.id, content: editContent.value })
|
||||
ElMessage.success('已保存并重新向量化')
|
||||
editDialog.value = false
|
||||
await loadChunks()
|
||||
} finally {
|
||||
editSaving.value = false
|
||||
}
|
||||
}
|
||||
</script>
|
||||
|
||||
<style scoped>
|
||||
.dd-head {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
gap: 8px;
|
||||
margin-bottom: 12px;
|
||||
}
|
||||
.dd-title {
|
||||
font-size: 16px;
|
||||
font-weight: 600;
|
||||
}
|
||||
.dd-upload {
|
||||
margin-bottom: 12px;
|
||||
}
|
||||
.dd-page-bar {
|
||||
margin-top: 10px;
|
||||
justify-content: flex-end;
|
||||
}
|
||||
.chunk-toolbar {
|
||||
margin-bottom: 10px;
|
||||
}
|
||||
.chunk-content {
|
||||
max-height: 60px;
|
||||
overflow: hidden;
|
||||
font-size: 13px;
|
||||
color: #606266;
|
||||
line-height: 1.6;
|
||||
}
|
||||
.edit-tip {
|
||||
margin-top: 6px;
|
||||
font-size: 12px;
|
||||
color: #909399;
|
||||
}
|
||||
</style>
|
||||
@@ -0,0 +1,121 @@
|
||||
<template>
|
||||
<div class="ds-page">
|
||||
<div class="ds-head">
|
||||
<el-button type="primary" @click="openCreate">新建数据集</el-button>
|
||||
</div>
|
||||
<el-table :data="datasets" v-loading="loading" empty-text="暂无数据集">
|
||||
<el-table-column prop="name" label="名称" min-width="160" show-overflow-tooltip />
|
||||
<el-table-column prop="description" label="描述" min-width="200" show-overflow-tooltip />
|
||||
<el-table-column label="向量模型" width="200">
|
||||
<template #default="{ row }">
|
||||
<span v-if="row.embedding_cfg_id">{{ embeddingName(row.embedding_cfg_id) }}</span>
|
||||
<el-tag v-else size="small" type="info">未绑定（仅全文检索）</el-tag>
|
||||
</template>
|
||||
</el-table-column>
|
||||
<el-table-column prop="created_at" label="创建时间" width="170" />
|
||||
<el-table-column label="操作" width="150" fixed="right">
|
||||
<template #default="{ row }">
|
||||
<el-button link type="primary" @click="$router.push('/datasets/' + row.id)">详情</el-button>
|
||||
<el-button link type="danger" @click="remove(row)">删除</el-button>
|
||||
</template>
|
||||
</el-table-column>
|
||||
</el-table>
|
||||
|
||||
<el-dialog v-model="dialogVisible" :title="editing ? '编辑数据集' : '新建数据集'" width="480px">
|
||||
<el-form :model="form" label-width="90px">
|
||||
<el-form-item label="名称" required>
|
||||
<el-input v-model="form.name" placeholder="如：产品手册" />
|
||||
</el-form-item>
|
||||
<el-form-item label="描述">
|
||||
<el-input v-model="form.description" type="textarea" :rows="2" />
|
||||
</el-form-item>
|
||||
<el-form-item label="向量模型">
|
||||
<el-select v-model="form.embedding_cfg_id" clearable placeholder="不绑定则仅全文检索" style="width: 100%">
|
||||
<el-option v-for="m in embedders" :key="m.id" :label="m.name" :value="m.id" />
|
||||
</el-select>
|
||||
</el-form-item>
|
||||
</el-form>
|
||||
<template #footer>
|
||||
<el-button @click="dialogVisible = false">取消</el-button>
|
||||
<el-button type="primary" :loading="saving" @click="save">保存</el-button>
|
||||
</template>
|
||||
</el-dialog>
|
||||
</div>
|
||||
</template>
|
||||
|
||||
<script setup>
|
||||
import { onMounted, ref } from 'vue'
|
||||
import { ElMessage, ElMessageBox } from 'element-plus'
|
||||
import { listDatasets, saveDataset, deleteDataset } from '../api/dataset.js'
|
||||
import { listModelConfigs } from '../api/model_config.js'
|
||||
|
||||
const datasets = ref([])
|
||||
const embedders = ref([])
|
||||
const loading = ref(false)
|
||||
const saving = ref(false)
|
||||
const dialogVisible = ref(false)
|
||||
const editing = ref(false)
|
||||
const form = ref({ id: 0, name: '', description: '', embedding_cfg_id: 0 })
|
||||
|
||||
onMounted(async () => {
|
||||
await load()
|
||||
try {
|
||||
embedders.value = await listModelConfigs('embedding')
|
||||
} catch { /* 忽略 */ }
|
||||
})
|
||||
|
||||
async function load() {
|
||||
loading.value = true
|
||||
try {
|
||||
datasets.value = await listDatasets()
|
||||
} finally {
|
||||
loading.value = false
|
||||
}
|
||||
}
|
||||
|
||||
function embeddingName(id) {
|
||||
const m = embedders.value.find(x => x.id === id)
|
||||
return m ? m.name : '（已删除的配置）'
|
||||
}
|
||||
|
||||
function openCreate() {
|
||||
editing.value = false
|
||||
form.value = { id: 0, name: '', description: '', embedding_cfg_id: 0 }
|
||||
dialogVisible.value = true
|
||||
}
|
||||
|
||||
async function save() {
|
||||
if (!form.value.name.trim()) {
|
||||
ElMessage.warning('请输入数据集名称')
|
||||
return
|
||||
}
|
||||
saving.value = true
|
||||
try {
|
||||
await saveDataset(form.value)
|
||||
ElMessage.success('保存成功')
|
||||
dialogVisible.value = false
|
||||
await load()
|
||||
} finally {
|
||||
saving.value = false
|
||||
}
|
||||
}
|
||||
|
||||
async function remove(row) {
|
||||
try {
|
||||
await ElMessageBox.confirm(`删除数据集「${row.name}」？`, '删除确认', { type: 'warning' })
|
||||
} catch {
|
||||
return
|
||||
}
|
||||
try {
|
||||
await deleteDataset(row.id)
|
||||
ElMessage.success('已删除')
|
||||
await load()
|
||||
} catch { /* request.js 已提示 */ }
|
||||
}
|
||||
</script>
|
||||
|
||||
<style scoped>
|
||||
.ds-head {
|
||||
margin-bottom: 12px;
|
||||
}
|
||||
</style>
|
||||
@@ -0,0 +1,113 @@
|
||||
<template>
|
||||
<div class="kg-page">
|
||||
<div class="kg-head">
|
||||
<el-select v-model="datasetId" placeholder="选择知识库" style="width: 260px" @change="load">
|
||||
<el-option v-for="d in datasets" :key="d.id" :label="d.name" :value="d.id" />
|
||||
</el-select>
|
||||
<span class="kg-tip">实体与关系由解析流水线中的 LLM 抽取生成，问答时命中实体自动注入一跳邻居</span>
|
||||
</div>
|
||||
|
||||
<el-row :gutter="12">
|
||||
<el-col :span="10">
|
||||
<el-card shadow="never" class="kg-card">
|
||||
<template #header>实体（{{ entityTotal }}）</template>
|
||||
<el-table :data="entities" size="small" v-loading="loading">
|
||||
<el-table-column prop="name" label="实体" min-width="140" show-overflow-tooltip />
|
||||
<el-table-column prop="entity_type" label="类型" width="110" />
|
||||
<el-table-column prop="chunk_id" label="来源分块" width="100" />
|
||||
</el-table>
|
||||
<el-pagination class="kg-page-bar" small layout="prev, pager, next" :total="entityTotal"
|
||||
:page-size="entityPageSize" v-model:current-page="entityPage" @current-change="loadEntities" />
|
||||
</el-card>
|
||||
</el-col>
|
||||
<el-col :span="14">
|
||||
<el-card shadow="never" class="kg-card">
|
||||
<template #header>关系（{{ relationTotal }}）</template>
|
||||
<el-table :data="relations" size="small" v-loading="loading">
|
||||
<el-table-column prop="head" label="主体" min-width="140" show-overflow-tooltip />
|
||||
<el-table-column prop="relation" label="关系" min-width="100" show-overflow-tooltip />
|
||||
<el-table-column prop="tail" label="客体" min-width="140" show-overflow-tooltip />
|
||||
<el-table-column prop="chunk_id" label="来源分块" width="100" />
|
||||
</el-table>
|
||||
<el-pagination class="kg-page-bar" small layout="prev, pager, next" :total="relationTotal"
|
||||
:page-size="relationPageSize" v-model:current-page="relationPage" @current-change="loadRelations" />
|
||||
</el-card>
|
||||
</el-col>
|
||||
</el-row>
|
||||
</div>
|
||||
</template>
|
||||
|
||||
<script setup>
|
||||
import { onMounted, ref } from 'vue'
|
||||
import { listDatasets } from '../api/dataset.js'
|
||||
import { listEntities, listRelations } from '../api/kg.js'
|
||||
|
||||
const datasets = ref([])
|
||||
const datasetId = ref(null)
|
||||
const loading = ref(false)
|
||||
|
||||
const entities = ref([])
|
||||
const entityTotal = ref(0)
|
||||
const entityPage = ref(1)
|
||||
const entityPageSize = 20
|
||||
|
||||
const relations = ref([])
|
||||
const relationTotal = ref(0)
|
||||
const relationPage = ref(1)
|
||||
const relationPageSize = 20
|
||||
|
||||
onMounted(async () => {
|
||||
try {
|
||||
datasets.value = await listDatasets()
|
||||
if (datasets.value.length) datasetId.value = datasets.value[0].id
|
||||
} catch { /* 忽略 */ }
|
||||
await load()
|
||||
})
|
||||
|
||||
async function load() {
|
||||
entityPage.value = 1
|
||||
relationPage.value = 1
|
||||
await Promise.all([loadEntities(), loadRelations()])
|
||||
}
|
||||
|
||||
async function loadEntities() {
|
||||
if (!datasetId.value) return
|
||||
loading.value = true
|
||||
try {
|
||||
const d = await listEntities({ dataset_id: datasetId.value, page: entityPage.value, page_size: entityPageSize })
|
||||
entities.value = d.list
|
||||
entityTotal.value = d.total
|
||||
} finally {
|
||||
loading.value = false
|
||||
}
|
||||
}
|
||||
|
||||
async function loadRelations() {
|
||||
if (!datasetId.value) return
|
||||
loading.value = true
|
||||
try {
|
||||
const d = await listRelations({ dataset_id: datasetId.value, page: relationPage.value, page_size: relationPageSize })
|
||||
relations.value = d.list
|
||||
relationTotal.value = d.total
|
||||
} finally {
|
||||
loading.value = false
|
||||
}
|
||||
}
|
||||
</script>
|
||||
|
||||
<style scoped>
|
||||
.kg-head {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
gap: 12px;
|
||||
margin-bottom: 12px;
|
||||
}
|
||||
.kg-tip {
|
||||
font-size: 13px;
|
||||
color: #909399;
|
||||
}
|
||||
.kg-page-bar {
|
||||
margin-top: 10px;
|
||||
justify-content: flex-end;
|
||||
}
|
||||
</style>
|
||||
@@ -0,0 +1,89 @@
|
||||
<template>
|
||||
<el-container class="layout">
|
||||
<el-aside width="200px" class="aside">
|
||||
<div class="logo">本地知识库</div>
|
||||
<el-menu :default-active="activeMenu" router class="menu">
|
||||
<el-menu-item index="/datasets">
|
||||
<el-icon><Folder /></el-icon>
|
||||
<span>数据集</span>
|
||||
</el-menu-item>
|
||||
<el-menu-item index="/chat">
|
||||
<el-icon><ChatDotRound /></el-icon>
|
||||
<span>问答</span>
|
||||
</el-menu-item>
|
||||
<el-menu-item index="/kg">
|
||||
<el-icon><Share /></el-icon>
|
||||
<span>知识图谱</span>
|
||||
</el-menu-item>
|
||||
<el-menu-item index="/settings">
|
||||
<el-icon><Setting /></el-icon>
|
||||
<span>设置</span>
|
||||
</el-menu-item>
|
||||
</el-menu>
|
||||
</el-aside>
|
||||
<el-container>
|
||||
<el-header class="header">
|
||||
<span>{{ route.meta.title || '' }}</span>
|
||||
<el-button link type="danger" @click="handleLogout">退出登录</el-button>
|
||||
</el-header>
|
||||
<el-main class="main">
|
||||
<router-view />
|
||||
</el-main>
|
||||
</el-container>
|
||||
</el-container>
|
||||
</template>
|
||||
|
||||
<script setup>
|
||||
import { computed } from 'vue'
|
||||
import { useRoute, useRouter } from 'vue-router'
|
||||
import { Folder, ChatDotRound, Setting, Share } from '@element-plus/icons-vue'
|
||||
import { useAuthStore } from '../stores/auth.js'
|
||||
|
||||
const route = useRoute()
|
||||
const router = useRouter()
|
||||
const auth = useAuthStore()
|
||||
|
||||
const activeMenu = computed(() => {
|
||||
if (route.path.startsWith('/datasets')) return '/datasets'
|
||||
return route.path
|
||||
})
|
||||
|
||||
function handleLogout() {
|
||||
auth.logout()
|
||||
router.push('/login')
|
||||
}
|
||||
</script>
|
||||
|
||||
<style scoped>
|
||||
.layout {
|
||||
height: 100vh;
|
||||
}
|
||||
.aside {
|
||||
background: #001529;
|
||||
}
|
||||
.logo {
|
||||
color: #fff;
|
||||
font-size: 16px;
|
||||
font-weight: 600;
|
||||
text-align: center;
|
||||
padding: 18px 0;
|
||||
}
|
||||
.menu {
|
||||
border-right: none;
|
||||
background: transparent;
|
||||
--el-menu-text-color: rgba(255, 255, 255, 0.65);
|
||||
--el-menu-hover-bg-color: rgba(255, 255, 255, 0.1);
|
||||
--el-menu-active-color: #fff;
|
||||
--el-menu-bg-color: transparent;
|
||||
}
|
||||
.header {
|
||||
display: flex;
|
||||
justify-content: space-between;
|
||||
align-items: center;
|
||||
border-bottom: 1px solid #eee;
|
||||
background: #fff;
|
||||
}
|
||||
.main {
|
||||
background: #f5f7fa;
|
||||
}
|
||||
</style>
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user