重构数据引擎和报表引擎
This commit is contained in:
@@ -27,3 +27,27 @@
|
||||
- prefetch 流程补充游标分页、参数构建、响应解析的详细说明
|
||||
- 新增「并发保护」章节，记录内存锁和调度器时序
|
||||
- 调度器章节补充 for+sleep 模式说明
|
||||
|
||||
## 钉钉平台新增
|
||||
- 创建 `seed_data_dingtalk.sql`：钉钉开放平台 + 部门列表接口
|
||||
- auth_type: OAUTH2，access_token 在 URL 查询参数中
|
||||
- 部门列表：POST + parameters_location=query + recursive 递归遍历，响应 `result` 为数组
|
||||
- 表名：`dingtalk_department`，冲突键：dept_id
|
||||
|
||||
## 递归遍历支持
|
||||
- 新增 `RecursiveConfig` + `parseRecursiveConfig` + `syncRecursive` 函数
|
||||
- `syncRecursive`：BFS 队列遍历，先查根级 → 对每个子部门递归查下级，防重复 + 最大深度限制
|
||||
- `buildReqBody` / `buildPrefetchParams` 增加过滤 `recursive` / `max_recursive_depth`
|
||||
|
||||
## 钉钉智能人事平台
|
||||
- 新建 `seed_data_dingtalk_hrm.sql`：钉钉智能人事专用平台
|
||||
- auth_type: OAUTH2 + Header 认证（`x-acs-dingtalk-access-token`），独立于 oapi 平台
|
||||
- 企业职位列表 `position_list`：POST + cursor 分页 + hasMore，API base `api.dingtalk.com`
|
||||
|
||||
## 钉钉角色列表 + 引擎增强
|
||||
- `parseRespExt` 增加 `has_more_field` 支持 + 数字游标(float64→string)
|
||||
- `syncWithPrefetch` 支持递归预取来源（BFS）
|
||||
- `buildReqBody` 增加 `pagination_mode: "offset"` 支持（计算 offset = (page-1)*pageSize）
|
||||
- `syncSingleAPI` 增加 hasMore 驱动的分页循环（钉钉 offset/size + hasMore）
|
||||
- 新增 `hasMoreCheck` 辅助函数
|
||||
- 新增 `seed_data_dingtalk.sql` 角色列表接口：偏移分页 + hasMore
|
||||
|
||||
@@ -0,0 +1,36 @@
|
||||
# 2026-06-03 工作记录
|
||||
|
||||
## 钉钉智能薪酬平台对接
|
||||
- 新增 `dingtalk_salary` 平台 + 2个接口到 `sql/seed_data_dingtalk_salary.sql`
|
||||
- 新认证类型 `APP_SIGNATURE`：app-id + signature 头部认证
|
||||
- 签名算法：MD5(body_string + app_secret).toUpperCase()
|
||||
|
||||
### 接口明细
|
||||
1. **salary_dept_list** — `POST /oapi/salary/dept/list`，无入参，单次拉取全量部门
|
||||
2. **salary_statistics** — `POST /oapi/salary/statistics/dept`，prefetch 自 salary_dept_list，row_inject statisticsMonth
|
||||
|
||||
### 代码改动
|
||||
- `api_client.go`: `applyAuthHeader` 扩展支持 `APP_SIGNATURE` 认证，新增 `applyAppSignatureAuth` + `computeBodySignature`
|
||||
- `platform_manager.go`: 新增 `APP_SIGNATURE` case，从 auth_config 提取 app_id/app_secret
|
||||
- `dynamic_sync.go`:
|
||||
- `toFloat64` 扩展支持 string→float64（适配 salary API 返回 code: "200" 字符串）
|
||||
- `buildReqBody`/`buildPrefetchParams` skip list 新增 `row_inject`
|
||||
- 新增 `injectRowFields` 函数：将请求参数注入响应行
|
||||
- `injectRowFields` 调用于 syncWithPrefetch 中3处 row 注入点
|
||||
|
||||
## 新增钉钉智能薪酬人力成本报表接口 (2026-06-03 第二阶段)
|
||||
|
||||
### 新增接口
|
||||
3. **salary_statistic_report** — `POST /oapi/salary/statistic/report/data`，无分页，single_record，获取指定月份的人力成本报表
|
||||
- 入参：`calBizId`（格式 yyyyMM"M"，如 "202606M"，为空默认当月）
|
||||
- 出参：`data.calBizId` + `data.sumStatisticsData`（成本项数组 JSONB）
|
||||
- 表：`dingtalk_salary_statistic_report`，conflict_key: `calBizId`
|
||||
|
||||
4. **salary_statistic_report_group** — `POST /oapi/salary/statistic/report/groupData`，无分页，single_record
|
||||
- 入参：`calBizId` + `salaryGroupName`（必填），使用 `row_inject` 注入 salaryGroupName
|
||||
- 表：`dingtalk_salary_statistic_report_group`，conflict_keys: `calBizId` + `salaryGroupName`
|
||||
|
||||
### 代码改动
|
||||
- `dynamic_sync.go`：修复主同步路径（非 prefetch）中 `injectRowFields` 未调用的问题
|
||||
- 在 5 处 savePage 调用前添加 injectRowFields：初始页、游标分页循环、hasMore 分页、普通分页（有/无 response_config）
|
||||
- 使 `row_inject` 在单记录（single_record）非 prefetch 接口中也能生效
|
||||
@@ -0,0 +1,15 @@
|
||||
# 2026-06-08 工作日志
|
||||
|
||||
## goview-report 项目初始化
|
||||
- 基于 GoView (gitee.com/bufanyun/go-view-server) 初始化报表引擎项目
|
||||
- 位置: `/Users/xujiaqian/GolandProjects/HDWL/goview-report/`
|
||||
- 改造内容:
|
||||
- 模块名 hotgo → goview-report
|
||||
- MySQL 驱动替换为 PostgreSQL 驱动 (github.com/gogf/gf/contrib/drivers/pgsql/v2)
|
||||
- Go 版本升级到 1.26.0，GoFrame 升级到 v2.10.0
|
||||
- config.yaml 中数据库连接改为 pgsql://postgres:root@127.0.0.1:5432/engine
|
||||
- 端口保持 8090，与 data-engine(:3002) 不冲突
|
||||
- 元数据表前缀 hg_，data-engine 同步表无前缀，同库不同前缀
|
||||
- 创建 Dockerfile + k8s/deployment+service+configmap
|
||||
- 编译验证通过
|
||||
- 与 data-engine 关系: 读写分离，goview-report 只读 PG 中的同步表做报表展示
|
||||
@@ -0,0 +1,18 @@
|
||||
# 2026-06-09 工作日志
|
||||
|
||||
## 公共查询接口开发
|
||||
- 新增 `/public/query` POST 接口，支持：
|
||||
- 字段白名单验证（只允许表定义中声明的字段）
|
||||
- 表名白名单验证（只允许 api_interface 中有 table_definition 的表）
|
||||
- WHERE 条件（支持 `_eq/_ne/_gt/_lt/_ge/_le/_like/_in/_between` 操作符）
|
||||
- GROUP BY / ORDER BY / 分页
|
||||
- 强制 tenant_id = 1 租户隔离
|
||||
- 新增辅助接口：
|
||||
- `GET /public/tables` - 获取可查询表列表
|
||||
- `GET /public/tables/{table}/columns` - 获取表字段列表
|
||||
- `DELETE /public/cache/clear` - 清除表缓存
|
||||
- 文件位置：
|
||||
- `model/dto/public/public_query_dto.go`
|
||||
- `service/public/public_query_service.go`
|
||||
- `controller/public/public_query_controller.go`
|
||||
- 同步修复 `dynamic_sync.go` 中残留的无效代码 "了呢"
|
||||
@@ -0,0 +1,179 @@
|
||||
# 2026-06-10
|
||||
|
||||
## 通用报表引擎 - 补充前端 CRUD 能力
|
||||
|
||||
为 `common/report` 包补充了配置的 CRUD 接口，使前端可以直接维护报表配置，无需操作数据库。
|
||||
|
||||
### 新增文件变更
|
||||
- `model/model.go`: 新增 SaveBusinessReq/SaveReportReq/SaveFieldReq/SaveExtractConfigReq 等请求结构体，SaveResult/DeleteResult 响应结构体
|
||||
- `config/loader.go`: 新增 Create/Update/Delete + GetByID 方法（Business/Report/Field/ExtractConfig 四种配置）
|
||||
- `api.go`: 新增 SaveBusiness/SaveReport/SaveField/SaveExtractConfig + Delete/Get 接口（新增/修改合一，id 为空则新增）
|
||||
- `example_usage.go`: 新增第五、六节，CRUD 调用示例 + 任意新平台零代码接入流程
|
||||
|
||||
### 关键设计决策
|
||||
- **Save 合一**：通过 `ID *int64` 区分新增（nil）和修改（有值），简化前端调用
|
||||
- **软删除**：status → INACTIVE + deleted_at=NOW()，数据不丢失
|
||||
- **缓存失效**：写操作后自动失效对应业务/报表缓存，保证下次读取一致性
|
||||
- 所有 CRUD 方法先调用 initTables() 确保表存在
|
||||
|
||||
### example_usage.go 完整重写
|
||||
将示例文件从分节的注释风格重写为6个完整场景的 Go 函数示例：
|
||||
1. **快手电商完整接入**（最详细，从业务注册到前端查询6步全流程）
|
||||
2. **已有配置的 CRUD 二次开发参考**（业务/报表/字段/抽取 增删改查）
|
||||
3. **任意平台接入通用模式**（淘宝示例，零硬编码）
|
||||
4. **外部业务服务调用**（goview-report 等直接 import）
|
||||
5. **Direct 模式示例**（不聚合的逐行抽取）
|
||||
6. **前端 HTTP API 交互流程**（伪代码，展示 API 调用顺序）
|
||||
|
||||
每个场景都有实际可运行的代码片段、生成的 SQL 预览、返回值输出示例。
|
||||
|
||||
## 报表引擎管理前端页面
|
||||
|
||||
为报表引擎创建了可直接使用的前端管理页面，访问 `/admin/report`。
|
||||
|
||||
### 新增文件
|
||||
- `controller/report/report_controller.go`: HTTP API Controller，16个端点暴露公共报表服务的全部能力（业务/报表/字段/抽取配置的 CRUD、数据抽取触发、自动建表、用户选择查询）
|
||||
- `controller/report/report_admin_controller.go`: 嵌入式HTML SPA，5个Tab页：
|
||||
- **业务管理**：列表 CRUD
|
||||
- **报表配置**：按业务筛选列表 CRUD
|
||||
- **字段配置**：按业务+报表筛选，维度/指标/筛选角色管理
|
||||
- **抽取配置**：按业务+报表筛选，支持 DIRECT/AGGREGATE 模式配置 + 一键执行抽取
|
||||
- **数据查询**：可视化维度选择器、指标聚合选择器、筛选条件构建器、查询结果展示
|
||||
- `main.go`: 注册 reportCtrl.ReportController 到 RouteRegister + `/admin/report` 页面路由
|
||||
|
||||
### 技术要点
|
||||
- 纯原生 HTML/CSS/JS，零框架依赖，与现有管理后台 `/admin` 风格统一
|
||||
- 所有 CRUD 通过 `ID *int64` 区分新增（nil）/修改（有值）
|
||||
- 查询Tab支持动态字段选择：维度多选、指标聚合方式选择、筛选条件动态构建（支持 BETWEEN/IN 等操作符）
|
||||
- 管理后台页面增加了返回链接 `← 返回管理后台`
|
||||
|
||||
## 修复 report_common_ddl.sql 语法错误
|
||||
|
||||
PostgreSQL 不支持 MySQL 风格的内联 `COMMENT '...'` 语法。修复了全部5张表的列定义：
|
||||
- 移除所有列定义中的内联 `COMMENT 'xxx'`（MySQL 语法）
|
||||
- 已有的 `COMMENT ON COLUMN` 独立语句保留不变
|
||||
- `report_extract_log` 表补充了缺失的 `COMMENT ON COLUMN` 语句（business_code/report_code/extract_code/stat_date/extract_type/status/total_count/success_count/fail_count）
|
||||
|
||||
关键区别：MySQL 支持 `column_name VARCHAR(64) COMMENT 'xxx'`，PostgreSQL 必须用 `COMMENT ON COLUMN table.column IS 'xxx'`。
|
||||
|
||||
## 补充 tenant_id 到报表系统表
|
||||
|
||||
5张报表系统表原先缺失 `tenant_id`，与项目核心表（`init_core_tables.sql`）不一致。修改内容：
|
||||
- **DDL SQL** (`sql/report_common_ddl.sql`)：5张表全部新增 `tenant_id BIGINT NOT NULL DEFAULT 0` 列，所有 UNIQUE 约束和索引都加入 tenant_id
|
||||
- **Go initTables** (`common/report/report.go`)：同步更新 5 张表的 DDL 定义，加上 tenant_id
|
||||
- 约束变更：`uk_business_report_code UNIQUE (business_code, report_code)` → `(tenant_id, business_code, report_code)`，其余类推
|
||||
|
||||
当前 report 模块租户支持为占位符级别（硬编码 tenant_id=1），后续需从 context 动态获取。
|
||||
- **单独迁移脚本**: `sql/report_add_tenant_id.sql`，对已存在的表执行 ALTER TABLE ADD COLUMN + 重建约束/索引
|
||||
|
||||
## 快手电商报表初始化种子数据
|
||||
|
||||
**初版（已废弃）**：字段名随意捏造，与真实 DDL 不对齐。
|
||||
|
||||
**重写版** `sql/seed_data_report_kuaishou.sql`（基于 seed_data_kuaishou.sql 实际 DDL）：
|
||||
- 1 条业务配置 (kuaishou_ecommerce)
|
||||
- 5 条报表配置：order_analysis / item_analysis / refund_analysis / cps_order_analysis / dropshipping_analysis
|
||||
- 76 条字段配置（订单23+商品15+售后12+分销13+代发13），含维度/指标/衍生字段
|
||||
- 5 条抽取配置（AGGREGATE 聚合模式，每个报表一条）
|
||||
|
||||
**严格设计原则**：
|
||||
- sourceField 对齐源表实际列名（camelCase：oid, status, createTime, totalFee, buyerNick 等）
|
||||
- targetField / field_code 使用 snake_case（stat_date, total_fee, buyer_nick 等）
|
||||
- 金额字段源表存分(BIGINT)，报表提供衍生元字段(CALCULATED expression / 100.0)
|
||||
- 增量抽取依据统一用 updateTime（毫秒时间戳 BIGINT）
|
||||
- transform_rules：ts_to_date FORMAT createTime → stat_date(yyyy-MM-dd)
|
||||
- 所有 INSERT 使用 ON CONFLICT DO NOTHING 保证幂等
|
||||
|
||||
## 修复报表管理前端编辑不显示数据问题
|
||||
|
||||
在 `report_admin_controller.go` 中修复了编辑弹窗不显示数据的问题：
|
||||
|
||||
**根因**：
|
||||
- GoFrame `MiddlewareHandlerResponse` 包裹响应为 `{code:0, data: <struct>}`，而单条查询返回结构体为 `getBusinessRes{Data: ...}` 序列化为 `{"data": {...}}`，最终响应为 `{code:0, data: {"data": {...}}}` 双层嵌套
|
||||
- 某些 GoFrame 版本可能序列化 Go 字段名（首字母大写 `Data`）而非 json tag（`data`），导致 `res.data` 为 undefined
|
||||
|
||||
**修复内容**：
|
||||
1. `api()` 函数增加兼容逻辑：当 `json.data.Data` 存在而 `json.data.data` 不存在时，自动取 `json.data.Data`
|
||||
2. `api()` 函数增加 `console.log` 调试日志，打印原始响应和解包后的数据
|
||||
3. 四个编辑函数（`openBizModal/openReportModal/openFieldModal/openExtractModal`）：
|
||||
- 增加 fallback：`res.data !== undefined ? res.data : res.Data`
|
||||
- 增加数据有效性校验：`if (!d || !d.id)` 时弹出错误提示并阻止打开空表单
|
||||
- 增加 `console.log` 调试日志
|
||||
|
||||
## 修复 report_report_config 查询 SQL 永假条件问题
|
||||
|
||||
**现象**：查询 `report_report_config` 时生成的 SQL 包含 `WHERE (("id" = 1) AND ("id" = 0 AND "business_code" = '' AND ...))`，id 不可能同时为 1 和 0，导致查询始终返回 0 行。
|
||||
|
||||
**根因**：`common/db/gfdb/gfdb.go` 中 `Model()` 方法使用了 `OmitNil()`（只过滤 nil 指针，不过滤零值）。当 GoFrame v2 内部的软删除机制（`deleted_at IS NULL`）自动构建 struct WHERE 条件时，由于 `OmitNil()` 不忽略零值字段，结构体中所有字段的零值（`id=0`, `business_code=""`, `report_code=""` 等）都被加入 WHERE 条件，与显式的 `Where("id", 1)` 叠加形成 `id=1 AND id=0` 的永假条件。
|
||||
|
||||
**修复**：`OmitNil()` **保留**，额外追加 `.OmitEmptyWhere()`，两者同时生效：
|
||||
- `OmitNil()`：保持 INSERT/UPDATE 的 Data() 行为不变（nil 指针/map 字段不写入 SQL，走 DB 默认值），避免破坏数据引擎写入逻辑
|
||||
- `OmitEmptyWhere()`：过滤 struct WHERE 条件中的零值字段，解决软删除 hook 注入零值条件的问题
|
||||
|
||||
最终代码：`OmitNil().OmitEmptyWhere()`
|
||||
|
||||
**影响范围**：`common/db/gfdb/gfdb.go` - 全局 DB Model 初始化。写入行为完全兼容，查询条件不再包含零值 struct 字段。
|
||||
|
||||
**安全性审计（2026-06-10）**：逐文件审查了数据引擎核心同步路径全部 40+ 处 `gfdb.Model()` 调用：
|
||||
- 100% 传字符串表名（非 struct），不会触发 struct WHERE 推断
|
||||
- 100% 使用 `Where("col", val)` 字符串键值对（非 struct/map），`OmitEmptyWhere()` 对此无效
|
||||
- `OmitEmptyWhere()` 不影响 `Data()`/`Insert()`/`Update()`/`Save()`
|
||||
- 结论：`OmitEmptyWhere()` 对数据引擎核心同步逻辑完全无影响，唯一生效场景是 `common/report` 软删除 hook
|
||||
|
||||
## 修复 GetXxxByID 方法零值结构体返回问题
|
||||
|
||||
**现象**：前端编辑报表/业务/字段/抽取配置时一直提示"未获取到XX数据（ID=N）"。与上述 OmitNil 问题同根：SQL 返回 0 行时，GoFrame 的 `One()` 可能不返回 `sql.ErrNoRows`（驱动差异），此时 `GetXxxByID` 判断 `err == nil` 就返回了零值结构体 `{ID: 0, ...}`。Controller 包装为 `{code:0, data:{"data":{id:0}}}`，前端 `!d.id` 判定失败弹出错误提示。
|
||||
|
||||
**修复**：`common/report/config/loader.go` 中全部 7 个 `One()` 调用方法，在 `err` 检查之外增加 `result.IsEmpty()` 检查：
|
||||
- `GetBusinessByID` / `GetReportByID` / `GetFieldByID` / `GetExtractConfigByID`：四者模式一致，`_` 改为 `result` 变量并追加 `if result.IsEmpty()`
|
||||
- `GetBusiness` / `GetReport`（by code）、`GetExtractLog`：同上，追加 `IsEmpty()` 守卫
|
||||
|
||||
**双重防御**：gfdb.go 的 `OmitEmptyWhere()` 是根本修复（消除错误 SQL）；loader.go 的 `IsEmpty()` 是兜底守卫（即使 SQL 返回 0 行也不返回零值结构体）。
|
||||
|
||||
## 修复编辑弹窗数据提取的三层兼容
|
||||
|
||||
**根因**：`api()` 返回 `json.data` 给调用者。当 GoFrame `MiddlewareHandlerResponse` 双层包裹时（`{code:0, data: {"data": {...}}}`），`api()` 返回 `{"data": {...}}`，调用者通过 `res.data` 正确提取。但当 GoFrame 单层包裹时（`{code:0, data: {...}}`），`api()` 直接返回 `{...config...}`，调用者 `res.data` 为 undefined → `d` 为 undefined → 触发 "未获取到XX数据" 错误。
|
||||
|
||||
**修复**：4 个编辑函数（`openBizModal`/`openReportModal`/`openFieldModal`/`openExtractModal`）的数据提取改为三层 fallback：
|
||||
```javascript
|
||||
const d = res && res.data !== undefined ? res.data : (res && res.Data !== undefined ? res.Data : res);
|
||||
```
|
||||
- 第一层：`res.data`（双层包裹，json tag 小写）
|
||||
- 第二层：`res.Data`（Go 字段名大写 D）
|
||||
- 第三层：`res` 本身（单层包裹，数据直接在 res 里）
|
||||
|
||||
同时 console.log 增加打印 `raw:` 和 `data:` 便于排查。
|
||||
|
||||
## 修复报表引擎实体缺失 TenantId 字段 → 查询 tenant_id 不匹配返回零值
|
||||
|
||||
**现象**：`GET /report/business?id=1` 返回 `code: 0` 但 data 为全零值 `{id: 0, businessCode: "", ...}`，尽管 DB 中实际有数据。
|
||||
|
||||
**根因**：四个报表引擎实体结构体均缺失 `TenantId` 字段，但 DDL 中表定义有 `tenant_id BIGINT NOT NULL DEFAULT 0`：
|
||||
1. `CreateBusiness` 序列化 `BusinessConfig` 时，JSON 中无 `tenant_id` → insert map 中无 `tenant_id` key
|
||||
2. `insertHook` 检查 `in.Data[i]["tenant_id"]` 是否存在 → 不存在 → 不设置 → DB 用 DEFAULT 0
|
||||
3. `selectHook` 从 `ctxWithUser` 获取 `TenantId=1`，注入 `WHERE tenant_id = 1`
|
||||
4. DB 行 `tenant_id=0` 与查询 `tenant_id=1` 不匹配 → 查不到行 → 返回零值结构体
|
||||
|
||||
**修复**（`common/report/model/model.go`）：给 `BusinessConfig`、`ReportConfig`、`FieldConfig`、`ExtractConfig` 四个结构体新增 `TenantId uint64 \`orm:"tenant_id" json:"tenant_id"\``。
|
||||
|
||||
**原理**：添加后序列化 JSON 包含 `"tenant_id": 0`，`insertHook` 检测到 key 存在且为零值，自动赋值为 `userInfo.TenantId`。
|
||||
|
||||
**调试日志**（`common/report/config/loader.go`）：`GetBusinessByID` 增加 `g.Log().Infof` 打印 `id`、`err`、`IsEmpty`、`biz.ID`、`biz.BusinessCode`，便于排查后续类似问题。同时补充 import `"github.com/gogf/gf/v2/frame/g"`。
|
||||
|
||||
## 修复 GoFrame One(&struct) 直接映射失效 → 改用两步模式
|
||||
|
||||
**现象**：修复 TenantId 后，SQL 能正确执行并返回1行（`[rows:1]`, `IsEmpty=false`），但 struct 字段全为零值（`biz.ID=0, biz.BusinessCode=""`）。
|
||||
|
||||
**根因**：`loader.go` 中 7 处 `One(&struct)` 直接传结构体指针，在 GoFrame v2.10.0 + `selectHook` 修改 SQL 的组合下，struct 自动映射失效 —— 查询返回 `gdb.Record` 但未 populate 结构体。
|
||||
|
||||
**修复**：全部 7 处改为项目已验证的两步模式（参照 `dao/dict/api_interface_dao.go`）：
|
||||
```go
|
||||
// 旧写法（struct 映射失效）
|
||||
result, err := gfdb.DB(ctx).Model(ctx, "table").Where(...).One(&biz)
|
||||
|
||||
// 新写法（先取 Record，再手动 Struct）
|
||||
r, err := gfdb.DB(ctx).Model(ctx, "table").Where(...).One()
|
||||
if err = r.Struct(&biz); err != nil { return nil, err }
|
||||
```
|
||||
|
||||
**影响的方法**：`GetBusiness`、`GetBusinessByID`、`GetReport`、`GetReportByID`、`GetFieldByID`、`GetExtractConfigByID`、`GetExtractLog`。
|
||||
@@ -73,11 +73,15 @@ auth_config: { "app_key": "...", "app_secret": "..." }
|
||||
| `cursor_pagination` | bool | 是否游标分页（true=游标，false/无=普通分页） | 通用 |
|
||||
| `time_field` | string | 增量时间字段名 | 通用 |
|
||||
| `time_field_mode` | string | `"range"`=beginTime/endTime模式（快手），`"filtering"`=filtering数组模式（腾讯，默认） | 通用 |
|
||||
| `full_sync_start_time` | int | 全量同步时的时间起点，毫秒时间戳。`lastSyncTime=0` 时优先读取此值，不存在则回退90天前（range）或不传时间过滤（filtering） | 通用 |
|
||||
| `prefetch` | object | 预取配置（见下方） | 需遍历实体的接口 |
|
||||
| `recursive` | object | 递归遍历配置（见下方） | 树形结构接口（如钉钉部门） |
|
||||
| `max_recursive_depth` | int | 递归最大深度，默认20 | 递归遍历使用 |
|
||||
| `body_wrapper_field` | string | 业务参数包装字段名，如`"param"` | 快手 |
|
||||
| `exclude_from_wrapper` | string[] | body_wrapper 时不包装的字段 | 快手 |
|
||||
| `top_level_params` | string[] | 保留在顶层的字段（备用） | 通用 |
|
||||
| `fields` | string[] | API 的 fields 参数 | 腾讯 |
|
||||
| `row_inject` | string[] | 将请求参数值注入到每行数据中（如 salaryGroupName），解决响应不含某请求参数但需存储的场景 | 通用（prefetch 和非 prefetch 均支持） |
|
||||
|
||||
### prefetch 预取配置
|
||||
```json
|
||||
@@ -198,6 +202,15 @@ WARN 接口 [kuaishou/order_list] 正在同步中，跳过重复请求
|
||||
|
||||
**调度器时序**：使用 `for { runAutoSync(); time.Sleep(interval) }` 模式，而非 `time.NewTicker`。每次同步**完全结束后**才开始计时 interval，避免前一次未跑完就启动下一次导致重叠。无论全量跑 30 分钟还是 70 分钟，下一次都在「本次完成时间 + interval」后执行。
|
||||
|
||||
## 三种遍历模式对比
|
||||
|
||||
| 模式 | 配置 | 适用场景 | 流程 |
|
||||
|------|------|---------|------|
|
||||
| **普通分页** | `page_param` + `page_size_param` | 列表接口分页 | 请求第1页 → 读 `page_info.total_page` → 遍历后续页 |
|
||||
| **游标分页** | `cursor_pagination: true` | 实时滚动列表（快手订单） | 首次 `cursor=""` → 响应返回 `cursor` → 直到 `""` 或 `"nomore"` |
|
||||
| **prefetch 预取** | `prefetch: {url, ...}` | 先取实体列表，再查详情 | 分页拉取实体来源列表 → 提取全部 ID → 并发查详情 |
|
||||
| **recursive 递归** | `recursive: {key_field, target_param}` | 树形结构（钉钉部门） | BFS 队列：根级调用 → 提取子ID → 逐级递归下级 → 防重复 |
|
||||
|
||||
## 新增平台 + 接口的操作步骤
|
||||
|
||||
### 操作步骤
|
||||
@@ -461,6 +474,69 @@ INSERT INTO api_interface (
|
||||
```
|
||||
注意：POST 方法且无 `parameters_location: "query"` 时，参数以 JSON body 形式发送。
|
||||
|
||||
#### 模板 E：OAuth2 + POST + 无分页（钉钉风格）
|
||||
```sql
|
||||
INSERT INTO api_datasource_platform (
|
||||
tenant_id, creator, created_at, updater, updated_at,
|
||||
platform_code, platform_name, description, status,
|
||||
api_base_url, auth_type,
|
||||
auth_config,
|
||||
rate_limit_per_minute, rate_limit_per_hour,
|
||||
concurrency_limit, request_timeout_ms, max_retries, retry_delay_ms
|
||||
) VALUES (
|
||||
1, 'admin', NOW(), 'admin', NOW(),
|
||||
'dingtalk', '钉钉', '钉钉开放平台数据同步', 'ACTIVE',
|
||||
'https://oapi.dingtalk.com', 'OAUTH2',
|
||||
'{
|
||||
"token_in_query": true,
|
||||
"query_key": "access_token"
|
||||
}'::jsonb,
|
||||
60, 3600, 5, 30000, 3, 1000
|
||||
);
|
||||
|
||||
INSERT INTO api_interface (
|
||||
tenant_id, creator, created_at, updater, updated_at,
|
||||
platform_id, name, code, url, method, status, auth_type,
|
||||
request_config, response_config, table_definition
|
||||
) VALUES (
|
||||
1, 'admin', NOW(), 'admin', NOW(),
|
||||
(SELECT id FROM api_datasource_platform WHERE platform_code = 'dingtalk'),
|
||||
'{接口名称}', '{接口编码}',
|
||||
'{相对路径}', 'POST', 'active', 'inherit',
|
||||
'{
|
||||
"parameters_location": "query",
|
||||
"page_param": "cursor",
|
||||
"page_size_param": "pageSize",
|
||||
"{业务参数}": {值}
|
||||
}'::jsonb,
|
||||
'{
|
||||
"success_field": "errcode",
|
||||
"success_value": 0,
|
||||
"message_field": "errmsg",
|
||||
"list_path": "result"
|
||||
}'::jsonb,
|
||||
'{
|
||||
"table_name": "{表名}",
|
||||
"columns": [
|
||||
{"name": "id", "type": "BIGINT", "comment": "ID"},
|
||||
{"name": "name", "type": "VARCHAR(300)", "comment": "名称"}
|
||||
],
|
||||
"conflict_keys": ["id"]
|
||||
}'::jsonb
|
||||
);
|
||||
```
|
||||
注意：钉钉使用 `access_token` 作为 URL 查询参数，通过 `token_in_query: true` + `query_key: "access_token"` 配置。POST 请求通过 `parameters_location: "query"` 将所有参数放在 URL 查询字符串中（DingTalk 同时支持 query 和 body 传参）。
|
||||
|
||||
如需递归遍历树形结构（如部门全量），在 `request_config` 中添加：
|
||||
```json
|
||||
"recursive": {
|
||||
"key_field": "dept_id", // 从响应行中提取哪个字段的值用于递归
|
||||
"target_param": "dept_id" // 递归参数注入到请求中的参数名
|
||||
},
|
||||
"max_recursive_depth": 20 // 可选，最大递归深度，默认20
|
||||
```
|
||||
递归流程：BFS 遍历，先查根级空参 → 提取每行的 `key_field` → 逐级注入 `target_param` 查询下级 → 防重复循环保护 → 所有结果合并入库。
|
||||
|
||||
## 新增接口步骤总结
|
||||
|
||||
1. **确定平台**：已有平台直接跳到第3步，否则先新增平台（参考对应认证类型的模板）
|
||||
@@ -474,6 +550,67 @@ INSERT INTO api_interface (
|
||||
- 是否需要遍历实体（prefetch）
|
||||
- 增量时间字段（字段名 + 值类型）
|
||||
4. **选择模板**并填入对应值
|
||||
|
||||
> ⚠️ **全量同步起始时间**：如果接口数据量很大，首次全量不想从 90 天前拉取，可以在 `request_config` 中加 `"full_sync_start_time": <毫秒时间戳>` 指定起点。例如只同步最近 30 天：
|
||||
> ```json
|
||||
> "full_sync_start_time": 1745942400000
|
||||
> ```
|
||||
> - **range 模式**（快手 `time_field_mode: "range"`）：全量时取此时间作为 beginTime
|
||||
> - **filtering 模式**（腾讯 `time_field_mode: "filtering"`）：全量时生成 `GREATER_EQUALS` 过滤条件
|
||||
> - **不配此字段**：行为不变（range 回退 90 天，filtering 不传时间过滤）
|
||||
|
||||
5. **执行 seed SQL**
|
||||
6. **在管理后台验证**：`http://{host}:3002/admin`
|
||||
|
||||
## 代码审计发现的项目规范
|
||||
|
||||
### 包命名规范（2026-06-03 审计）
|
||||
- `model/dto/dict/` 下的 `.go` 文件必须用 `package dict`（曾错误写为 `package api_feature`，已修复）
|
||||
- `model/entity/dict/` 下的 `.go` 文件用 `package dict`
|
||||
- `consts/api-feature/` 目录下的文件用 `package api_feature`
|
||||
- `consts/public/` 用 `package public`
|
||||
- **不要**在 `consts/dict/` 中重复定义 `PlatformStatus`/`ApiMethod` 等类型，应使用 `consts/api-feature` 中的定义
|
||||
|
||||
### 数据库状态值规范
|
||||
- `api_datasource_platform.status`：使用大写 `"ACTIVE"` / `"INACTIVE"`
|
||||
- `api_interface.status`：使用小写 `"active"` / `"inactive"`
|
||||
- 代码中查平台状态时不要使用 `api_feature.PlatformStatusActive`（其值为小写），应直接用 `"ACTIVE"` 字符串
|
||||
|
||||
### Known Bad Practices（已知待改进项）
|
||||
- 敏感信息（access_token / app_secret）仍分散在 SQL seed文件和 config.yml 中，建议迁移到环境变量或 Vault
|
||||
- `consts/dict/consts.go` 和 `scheduler/run_sync_task_log_task.go` 可能随业务扩展需要更新
|
||||
|
||||
## 通用报表引擎 (common/report)
|
||||
|
||||
2026-06-10: 通用报表公共包，配置驱动的报表子系统，支持前端自由选择维度/指标/筛选/时间查询。
|
||||
|
||||
### 架构分层 (9个Go文件)
|
||||
```
|
||||
common/report/
|
||||
├── api.go # ReportService 统一门面（单例）
|
||||
├── report.go # 5张系统表 DDL
|
||||
├── example_usage.go # 完整使用示例文档
|
||||
├── model/model.go # 实体/请求/响应/常量
|
||||
├── config/loader.go # 配置加载器（读缓存+CRUD）
|
||||
├── builder/sql_builder.go # 动态 SQL 构建
|
||||
├── executor/executor.go # 查询执行器
|
||||
├── extract/extract.go # 天级数据抽取（DIRECT/AGGREGATE）
|
||||
└── ddlsync/creator.go # 统计宽表自动创建
|
||||
```
|
||||
|
||||
### 核心能力
|
||||
1. **自动建表**：根据 FieldConfig 自动 CREATE TABLE stat_xxx
|
||||
2. **数据抽取**：DIRECT（逐行）/ AGGREGATE（聚合 SUM/COUNT）模式，幂等保证
|
||||
3. **动态查询**：前端选维度+指标+筛选+时间 → 实时构建 SQL → 分页返回
|
||||
4. **配置 CRUD**（2026-06-10 新增）：BusinessConfig/ReportConfig/FieldConfig/ExtractConfig 全部可前端维护
|
||||
|
||||
### 对外接口 (ReportService)
|
||||
- 查询：`QueryReportByUserSelect` / `GetReportFields` / `GetAllBusinesses` / `GetAllReports`
|
||||
- 抽取：`ExtractDailyData` / `AutoCreateStatTable`
|
||||
- CRUD：`SaveBusiness/DeleteBusiness/GetBusiness`、`SaveReport/DeleteReport/GetReport`、`SaveField/DeleteField/GetField`、`SaveExtractConfig/DeleteExtractConfig/GetExtractConfig/GetExtractConfigs`
|
||||
|
||||
### 设计原则
|
||||
- 零硬编码：任意平台/接口通过配置接入，不改代码
|
||||
- 配置驱动：5张 PG 表存储全部配置，CRUD API 支持前端维护
|
||||
- 单例模式：`report.GetService()` 可在任意业务服务中直接调用
|
||||
|
||||
|
||||
@@ -0,0 +1,477 @@
|
||||
package report
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"strings"
|
||||
|
||||
"dataengine/common/report/config"
|
||||
"dataengine/common/report/ddlsync"
|
||||
"dataengine/common/report/executor"
|
||||
"dataengine/common/report/extract"
|
||||
"dataengine/common/report/model"
|
||||
|
||||
"gitea.redpowerfuture.com/red-future/common/db/gfdb"
|
||||
)
|
||||
|
||||
// ReportService 报表公共服务
|
||||
// 对外暴露的统一接口
|
||||
type ReportService struct {
|
||||
configLoader *config.ConfigLoader
|
||||
tableCreator *ddlsync.StatTableCreator
|
||||
queryExecutor *executor.QueryExecutor
|
||||
dailyExtractor *extract.DailyExtractor
|
||||
}
|
||||
|
||||
var defaultService *ReportService
|
||||
|
||||
// GetService 获取报表服务单例
|
||||
func GetService() *ReportService {
|
||||
if defaultService == nil {
|
||||
defaultService = &ReportService{
|
||||
configLoader: config.GetLoader(),
|
||||
tableCreator: ddlsync.NewStatTableCreator(),
|
||||
queryExecutor: executor.NewQueryExecutor(),
|
||||
dailyExtractor: extract.NewDailyExtractor(),
|
||||
}
|
||||
}
|
||||
return defaultService
|
||||
}
|
||||
|
||||
// ============================================================
|
||||
// 核心接口 1: 自动创建统计宽表
|
||||
// 首次抽取前调用
|
||||
// ============================================================
|
||||
|
||||
// AutoCreateStatTable 根据配置自动创建统计宽表
|
||||
// businessCode: 业务编码
|
||||
// reportCode: 报表编码
|
||||
func (s *ReportService) AutoCreateStatTable(ctx context.Context, businessCode, reportCode string) (*model.AutoCreateStatTableResp, error) {
|
||||
// 初始化系统表
|
||||
if err := initTables(ctx); err != nil {
|
||||
return nil, fmt.Errorf("初始化系统表失败: %w", err)
|
||||
}
|
||||
|
||||
resp, err := s.tableCreator.AutoCreateStatTable(ctx, businessCode, reportCode)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("AutoCreateStatTable 失败: %w", err)
|
||||
}
|
||||
|
||||
return resp, nil
|
||||
}
|
||||
|
||||
// ============================================================
|
||||
// 核心接口 2: 按天抽取数据
|
||||
// 业务层定时任务调用
|
||||
// ============================================================
|
||||
|
||||
// ExtractDailyData 按天抽取数据
|
||||
// businessCode: 业务编码
|
||||
// reportCode: 报表编码
|
||||
// statDate: 统计日期 yyyy-MM-dd
|
||||
// executor: 执行人
|
||||
func (s *ReportService) ExtractDailyData(ctx context.Context, businessCode, reportCode, statDate, executor string) (*model.ExtractDailyDataResp, error) {
|
||||
// 1. 先确保统计宽表存在
|
||||
report, err := s.configLoader.GetReport(ctx, businessCode, reportCode)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("获取报表配置失败: %w", err)
|
||||
}
|
||||
|
||||
// 检查表是否存在
|
||||
result, err := gfdb.DB(ctx).GetAll(ctx,
|
||||
"SELECT EXISTS (SELECT 1 FROM pg_tables WHERE tablename = $1) AS exists",
|
||||
strings.ToLower(report.StatTableName))
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("检查统计宽表失败: %w", err)
|
||||
}
|
||||
tableExists := false
|
||||
if len(result) > 0 {
|
||||
tableExists = result[0]["exists"].Bool()
|
||||
}
|
||||
if !tableExists {
|
||||
// 表不存在，先创建
|
||||
if _, createErr := s.AutoCreateStatTable(ctx, businessCode, reportCode); createErr != nil {
|
||||
return nil, fmt.Errorf("创建统计宽表失败: %w", createErr)
|
||||
}
|
||||
}
|
||||
|
||||
resp, err := s.dailyExtractor.ExtractDailyData(ctx, businessCode, reportCode, statDate, executor)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("ExtractDailyData 失败: %w", err)
|
||||
}
|
||||
|
||||
// 清除缓存
|
||||
s.configLoader.InvalidateCache(businessCode, reportCode)
|
||||
|
||||
return resp, nil
|
||||
}
|
||||
|
||||
// ============================================================
|
||||
// 核心接口 3: 用户选择查询（最核心）
|
||||
// 前端用户选择条件 → 实时构建SQL → 返回报表数据
|
||||
// ============================================================
|
||||
|
||||
// QueryReportByUserSelect 根据用户选择实时查询报表数据
|
||||
// 不是自动生成报表，是用户在前端选择维度/指标/筛选/时间后实时查询展示
|
||||
func (s *ReportService) QueryReportByUserSelect(ctx context.Context, req *model.UserSelectQueryReq) (*model.UserSelectQueryResp, error) {
|
||||
// 参数校验
|
||||
if req.BusinessCode == "" {
|
||||
return nil, fmt.Errorf("businessCode 不能为空")
|
||||
}
|
||||
if req.ReportCode == "" {
|
||||
return nil, fmt.Errorf("reportCode 不能为空")
|
||||
}
|
||||
|
||||
resp, err := s.queryExecutor.QueryReportByUserSelect(ctx, req)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("QueryReportByUserSelect 失败: %w", err)
|
||||
}
|
||||
|
||||
return resp, nil
|
||||
}
|
||||
|
||||
// ============================================================
|
||||
// 辅助接口
|
||||
// ============================================================
|
||||
|
||||
// GetReportFields 获取报表可用字段（按维度/指标/筛选分类）
|
||||
func (s *ReportService) GetReportFields(ctx context.Context, businessCode, reportCode string) (*model.GetReportFieldsResp, error) {
|
||||
resp, err := s.configLoader.GetReportFields(ctx, businessCode, reportCode)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("GetReportFields 失败: %w", err)
|
||||
}
|
||||
return resp, nil
|
||||
}
|
||||
|
||||
// GetAllBusinesses 获取所有启用业务列表
|
||||
func (s *ReportService) GetAllBusinesses(ctx context.Context) ([]model.BusinessConfig, error) {
|
||||
return s.configLoader.GetAllBusinesses(ctx)
|
||||
}
|
||||
|
||||
// GetAllReports 获取业务下所有报表列表
|
||||
func (s *ReportService) GetAllReports(ctx context.Context, businessCode string) ([]model.ReportConfig, error) {
|
||||
return s.configLoader.GetAllReports(ctx, businessCode)
|
||||
}
|
||||
|
||||
// InvalidateCache 失效指定业务报表缓存
|
||||
func (s *ReportService) InvalidateCache(businessCode, reportCode string) {
|
||||
s.configLoader.InvalidateCache(businessCode, reportCode)
|
||||
}
|
||||
|
||||
// InitSystemTables 初始化系统表
|
||||
func (s *ReportService) InitSystemTables(ctx context.Context) error {
|
||||
return initTables(ctx)
|
||||
}
|
||||
|
||||
// ============================================================
|
||||
// 配置 CRUD: 业务
|
||||
// ============================================================
|
||||
|
||||
// SaveBusiness 保存业务配置（新增/修改合一）
|
||||
func (s *ReportService) SaveBusiness(ctx context.Context, req *model.SaveBusinessReq) (*model.SaveResult, error) {
|
||||
if err := initTables(ctx); err != nil {
|
||||
return nil, fmt.Errorf("初始化系统表失败: %w", err)
|
||||
}
|
||||
|
||||
biz := &model.BusinessConfig{
|
||||
BusinessCode: req.BusinessCode,
|
||||
BusinessName: req.BusinessName,
|
||||
Description: req.Description,
|
||||
Status: req.Status,
|
||||
Config: req.Config,
|
||||
Creator: req.Operator,
|
||||
Updater: req.Operator,
|
||||
}
|
||||
|
||||
if req.Status == "" {
|
||||
biz.Status = model.StatusActive
|
||||
}
|
||||
if biz.Config == nil {
|
||||
biz.Config = make(map[string]interface{})
|
||||
}
|
||||
|
||||
if req.ID != nil && *req.ID > 0 {
|
||||
// 更新
|
||||
biz.ID = *req.ID
|
||||
if err := s.configLoader.UpdateBusiness(ctx, biz); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return &model.SaveResult{Success: true, ID: *req.ID, Message: "更新成功"}, nil
|
||||
}
|
||||
|
||||
// 新增
|
||||
id, err := s.configLoader.CreateBusiness(ctx, biz)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return &model.SaveResult{Success: true, ID: id, Message: "创建成功"}, nil
|
||||
}
|
||||
|
||||
// DeleteBusiness 删除业务配置
|
||||
func (s *ReportService) DeleteBusiness(ctx context.Context, id int64) (*model.DeleteResult, error) {
|
||||
biz, err := s.configLoader.GetBusinessByID(ctx, id)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if err := s.configLoader.DeleteBusiness(ctx, id, biz.BusinessCode); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return &model.DeleteResult{Success: true, Message: "删除成功"}, nil
|
||||
}
|
||||
|
||||
// GetBusiness 获取单个业务配置
|
||||
func (s *ReportService) GetBusiness(ctx context.Context, id int64) (*model.BusinessConfig, error) {
|
||||
return s.configLoader.GetBusinessByID(ctx, id)
|
||||
}
|
||||
|
||||
// ============================================================
|
||||
// 配置 CRUD: 报表
|
||||
// ============================================================
|
||||
|
||||
// SaveReport 保存报表配置（新增/修改合一）
|
||||
func (s *ReportService) SaveReport(ctx context.Context, req *model.SaveReportReq) (*model.SaveResult, error) {
|
||||
if err := initTables(ctx); err != nil {
|
||||
return nil, fmt.Errorf("初始化系统表失败: %w", err)
|
||||
}
|
||||
|
||||
rpt := &model.ReportConfig{
|
||||
BusinessCode: req.BusinessCode,
|
||||
ReportCode: req.ReportCode,
|
||||
ReportName: req.ReportName,
|
||||
Description: req.Description,
|
||||
Status: req.Status,
|
||||
StatTableName: req.StatTableName,
|
||||
StatTableComment: req.StatTableComment,
|
||||
DateField: req.DateField,
|
||||
PrimaryKeys: req.PrimaryKeys,
|
||||
ConflictKeys: req.ConflictKeys,
|
||||
Config: req.Config,
|
||||
Creator: req.Operator,
|
||||
Updater: req.Operator,
|
||||
}
|
||||
|
||||
if req.Status == "" {
|
||||
rpt.Status = model.StatusActive
|
||||
}
|
||||
if rpt.DateField == "" {
|
||||
rpt.DateField = "stat_date"
|
||||
}
|
||||
if rpt.PrimaryKeys == nil {
|
||||
rpt.PrimaryKeys = []string{"id"}
|
||||
}
|
||||
if rpt.ConflictKeys == nil {
|
||||
rpt.ConflictKeys = []string{rpt.DateField}
|
||||
}
|
||||
if rpt.Config == nil {
|
||||
rpt.Config = make(map[string]interface{})
|
||||
}
|
||||
|
||||
if req.ID != nil && *req.ID > 0 {
|
||||
rpt.ID = *req.ID
|
||||
if err := s.configLoader.UpdateReport(ctx, rpt); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return &model.SaveResult{Success: true, ID: *req.ID, Message: "更新成功"}, nil
|
||||
}
|
||||
|
||||
id, err := s.configLoader.CreateReport(ctx, rpt)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return &model.SaveResult{Success: true, ID: id, Message: "创建成功"}, nil
|
||||
}
|
||||
|
||||
// DeleteReport 删除报表配置
|
||||
func (s *ReportService) DeleteReport(ctx context.Context, id int64) (*model.DeleteResult, error) {
|
||||
rpt, err := s.configLoader.GetReportByID(ctx, id)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if err := s.configLoader.DeleteReport(ctx, id, rpt.BusinessCode, rpt.ReportCode); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return &model.DeleteResult{Success: true, Message: "删除成功"}, nil
|
||||
}
|
||||
|
||||
// GetReport 获取单个报表配置
|
||||
func (s *ReportService) GetReport(ctx context.Context, id int64) (*model.ReportConfig, error) {
|
||||
return s.configLoader.GetReportByID(ctx, id)
|
||||
}
|
||||
|
||||
// ============================================================
|
||||
// 配置 CRUD: 字段
|
||||
// ============================================================
|
||||
|
||||
// SaveField 保存字段配置（新增/修改合一）
|
||||
func (s *ReportService) SaveField(ctx context.Context, req *model.SaveFieldReq) (*model.SaveResult, error) {
|
||||
if err := initTables(ctx); err != nil {
|
||||
return nil, fmt.Errorf("初始化系统表失败: %w", err)
|
||||
}
|
||||
|
||||
field := &model.FieldConfig{
|
||||
BusinessCode: req.BusinessCode,
|
||||
ReportCode: req.ReportCode,
|
||||
FieldCode: req.FieldCode,
|
||||
FieldName: req.FieldName,
|
||||
FieldType: req.FieldType,
|
||||
DataType: req.DataType,
|
||||
FieldRole: req.FieldRole,
|
||||
IsAggregatable: req.IsAggregatable,
|
||||
IsFilterable: req.IsFilterable,
|
||||
IsQueryable: req.IsQueryable,
|
||||
IsSortable: req.IsSortable,
|
||||
DefaultAggregate: req.DefaultAggregate,
|
||||
ValidAggregates: req.ValidAggregates,
|
||||
FilterOperators: req.FilterOperators,
|
||||
Expression: req.Expression,
|
||||
ExpressionType: req.ExpressionType,
|
||||
FormatPattern: req.FormatPattern,
|
||||
Unit: req.Unit,
|
||||
DictCode: req.DictCode,
|
||||
SortOrder: req.SortOrder,
|
||||
GroupName: req.GroupName,
|
||||
Status: req.Status,
|
||||
Creator: req.Operator,
|
||||
Updater: req.Operator,
|
||||
}
|
||||
|
||||
if req.Status == "" {
|
||||
field.Status = model.StatusActive
|
||||
}
|
||||
if field.DataType == "" {
|
||||
field.DataType = model.FieldTypeString
|
||||
}
|
||||
if field.ValidAggregates == nil {
|
||||
field.ValidAggregates = []string{}
|
||||
}
|
||||
if field.FilterOperators == nil {
|
||||
field.FilterOperators = []string{"=", "!=", ">", "<", ">=", "<=", "IN", "LIKE", "BETWEEN"}
|
||||
}
|
||||
|
||||
if req.ID != nil && *req.ID > 0 {
|
||||
field.ID = *req.ID
|
||||
if err := s.configLoader.UpdateField(ctx, field); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return &model.SaveResult{Success: true, ID: *req.ID, Message: "更新成功"}, nil
|
||||
}
|
||||
|
||||
id, err := s.configLoader.CreateField(ctx, field)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return &model.SaveResult{Success: true, ID: id, Message: "创建成功"}, nil
|
||||
}
|
||||
|
||||
// DeleteField 删除字段配置
|
||||
func (s *ReportService) DeleteField(ctx context.Context, id int64) (*model.DeleteResult, error) {
|
||||
field, err := s.configLoader.GetFieldByID(ctx, id)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if err := s.configLoader.DeleteField(ctx, id, field.BusinessCode, field.ReportCode); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return &model.DeleteResult{Success: true, Message: "删除成功"}, nil
|
||||
}
|
||||
|
||||
// GetField 获取单个字段配置
|
||||
func (s *ReportService) GetField(ctx context.Context, id int64) (*model.FieldConfig, error) {
|
||||
return s.configLoader.GetFieldByID(ctx, id)
|
||||
}
|
||||
|
||||
// ============================================================
|
||||
// 配置 CRUD: 抽取配置
|
||||
// ============================================================
|
||||
|
||||
// SaveExtractConfig 保存抽取配置（新增/修改合一）
|
||||
func (s *ReportService) SaveExtractConfig(ctx context.Context, req *model.SaveExtractConfigReq) (*model.SaveResult, error) {
|
||||
if err := initTables(ctx); err != nil {
|
||||
return nil, fmt.Errorf("初始化系统表失败: %w", err)
|
||||
}
|
||||
|
||||
ec := &model.ExtractConfig{
|
||||
BusinessCode: req.BusinessCode,
|
||||
ReportCode: req.ReportCode,
|
||||
ExtractCode: req.ExtractCode,
|
||||
ExtractName: req.ExtractName,
|
||||
SourceTableName: req.SourceTableName,
|
||||
SourceTableAlias: req.SourceTableAlias,
|
||||
TargetTableName: req.TargetTableName,
|
||||
IsEnabled: req.IsEnabled,
|
||||
ExtractType: req.ExtractType,
|
||||
ExtractMode: req.ExtractMode,
|
||||
ExtractKeyField: req.ExtractKeyField,
|
||||
ExtractKeyFormat: req.ExtractKeyFormat,
|
||||
GroupByFields: req.GroupByFields,
|
||||
FilterExpression: req.FilterExpression,
|
||||
JoinConfigs: req.JoinConfigs,
|
||||
FieldMappings: req.FieldMappings,
|
||||
TransformRules: req.TransformRules,
|
||||
BatchSize: req.BatchSize,
|
||||
Status: req.Status,
|
||||
Creator: req.Operator,
|
||||
Updater: req.Operator,
|
||||
}
|
||||
|
||||
if req.Status == "" {
|
||||
ec.Status = model.StatusActive
|
||||
}
|
||||
if ec.ExtractType == "" {
|
||||
ec.ExtractType = model.ExtractTypeIncremental
|
||||
}
|
||||
if ec.ExtractMode == "" {
|
||||
ec.ExtractMode = model.ExtractModeDirect
|
||||
}
|
||||
if ec.BatchSize == 0 {
|
||||
ec.BatchSize = 1000
|
||||
}
|
||||
if ec.JoinConfigs == nil {
|
||||
ec.JoinConfigs = []model.JoinConfig{}
|
||||
}
|
||||
if ec.FieldMappings == nil {
|
||||
ec.FieldMappings = []model.FieldMapping{}
|
||||
}
|
||||
if ec.TransformRules == nil {
|
||||
ec.TransformRules = []model.TransformRule{}
|
||||
}
|
||||
if ec.GroupByFields == nil {
|
||||
ec.GroupByFields = []string{}
|
||||
}
|
||||
|
||||
if req.ID != nil && *req.ID > 0 {
|
||||
ec.ID = *req.ID
|
||||
if err := s.configLoader.UpdateExtractConfig(ctx, ec); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return &model.SaveResult{Success: true, ID: *req.ID, Message: "更新成功"}, nil
|
||||
}
|
||||
|
||||
id, err := s.configLoader.CreateExtractConfig(ctx, ec)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return &model.SaveResult{Success: true, ID: id, Message: "创建成功"}, nil
|
||||
}
|
||||
|
||||
// DeleteExtractConfig 删除抽取配置
|
||||
func (s *ReportService) DeleteExtractConfig(ctx context.Context, id int64) (*model.DeleteResult, error) {
|
||||
ec, err := s.configLoader.GetExtractConfigByID(ctx, id)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if err := s.configLoader.DeleteExtractConfig(ctx, id, ec.BusinessCode, ec.ReportCode); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return &model.DeleteResult{Success: true, Message: "删除成功"}, nil
|
||||
}
|
||||
|
||||
// GetExtractConfig 获取单个抽取配置
|
||||
func (s *ReportService) GetExtractConfig(ctx context.Context, id int64) (*model.ExtractConfig, error) {
|
||||
return s.configLoader.GetExtractConfigByID(ctx, id)
|
||||
}
|
||||
|
||||
// GetExtractConfigs 获取业务报表下所有抽取配置
|
||||
func (s *ReportService) GetExtractConfigs(ctx context.Context, businessCode, reportCode string) ([]model.ExtractConfig, error) {
|
||||
return s.configLoader.GetExtractConfigs(ctx, businessCode, reportCode)
|
||||
}
|
||||
@@ -0,0 +1,518 @@
|
||||
package builder
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"regexp"
|
||||
"strings"
|
||||
|
||||
"dataengine/common/report/config"
|
||||
"dataengine/common/report/model"
|
||||
)
|
||||
|
||||
// SQLBuilder 动态SQL构建器
|
||||
type SQLBuilder struct {
|
||||
loader *config.ConfigLoader
|
||||
}
|
||||
|
||||
// NewSQLBuilder 创建SQL构建器
|
||||
func NewSQLBuilder() *SQLBuilder {
|
||||
return &SQLBuilder{
|
||||
loader: config.GetLoader(),
|
||||
}
|
||||
}
|
||||
|
||||
// BuildQuerySQL 根据用户选择构建查询SQL
|
||||
func (b *SQLBuilder) BuildQuerySQL(ctx context.Context, req *model.UserSelectQueryReq) (string, []interface{}, map[string]interface{}, error) {
|
||||
// 1. 校验配置
|
||||
report, err := b.loader.GetReport(ctx, req.BusinessCode, req.ReportCode)
|
||||
if err != nil {
|
||||
return "", nil, nil, fmt.Errorf("获取报表配置失败: %w", err)
|
||||
}
|
||||
|
||||
fieldMap, err := b.loader.GetFieldMap(ctx, req.BusinessCode, req.ReportCode)
|
||||
if err != nil {
|
||||
return "", nil, nil, fmt.Errorf("获取字段配置失败: %w", err)
|
||||
}
|
||||
|
||||
tableName := report.StatTableName
|
||||
|
||||
// 2. 构建 SELECT 部分
|
||||
selectClause, err := b.buildSelectClause(req, fieldMap, report)
|
||||
if err != nil {
|
||||
return "", nil, nil, err
|
||||
}
|
||||
|
||||
// 3. 构建 FROM 部分
|
||||
fromClause := tableName
|
||||
|
||||
// 4. 构建 WHERE 部分
|
||||
whereClause, whereArgs, err := b.buildWhereClause(ctx, req, fieldMap, report)
|
||||
if err != nil {
|
||||
return "", nil, nil, err
|
||||
}
|
||||
|
||||
// 5. 构建 GROUP BY 部分
|
||||
groupByClause, err := b.buildGroupByClause(req, fieldMap)
|
||||
if err != nil {
|
||||
return "", nil, nil, err
|
||||
}
|
||||
|
||||
// 6. 构建 ORDER BY 部分
|
||||
orderByClause, err := b.buildOrderByClause(req, fieldMap)
|
||||
if err != nil {
|
||||
return "", nil, nil, err
|
||||
}
|
||||
|
||||
// 7. 组合完整SQL
|
||||
var sql strings.Builder
|
||||
sql.WriteString("SELECT ")
|
||||
sql.WriteString(selectClause)
|
||||
sql.WriteString(" FROM ")
|
||||
sql.WriteString(fromClause)
|
||||
|
||||
if whereClause != "" {
|
||||
sql.WriteString(" WHERE ")
|
||||
sql.WriteString(whereClause)
|
||||
}
|
||||
|
||||
if groupByClause != "" {
|
||||
sql.WriteString(" GROUP BY ")
|
||||
sql.WriteString(groupByClause)
|
||||
}
|
||||
|
||||
if orderByClause != "" {
|
||||
sql.WriteString(" ORDER BY ")
|
||||
sql.WriteString(orderByClause)
|
||||
}
|
||||
|
||||
// 8. 统计总数SQL
|
||||
countSql := "SELECT COUNT(*) FROM " + fromClause
|
||||
if whereClause != "" {
|
||||
countSql += " WHERE " + whereClause
|
||||
}
|
||||
if groupByClause != "" {
|
||||
countSql = fmt.Sprintf("SELECT COUNT(*) FROM (SELECT 1 FROM %s WHERE %s GROUP BY %s) AS t",
|
||||
fromClause, whereClause, groupByClause)
|
||||
}
|
||||
|
||||
metadata := map[string]interface{}{
|
||||
"countSql": countSql,
|
||||
"tableName": tableName,
|
||||
"reportConfig": report,
|
||||
}
|
||||
|
||||
return sql.String(), whereArgs, metadata, nil
|
||||
}
|
||||
|
||||
// buildSelectClause 构建SELECT子句
|
||||
func (b *SQLBuilder) buildSelectClause(req *model.UserSelectQueryReq, fieldMap map[string]*model.FieldConfig, report *model.ReportConfig) (string, error) {
|
||||
var selectParts []string
|
||||
|
||||
// 1. 添加维度字段
|
||||
for _, dim := range req.Dimensions {
|
||||
dim = strings.TrimSpace(dim)
|
||||
if dim == "" {
|
||||
continue
|
||||
}
|
||||
fc, ok := fieldMap[dim]
|
||||
if !ok {
|
||||
return "", fmt.Errorf("维度字段不存在: %s", dim)
|
||||
}
|
||||
if fc.FieldRole != model.RoleDimension && fc.FieldRole != model.RoleFilter {
|
||||
return "", fmt.Errorf("字段 %s 不可作为维度", dim)
|
||||
}
|
||||
selectParts = append(selectParts, dim)
|
||||
}
|
||||
|
||||
// 2. 添加指标字段(含聚合)
|
||||
if len(req.Indicators) == 0 {
|
||||
return "", fmt.Errorf("必须选择至少一个指标")
|
||||
}
|
||||
|
||||
for _, ind := range req.Indicators {
|
||||
fc, ok := fieldMap[ind.FieldCode]
|
||||
if !ok {
|
||||
return "", fmt.Errorf("指标字段不存在: %s", ind.FieldCode)
|
||||
}
|
||||
|
||||
alias := ind.Alias
|
||||
if alias == "" {
|
||||
alias = ind.FieldCode
|
||||
}
|
||||
|
||||
agg := strings.ToUpper(ind.Aggregate)
|
||||
if agg == "" {
|
||||
agg = fc.DefaultAggregate
|
||||
if agg == "" {
|
||||
agg = model.AggregateSum
|
||||
}
|
||||
}
|
||||
|
||||
// 校验聚合方式
|
||||
if len(fc.ValidAggregates) > 0 {
|
||||
valid := false
|
||||
for _, v := range fc.ValidAggregates {
|
||||
if strings.ToUpper(v) == agg {
|
||||
valid = true
|
||||
break
|
||||
}
|
||||
}
|
||||
if !valid {
|
||||
return "", fmt.Errorf("字段 %s 不支持聚合方式 %s", ind.FieldCode, agg)
|
||||
}
|
||||
}
|
||||
|
||||
// 处理衍生指标(表达式)
|
||||
if fc.ExpressionType == "CALCULATED" && fc.Expression != "" {
|
||||
expr := b.parseExpression(fc.Expression, req.Indicators)
|
||||
selectParts = append(selectParts, fmt.Sprintf("%s AS %s", expr, alias))
|
||||
} else {
|
||||
selectParts = append(selectParts, fmt.Sprintf("%s(%s) AS %s", agg, ind.FieldCode, alias))
|
||||
}
|
||||
}
|
||||
|
||||
// 3. 添加时间分组字段
|
||||
if req.TimeGroup != "" && req.TimeGroup != "day" {
|
||||
dateField := report.DateField
|
||||
if dateField == "" {
|
||||
dateField = "stat_date"
|
||||
}
|
||||
timeGroupExpr := b.buildTimeGroupExpr(dateField, req.TimeGroup)
|
||||
selectParts = append(selectParts, timeGroupExpr)
|
||||
}
|
||||
|
||||
return strings.Join(selectParts, ", "), nil
|
||||
}
|
||||
|
||||
// buildWhereClause 构建WHERE子句
|
||||
func (b *SQLBuilder) buildWhereClause(ctx context.Context, req *model.UserSelectQueryReq, fieldMap map[string]*model.FieldConfig, report *model.ReportConfig) (string, []interface{}, error) {
|
||||
var conditions []string
|
||||
var args []interface{}
|
||||
|
||||
// 1. 租户过滤
|
||||
conditions = append(conditions, "tenant_id = 1")
|
||||
|
||||
// 2. 时间范围过滤
|
||||
if req.TimeRange != nil {
|
||||
dateField := report.DateField
|
||||
if dateField == "" {
|
||||
dateField = "stat_date"
|
||||
}
|
||||
|
||||
if req.TimeRange.StartDate != "" {
|
||||
conditions = append(conditions, fmt.Sprintf("%s >= ?", dateField))
|
||||
args = append(args, req.TimeRange.StartDate)
|
||||
}
|
||||
if req.TimeRange.EndDate != "" {
|
||||
conditions = append(conditions, fmt.Sprintf("%s <= ?", dateField))
|
||||
args = append(args, req.TimeRange.EndDate)
|
||||
}
|
||||
}
|
||||
|
||||
// 3. 业务过滤
|
||||
conditions = append(conditions, "business_code = ?")
|
||||
args = append(args, req.BusinessCode)
|
||||
|
||||
// 4. 用户筛选条件
|
||||
for _, filter := range req.Filters {
|
||||
fc, ok := fieldMap[filter.FieldCode]
|
||||
if !ok {
|
||||
return "", nil, fmt.Errorf("筛选字段不存在: %s", filter.FieldCode)
|
||||
}
|
||||
|
||||
if !fc.IsFilterable {
|
||||
return "", nil, fmt.Errorf("字段 %s 不可用于筛选", filter.FieldCode)
|
||||
}
|
||||
|
||||
op := strings.ToUpper(filter.Operator)
|
||||
if op == "" {
|
||||
op = "="
|
||||
}
|
||||
|
||||
// 校验操作符
|
||||
if len(fc.FilterOperators) > 0 {
|
||||
valid := false
|
||||
for _, v := range fc.FilterOperators {
|
||||
if strings.ToUpper(v) == op {
|
||||
valid = true
|
||||
break
|
||||
}
|
||||
}
|
||||
if !valid {
|
||||
return "", nil, fmt.Errorf("字段 %s 不支持操作符 %s", filter.FieldCode, op)
|
||||
}
|
||||
}
|
||||
|
||||
cond, vals, err := b.buildFilterCondition(filter, op, fc.FieldType)
|
||||
if err != nil {
|
||||
return "", nil, err
|
||||
}
|
||||
conditions = append(conditions, cond)
|
||||
args = append(args, vals...)
|
||||
}
|
||||
|
||||
return strings.Join(conditions, " AND "), args, nil
|
||||
}
|
||||
|
||||
// buildFilterCondition 构建单个筛选条件
|
||||
func (b *SQLBuilder) buildFilterCondition(filter model.FilterCondition, op string, fieldType string) (string, []interface{}, error) {
|
||||
field := filter.FieldCode
|
||||
var args []interface{}
|
||||
|
||||
switch op {
|
||||
case "=":
|
||||
return fmt.Sprintf("%s = ?", field), []interface{}{filter.Value}, nil
|
||||
case "!=":
|
||||
return fmt.Sprintf("%s != ?", field), []interface{}{filter.Value}, nil
|
||||
case ">":
|
||||
return fmt.Sprintf("%s > ?", field), []interface{}{filter.Value}, nil
|
||||
case "<":
|
||||
return fmt.Sprintf("%s < ?", field), []interface{}{filter.Value}, nil
|
||||
case ">=":
|
||||
return fmt.Sprintf("%s >= ?", field), []interface{}{filter.Value}, nil
|
||||
case "<=":
|
||||
return fmt.Sprintf("%s <= ?", field), []interface{}{filter.Value}, nil
|
||||
case "IN":
|
||||
values, err := b.convertToSlice(filter.Value)
|
||||
if err != nil {
|
||||
return "", nil, err
|
||||
}
|
||||
placeholders := make([]string, len(values))
|
||||
for i := range values {
|
||||
placeholders[i] = "?"
|
||||
args = append(args, values[i])
|
||||
}
|
||||
return fmt.Sprintf("%s IN (%s)", field, strings.Join(placeholders, ",")), args, nil
|
||||
case "LIKE":
|
||||
return fmt.Sprintf("%s LIKE ?", field), []interface{}{"%" + fmt.Sprintf("%v", filter.Value) + "%"}, nil
|
||||
case "BETWEEN":
|
||||
return fmt.Sprintf("%s BETWEEN ? AND ?", field), []interface{}{filter.Value, filter.Value2}, nil
|
||||
default:
|
||||
return fmt.Sprintf("%s = ?", field), []interface{}{filter.Value}, nil
|
||||
}
|
||||
}
|
||||
|
||||
// buildGroupByClause 构建GROUP BY子句
|
||||
func (b *SQLBuilder) buildGroupByClause(req *model.UserSelectQueryReq, fieldMap map[string]*model.FieldConfig) (string, error) {
|
||||
var groupFields []string
|
||||
for _, dim := range req.Dimensions {
|
||||
fc, ok := fieldMap[dim]
|
||||
if !ok {
|
||||
continue
|
||||
}
|
||||
if fc.FieldRole == model.RoleDimension || fc.FieldRole == model.RoleFilter {
|
||||
groupFields = append(groupFields, dim)
|
||||
}
|
||||
}
|
||||
|
||||
if len(groupFields) == 0 {
|
||||
return "", nil
|
||||
}
|
||||
|
||||
return strings.Join(groupFields, ", "), nil
|
||||
}
|
||||
|
||||
// buildOrderByClause 构建ORDER BY子句
|
||||
func (b *SQLBuilder) buildOrderByClause(req *model.UserSelectQueryReq, fieldMap map[string]*model.FieldConfig) (string, error) {
|
||||
if len(req.OrderBy) == 0 {
|
||||
return "", nil
|
||||
}
|
||||
|
||||
var orderParts []string
|
||||
for _, order := range req.OrderBy {
|
||||
field := order.FieldCode
|
||||
dir := strings.ToUpper(order.Direction)
|
||||
if dir == "" {
|
||||
dir = "ASC"
|
||||
}
|
||||
if dir != "ASC" && dir != "DESC" {
|
||||
return "", fmt.Errorf("排序方向必须是 ASC 或 DESC")
|
||||
}
|
||||
|
||||
fc, ok := fieldMap[field]
|
||||
if !ok {
|
||||
return "", fmt.Errorf("排序字段不存在: %s", field)
|
||||
}
|
||||
|
||||
if !fc.IsSortable {
|
||||
return "", fmt.Errorf("字段 %s 不可排序", field)
|
||||
}
|
||||
|
||||
orderParts = append(orderParts, fmt.Sprintf("%s %s", field, dir))
|
||||
}
|
||||
|
||||
return strings.Join(orderParts, ", "), nil
|
||||
}
|
||||
|
||||
// buildTimeGroupExpr 构建时间分组表达式
|
||||
func (b *SQLBuilder) buildTimeGroupExpr(dateField, timeGroup string) string {
|
||||
switch timeGroup {
|
||||
case "week":
|
||||
return fmt.Sprintf("DATE_TRUNC('week', %s::date)::text AS time_group", dateField)
|
||||
case "month":
|
||||
return fmt.Sprintf("TO_CHAR(%s::date, 'YYYY-MM') AS time_group", dateField)
|
||||
case "quarter":
|
||||
return "TO_CHAR(" + dateField + "::date, 'YYYY-\"Q\"Q') AS time_group"
|
||||
default:
|
||||
return dateField + " AS time_group"
|
||||
}
|
||||
}
|
||||
|
||||
// parseExpression 解析衍生指标表达式
|
||||
func (b *SQLBuilder) parseExpression(expr string, indicators []model.IndicatorSelect) string {
|
||||
re := regexp.MustCompile(`\{([^}]+)\}`)
|
||||
return re.ReplaceAllStringFunc(expr, func(match string) string {
|
||||
fieldCode := match[1 : len(match)-1]
|
||||
for _, ind := range indicators {
|
||||
if ind.FieldCode == fieldCode {
|
||||
return fieldCode
|
||||
}
|
||||
}
|
||||
return match
|
||||
})
|
||||
}
|
||||
|
||||
// convertToSlice 转换为切片
|
||||
func (b *SQLBuilder) convertToSlice(v interface{}) ([]interface{}, error) {
|
||||
switch val := v.(type) {
|
||||
case []interface{}:
|
||||
return val, nil
|
||||
case []string:
|
||||
result := make([]interface{}, len(val))
|
||||
for i, s := range val {
|
||||
result[i] = s
|
||||
}
|
||||
return result, nil
|
||||
case string:
|
||||
parts := strings.Split(val, ",")
|
||||
result := make([]interface{}, len(parts))
|
||||
for i, p := range parts {
|
||||
result[i] = strings.TrimSpace(p)
|
||||
}
|
||||
return result, nil
|
||||
default:
|
||||
return []interface{}{v}, nil
|
||||
}
|
||||
}
|
||||
|
||||
// BuildCountSQL 构建统计总数SQL
|
||||
func (b *SQLBuilder) BuildCountSQL(sql string) string {
|
||||
sql = regexp.MustCompile(`(?i)SELECT\s+.*?\s+FROM`).ReplaceAllString(sql, "SELECT COUNT(*) FROM")
|
||||
return sql
|
||||
}
|
||||
|
||||
// AddLimit 添加分页
|
||||
func (b *SQLBuilder) AddLimit(sql string, page, pageSize int) string {
|
||||
if page < 1 {
|
||||
page = 1
|
||||
}
|
||||
if pageSize < 1 {
|
||||
pageSize = 20
|
||||
}
|
||||
if pageSize > 1000 {
|
||||
pageSize = 1000
|
||||
}
|
||||
offset := (page - 1) * pageSize
|
||||
return fmt.Sprintf("%s LIMIT %d OFFSET %d", sql, pageSize, offset)
|
||||
}
|
||||
|
||||
// GenerateInsertSQL 生成upsert SQL
|
||||
func (b *SQLBuilder) GenerateInsertSQL(tableName string, columns []string, conflictKeys []string) string {
|
||||
cols := strings.Join(columns, ", ")
|
||||
placeholders := make([]string, len(columns))
|
||||
for i := range columns {
|
||||
placeholders[i] = fmt.Sprintf("$%d", i+1)
|
||||
}
|
||||
placeholdersStr := strings.Join(placeholders, ", ")
|
||||
|
||||
var updateParts []string
|
||||
for _, col := range columns {
|
||||
if col == "id" || col == "created_at" {
|
||||
continue
|
||||
}
|
||||
updateParts = append(updateParts, fmt.Sprintf("%s = EXCLUDED.%s", col, col))
|
||||
}
|
||||
|
||||
sql := fmt.Sprintf("INSERT INTO %s (%s) VALUES (%s)", tableName, cols, placeholdersStr)
|
||||
|
||||
if len(conflictKeys) > 0 {
|
||||
sql += " ON CONFLICT (" + strings.Join(conflictKeys, ", ") + ")"
|
||||
sql += " DO UPDATE SET " + strings.Join(updateParts, ", ")
|
||||
}
|
||||
|
||||
return sql
|
||||
}
|
||||
|
||||
// BuildExtractSQL 构建数据抽取SQL
|
||||
func (b *SQLBuilder) BuildExtractSQL(ctx context.Context, extractConfig *model.ExtractConfig, statDate string) (string, []interface{}, error) {
|
||||
var selectParts []string
|
||||
var args []interface{}
|
||||
|
||||
// 基础字段
|
||||
selectParts = append(selectParts, "tenant_id")
|
||||
selectParts = append(selectParts, fmt.Sprintf("'%s' AS business_code", extractConfig.BusinessCode))
|
||||
selectParts = append(selectParts, fmt.Sprintf("'%s' AS stat_date", statDate))
|
||||
|
||||
// 字段映射
|
||||
sourceTable := extractConfig.SourceTableName
|
||||
if extractConfig.SourceTableAlias != "" {
|
||||
sourceTable = extractConfig.SourceTableAlias
|
||||
}
|
||||
|
||||
for _, mapping := range extractConfig.FieldMappings {
|
||||
targetField := mapping.TargetField
|
||||
sourceField := mapping.SourceField
|
||||
|
||||
if mapping.TransformRule != nil {
|
||||
expr := b.applyTransformRule(mapping.TransformRule, sourceField)
|
||||
selectParts = append(selectParts, fmt.Sprintf("%s AS %s", expr, targetField))
|
||||
} else {
|
||||
selectParts = append(selectParts, fmt.Sprintf("%s.%s AS %s", sourceTable, sourceField, targetField))
|
||||
}
|
||||
}
|
||||
|
||||
// 构建 FROM 和 JOIN
|
||||
fromClause := extractConfig.SourceTableName
|
||||
if extractConfig.SourceTableAlias != "" {
|
||||
fromClause += " " + extractConfig.SourceTableAlias
|
||||
}
|
||||
|
||||
for _, join := range extractConfig.JoinConfigs {
|
||||
joinType := "LEFT JOIN"
|
||||
if strings.ToUpper(join.JoinType) == "INNER" {
|
||||
joinType = "INNER JOIN"
|
||||
} else if strings.ToUpper(join.JoinType) == "RIGHT" {
|
||||
joinType = "RIGHT JOIN"
|
||||
}
|
||||
fromClause += fmt.Sprintf(" %s %s %s ON %s", joinType, join.JoinTable, join.JoinAlias, join.JoinCondition)
|
||||
}
|
||||
|
||||
// WHERE 条件
|
||||
whereClause := ""
|
||||
if extractConfig.FilterExpression != "" {
|
||||
whereClause = " WHERE " + extractConfig.FilterExpression
|
||||
}
|
||||
|
||||
sql := fmt.Sprintf("SELECT %s FROM %s%s", strings.Join(selectParts, ", "), fromClause, whereClause)
|
||||
|
||||
return sql, args, nil
|
||||
}
|
||||
|
||||
// applyTransformRule 应用转换规则
|
||||
func (b *SQLBuilder) applyTransformRule(rule *model.TransformRule, sourceField string) string {
|
||||
switch rule.RuleType {
|
||||
case "CALCULATE":
|
||||
if rule.Expression != "" {
|
||||
return strings.ReplaceAll(rule.Expression, "{source}", sourceField)
|
||||
}
|
||||
case "FORMAT":
|
||||
if rule.Format != "" {
|
||||
return fmt.Sprintf("TO_CHAR(%s, '%s')", sourceField, rule.Format)
|
||||
}
|
||||
case "MAPPING":
|
||||
// 运行时映射，需要在代码中处理
|
||||
return sourceField
|
||||
}
|
||||
return sourceField
|
||||
}
|
||||
@@ -0,0 +1,664 @@
|
||||
package config
|
||||
|
||||
import (
|
||||
"context"
|
||||
"database/sql"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"sync"
|
||||
|
||||
"dataengine/common/report/model"
|
||||
|
||||
"gitea.redpowerfuture.com/red-future/common/db/gfdb"
|
||||
"github.com/gogf/gf/v2/frame/g"
|
||||
)
|
||||
|
||||
// ConfigLoader 配置加载器
|
||||
type ConfigLoader struct {
|
||||
mu sync.RWMutex
|
||||
// 缓存
|
||||
businessCache map[string]*model.BusinessConfig
|
||||
reportCache map[string]*model.ReportConfig
|
||||
fieldCache map[string][]model.FieldConfig
|
||||
extractCache map[string][]model.ExtractConfig
|
||||
}
|
||||
|
||||
var (
|
||||
defaultLoader *ConfigLoader
|
||||
once sync.Once
|
||||
)
|
||||
|
||||
// GetLoader 获取配置加载器单例
|
||||
func GetLoader() *ConfigLoader {
|
||||
once.Do(func() {
|
||||
defaultLoader = &ConfigLoader{
|
||||
businessCache: make(map[string]*model.BusinessConfig),
|
||||
reportCache: make(map[string]*model.ReportConfig),
|
||||
fieldCache: make(map[string][]model.FieldConfig),
|
||||
extractCache: make(map[string][]model.ExtractConfig),
|
||||
}
|
||||
})
|
||||
return defaultLoader
|
||||
}
|
||||
|
||||
// GetBusiness 获取业务配置
|
||||
func (l *ConfigLoader) GetBusiness(ctx context.Context, businessCode string) (*model.BusinessConfig, error) {
|
||||
l.mu.RLock()
|
||||
if biz, ok := l.businessCache[businessCode]; ok {
|
||||
l.mu.RUnlock()
|
||||
return biz, nil
|
||||
}
|
||||
l.mu.RUnlock()
|
||||
|
||||
var biz model.BusinessConfig
|
||||
r, err := gfdb.DB(ctx).Model(ctx, "report_business_config").
|
||||
Where("business_code", businessCode).
|
||||
Where("status", model.StatusActive).
|
||||
One()
|
||||
if err != nil {
|
||||
if err == sql.ErrNoRows {
|
||||
return nil, fmt.Errorf("业务配置不存在: %s", businessCode)
|
||||
}
|
||||
return nil, err
|
||||
}
|
||||
if r.IsEmpty() {
|
||||
return nil, fmt.Errorf("业务配置不存在: %s", businessCode)
|
||||
}
|
||||
if err = r.Struct(&biz); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
if biz.Config == nil {
|
||||
biz.Config = make(map[string]interface{})
|
||||
}
|
||||
|
||||
l.mu.Lock()
|
||||
l.businessCache[businessCode] = &biz
|
||||
l.mu.Unlock()
|
||||
|
||||
return &biz, nil
|
||||
}
|
||||
|
||||
// GetReport 获取报表配置
|
||||
func (l *ConfigLoader) GetReport(ctx context.Context, businessCode, reportCode string) (*model.ReportConfig, error) {
|
||||
key := businessCode + ":" + reportCode
|
||||
l.mu.RLock()
|
||||
if rpt, ok := l.reportCache[key]; ok {
|
||||
l.mu.RUnlock()
|
||||
return rpt, nil
|
||||
}
|
||||
l.mu.RUnlock()
|
||||
|
||||
var rpt model.ReportConfig
|
||||
r, err := gfdb.DB(ctx).Model(ctx, "report_report_config").
|
||||
Where("business_code", businessCode).
|
||||
Where("report_code", reportCode).
|
||||
Where("status", model.StatusActive).
|
||||
One()
|
||||
if err != nil {
|
||||
if err == sql.ErrNoRows {
|
||||
return nil, fmt.Errorf("报表配置不存在: %s/%s", businessCode, reportCode)
|
||||
}
|
||||
return nil, err
|
||||
}
|
||||
if r.IsEmpty() {
|
||||
return nil, fmt.Errorf("报表配置不存在: %s/%s", businessCode, reportCode)
|
||||
}
|
||||
if err = r.Struct(&rpt); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
if rpt.PrimaryKeys == nil {
|
||||
rpt.PrimaryKeys = []string{"id"}
|
||||
}
|
||||
if rpt.ConflictKeys == nil {
|
||||
rpt.ConflictKeys = []string{"stat_date"}
|
||||
}
|
||||
if rpt.Config == nil {
|
||||
rpt.Config = make(map[string]interface{})
|
||||
}
|
||||
|
||||
l.mu.Lock()
|
||||
l.reportCache[key] = &rpt
|
||||
l.mu.Unlock()
|
||||
|
||||
return &rpt, nil
|
||||
}
|
||||
|
||||
// GetFields 获取报表字段配置
|
||||
func (l *ConfigLoader) GetFields(ctx context.Context, businessCode, reportCode string) ([]model.FieldConfig, error) {
|
||||
key := businessCode + ":" + reportCode
|
||||
l.mu.RLock()
|
||||
if fields, ok := l.fieldCache[key]; ok {
|
||||
l.mu.RUnlock()
|
||||
return fields, nil
|
||||
}
|
||||
l.mu.RUnlock()
|
||||
|
||||
var fields []model.FieldConfig
|
||||
err := gfdb.DB(ctx).Model(ctx, "report_field_config").
|
||||
Where("business_code", businessCode).
|
||||
Where("report_code", reportCode).
|
||||
Where("status", model.StatusActive).
|
||||
Order("sort_order ASC").
|
||||
Scan(&fields)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
for i := range fields {
|
||||
if fields[i].ValidAggregates == nil {
|
||||
fields[i].ValidAggregates = []string{}
|
||||
}
|
||||
if fields[i].FilterOperators == nil {
|
||||
fields[i].FilterOperators = []string{"=", "!=", ">", "<", ">=", "<=", "IN", "LIKE", "BETWEEN"}
|
||||
}
|
||||
}
|
||||
|
||||
l.mu.Lock()
|
||||
l.fieldCache[key] = fields
|
||||
l.mu.Unlock()
|
||||
|
||||
return fields, nil
|
||||
}
|
||||
|
||||
// GetFieldMap 获取字段配置Map
|
||||
func (l *ConfigLoader) GetFieldMap(ctx context.Context, businessCode, reportCode string) (map[string]*model.FieldConfig, error) {
|
||||
fields, err := l.GetFields(ctx, businessCode, reportCode)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
fieldMap := make(map[string]*model.FieldConfig)
|
||||
for i := range fields {
|
||||
fieldMap[fields[i].FieldCode] = &fields[i]
|
||||
}
|
||||
return fieldMap, nil
|
||||
}
|
||||
|
||||
// GetExtractConfigs 获取抽取配置
|
||||
func (l *ConfigLoader) GetExtractConfigs(ctx context.Context, businessCode, reportCode string) ([]model.ExtractConfig, error) {
|
||||
key := businessCode + ":" + reportCode
|
||||
l.mu.RLock()
|
||||
if configs, ok := l.extractCache[key]; ok {
|
||||
l.mu.RUnlock()
|
||||
return configs, nil
|
||||
}
|
||||
l.mu.RUnlock()
|
||||
|
||||
var configs []model.ExtractConfig
|
||||
err := gfdb.DB(ctx).Model(ctx, "report_extract_config").
|
||||
Where("business_code", businessCode).
|
||||
Where("report_code", reportCode).
|
||||
Where("status", model.StatusActive).
|
||||
Where("is_enabled", true).
|
||||
Scan(&configs)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
for i := range configs {
|
||||
if configs[i].JoinConfigs == nil {
|
||||
configs[i].JoinConfigs = []model.JoinConfig{}
|
||||
}
|
||||
if configs[i].FieldMappings == nil {
|
||||
configs[i].FieldMappings = []model.FieldMapping{}
|
||||
}
|
||||
if configs[i].TransformRules == nil {
|
||||
configs[i].TransformRules = []model.TransformRule{}
|
||||
}
|
||||
if configs[i].GroupByFields == nil {
|
||||
configs[i].GroupByFields = []string{}
|
||||
}
|
||||
if configs[i].ExtractMode == "" {
|
||||
configs[i].ExtractMode = model.ExtractModeDirect
|
||||
}
|
||||
}
|
||||
|
||||
l.mu.Lock()
|
||||
l.extractCache[key] = configs
|
||||
l.mu.Unlock()
|
||||
|
||||
return configs, nil
|
||||
}
|
||||
|
||||
// GetExtractLog 获取抽取记录
|
||||
func (l *ConfigLoader) GetExtractLog(ctx context.Context, businessCode, reportCode, extractCode, statDate string) (*model.ExtractLog, error) {
|
||||
var log model.ExtractLog
|
||||
r, err := gfdb.DB(ctx).Model(ctx, "report_extract_log").
|
||||
Where("business_code", businessCode).
|
||||
Where("report_code", reportCode).
|
||||
Where("extract_code", extractCode).
|
||||
Where("stat_date", statDate).
|
||||
One()
|
||||
if err != nil {
|
||||
if err == sql.ErrNoRows {
|
||||
return nil, nil
|
||||
}
|
||||
return nil, err
|
||||
}
|
||||
if r.IsEmpty() {
|
||||
return nil, nil
|
||||
}
|
||||
if err = r.Struct(&log); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return &log, nil
|
||||
}
|
||||
|
||||
// CreateExtractLog 创建抽取记录
|
||||
func (l *ConfigLoader) CreateExtractLog(ctx context.Context, log *model.ExtractLog) error {
|
||||
data, _ := json.Marshal(log)
|
||||
var m map[string]interface{}
|
||||
json.Unmarshal(data, &m)
|
||||
delete(m, "id")
|
||||
_, err := gfdb.DB(ctx).Model(ctx, "report_extract_log").Data(m).Save()
|
||||
return err
|
||||
}
|
||||
|
||||
// UpdateExtractLog 更新抽取记录
|
||||
func (l *ConfigLoader) UpdateExtractLog(ctx context.Context, log *model.ExtractLog) error {
|
||||
data, _ := json.Marshal(log)
|
||||
var m map[string]interface{}
|
||||
json.Unmarshal(data, &m)
|
||||
delete(m, "id")
|
||||
delete(m, "created_at")
|
||||
_, err := gfdb.DB(ctx).Model(ctx, "report_extract_log").
|
||||
Where("business_code", log.BusinessCode).
|
||||
Where("report_code", log.ReportCode).
|
||||
Where("extract_code", log.ExtractCode).
|
||||
Where("stat_date", log.StatDate).
|
||||
Data(m).
|
||||
Update()
|
||||
return err
|
||||
}
|
||||
|
||||
// InvalidateCache 失效缓存
|
||||
func (l *ConfigLoader) InvalidateCache(businessCode, reportCode string) {
|
||||
l.mu.Lock()
|
||||
delete(l.businessCache, businessCode)
|
||||
delete(l.reportCache, businessCode+":"+reportCode)
|
||||
delete(l.fieldCache, businessCode+":"+reportCode)
|
||||
delete(l.extractCache, businessCode+":"+reportCode)
|
||||
l.mu.Unlock()
|
||||
}
|
||||
|
||||
// InvalidateBusinessCache 只失效业务缓存（不影响报表/字段）
|
||||
func (l *ConfigLoader) InvalidateBusinessCache(businessCode string) {
|
||||
l.mu.Lock()
|
||||
delete(l.businessCache, businessCode)
|
||||
l.mu.Unlock()
|
||||
}
|
||||
|
||||
// ============================================================
|
||||
// CRUD: BusinessConfig
|
||||
// ============================================================
|
||||
|
||||
// CreateBusiness 创建业务配置
|
||||
func (l *ConfigLoader) CreateBusiness(ctx context.Context, biz *model.BusinessConfig) (int64, error) {
|
||||
data, _ := json.Marshal(biz)
|
||||
var m map[string]interface{}
|
||||
json.Unmarshal(data, &m)
|
||||
delete(m, "id")
|
||||
delete(m, "created_at")
|
||||
delete(m, "updated_at")
|
||||
delete(m, "deleted_at")
|
||||
|
||||
result, err := gfdb.DB(ctx).Model(ctx, "report_business_config").Data(m).Insert()
|
||||
if err != nil {
|
||||
return 0, fmt.Errorf("创建业务配置失败: %w", err)
|
||||
}
|
||||
id, _ := result.LastInsertId()
|
||||
l.InvalidateBusinessCache(biz.BusinessCode)
|
||||
return id, nil
|
||||
}
|
||||
|
||||
// UpdateBusiness 更新业务配置
|
||||
func (l *ConfigLoader) UpdateBusiness(ctx context.Context, biz *model.BusinessConfig) error {
|
||||
data, _ := json.Marshal(biz)
|
||||
var m map[string]interface{}
|
||||
json.Unmarshal(data, &m)
|
||||
delete(m, "id")
|
||||
delete(m, "created_at")
|
||||
delete(m, "deleted_at")
|
||||
|
||||
_, err := gfdb.DB(ctx).Model(ctx, "report_business_config").
|
||||
Where("id", biz.ID).
|
||||
Data(m).
|
||||
Update()
|
||||
if err != nil {
|
||||
return fmt.Errorf("更新业务配置失败: %w", err)
|
||||
}
|
||||
l.InvalidateBusinessCache(biz.BusinessCode)
|
||||
return nil
|
||||
}
|
||||
|
||||
// DeleteBusiness 删除业务配置（软删除）
|
||||
func (l *ConfigLoader) DeleteBusiness(ctx context.Context, id int64, businessCode string) error {
|
||||
_, err := gfdb.DB(ctx).Model(ctx, "report_business_config").
|
||||
Where("id", id).
|
||||
Data(map[string]interface{}{
|
||||
"status": model.StatusInactive,
|
||||
"deleted_at": "NOW()",
|
||||
}).
|
||||
Update()
|
||||
if err != nil {
|
||||
return fmt.Errorf("删除业务配置失败: %w", err)
|
||||
}
|
||||
l.InvalidateBusinessCache(businessCode)
|
||||
return nil
|
||||
}
|
||||
|
||||
// GetBusinessByID 根据ID获取业务配置
|
||||
func (l *ConfigLoader) GetBusinessByID(ctx context.Context, id int64) (*model.BusinessConfig, error) {
|
||||
var biz model.BusinessConfig
|
||||
r, err := gfdb.DB(ctx).Model(ctx, "report_business_config").
|
||||
Where("id", id).
|
||||
One()
|
||||
if err != nil {
|
||||
g.Log().Infof(ctx, "[GetBusinessByID] id=%d, err=%v", id, err)
|
||||
if err == sql.ErrNoRows {
|
||||
return nil, fmt.Errorf("业务配置不存在: id=%d", id)
|
||||
}
|
||||
return nil, err
|
||||
}
|
||||
if r.IsEmpty() {
|
||||
return nil, fmt.Errorf("业务配置不存在: id=%d", id)
|
||||
}
|
||||
if err = r.Struct(&biz); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
g.Log().Infof(ctx, "[GetBusinessByID] id=%d, biz.ID=%d, biz.BusinessCode=%s",
|
||||
id, biz.ID, biz.BusinessCode)
|
||||
return &biz, nil
|
||||
}
|
||||
|
||||
// ============================================================
|
||||
// CRUD: ReportConfig
|
||||
// ============================================================
|
||||
|
||||
// CreateReport 创建报表配置
|
||||
func (l *ConfigLoader) CreateReport(ctx context.Context, rpt *model.ReportConfig) (int64, error) {
|
||||
data, _ := json.Marshal(rpt)
|
||||
var m map[string]interface{}
|
||||
json.Unmarshal(data, &m)
|
||||
delete(m, "id")
|
||||
delete(m, "created_at")
|
||||
delete(m, "updated_at")
|
||||
delete(m, "deleted_at")
|
||||
|
||||
result, err := gfdb.DB(ctx).Model(ctx, "report_report_config").Data(m).Insert()
|
||||
if err != nil {
|
||||
return 0, fmt.Errorf("创建报表配置失败: %w", err)
|
||||
}
|
||||
id, _ := result.LastInsertId()
|
||||
l.InvalidateCache(rpt.BusinessCode, rpt.ReportCode)
|
||||
return id, nil
|
||||
}
|
||||
|
||||
// UpdateReport 更新报表配置
|
||||
func (l *ConfigLoader) UpdateReport(ctx context.Context, rpt *model.ReportConfig) error {
|
||||
data, _ := json.Marshal(rpt)
|
||||
var m map[string]interface{}
|
||||
json.Unmarshal(data, &m)
|
||||
delete(m, "id")
|
||||
delete(m, "created_at")
|
||||
delete(m, "deleted_at")
|
||||
|
||||
_, err := gfdb.DB(ctx).Model(ctx, "report_report_config").
|
||||
Where("id", rpt.ID).
|
||||
Data(m).
|
||||
Update()
|
||||
if err != nil {
|
||||
return fmt.Errorf("更新报表配置失败: %w", err)
|
||||
}
|
||||
l.InvalidateCache(rpt.BusinessCode, rpt.ReportCode)
|
||||
return nil
|
||||
}
|
||||
|
||||
// DeleteReport 删除报表配置（软删除）
|
||||
func (l *ConfigLoader) DeleteReport(ctx context.Context, id int64, businessCode, reportCode string) error {
|
||||
_, err := gfdb.DB(ctx).Model(ctx, "report_report_config").
|
||||
Where("id", id).
|
||||
Data(map[string]interface{}{
|
||||
"status": model.StatusInactive,
|
||||
"deleted_at": "NOW()",
|
||||
}).
|
||||
Update()
|
||||
if err != nil {
|
||||
return fmt.Errorf("删除报表配置失败: %w", err)
|
||||
}
|
||||
l.InvalidateCache(businessCode, reportCode)
|
||||
return nil
|
||||
}
|
||||
|
||||
// GetReportByID 根据ID获取报表配置
|
||||
func (l *ConfigLoader) GetReportByID(ctx context.Context, id int64) (*model.ReportConfig, error) {
|
||||
var rpt model.ReportConfig
|
||||
r, err := gfdb.DB(ctx).Model(ctx, "report_report_config").
|
||||
Where("id", id).
|
||||
One()
|
||||
if err != nil {
|
||||
if err == sql.ErrNoRows {
|
||||
return nil, fmt.Errorf("报表配置不存在: id=%d", id)
|
||||
}
|
||||
return nil, err
|
||||
}
|
||||
if r.IsEmpty() {
|
||||
return nil, fmt.Errorf("报表配置不存在: id=%d", id)
|
||||
}
|
||||
if err = r.Struct(&rpt); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return &rpt, nil
|
||||
}
|
||||
|
||||
// ============================================================
|
||||
// CRUD: FieldConfig
|
||||
// ============================================================
|
||||
|
||||
// CreateField 创建字段配置
|
||||
func (l *ConfigLoader) CreateField(ctx context.Context, field *model.FieldConfig) (int64, error) {
|
||||
data, _ := json.Marshal(field)
|
||||
var m map[string]interface{}
|
||||
json.Unmarshal(data, &m)
|
||||
delete(m, "id")
|
||||
delete(m, "created_at")
|
||||
delete(m, "updated_at")
|
||||
delete(m, "deleted_at")
|
||||
|
||||
result, err := gfdb.DB(ctx).Model(ctx, "report_field_config").Data(m).Insert()
|
||||
if err != nil {
|
||||
return 0, fmt.Errorf("创建字段配置失败: %w", err)
|
||||
}
|
||||
id, _ := result.LastInsertId()
|
||||
l.InvalidateCache(field.BusinessCode, field.ReportCode)
|
||||
return id, nil
|
||||
}
|
||||
|
||||
// UpdateField 更新字段配置
|
||||
func (l *ConfigLoader) UpdateField(ctx context.Context, field *model.FieldConfig) error {
|
||||
data, _ := json.Marshal(field)
|
||||
var m map[string]interface{}
|
||||
json.Unmarshal(data, &m)
|
||||
delete(m, "id")
|
||||
delete(m, "created_at")
|
||||
delete(m, "deleted_at")
|
||||
|
||||
_, err := gfdb.DB(ctx).Model(ctx, "report_field_config").
|
||||
Where("id", field.ID).
|
||||
Data(m).
|
||||
Update()
|
||||
if err != nil {
|
||||
return fmt.Errorf("更新字段配置失败: %w", err)
|
||||
}
|
||||
l.InvalidateCache(field.BusinessCode, field.ReportCode)
|
||||
return nil
|
||||
}
|
||||
|
||||
// DeleteField 删除字段配置（软删除）
|
||||
func (l *ConfigLoader) DeleteField(ctx context.Context, id int64, businessCode, reportCode string) error {
|
||||
_, err := gfdb.DB(ctx).Model(ctx, "report_field_config").
|
||||
Where("id", id).
|
||||
Data(map[string]interface{}{
|
||||
"status": model.StatusInactive,
|
||||
"deleted_at": "NOW()",
|
||||
}).
|
||||
Update()
|
||||
if err != nil {
|
||||
return fmt.Errorf("删除字段配置失败: %w", err)
|
||||
}
|
||||
l.InvalidateCache(businessCode, reportCode)
|
||||
return nil
|
||||
}
|
||||
|
||||
// GetFieldByID 根据ID获取字段配置
|
||||
func (l *ConfigLoader) GetFieldByID(ctx context.Context, id int64) (*model.FieldConfig, error) {
|
||||
var field model.FieldConfig
|
||||
r, err := gfdb.DB(ctx).Model(ctx, "report_field_config").
|
||||
Where("id", id).
|
||||
One()
|
||||
if err != nil {
|
||||
if err == sql.ErrNoRows {
|
||||
return nil, fmt.Errorf("字段配置不存在: id=%d", id)
|
||||
}
|
||||
return nil, err
|
||||
}
|
||||
if r.IsEmpty() {
|
||||
return nil, fmt.Errorf("字段配置不存在: id=%d", id)
|
||||
}
|
||||
if err = r.Struct(&field); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return &field, nil
|
||||
}
|
||||
|
||||
// ============================================================
|
||||
// CRUD: ExtractConfig
|
||||
// ============================================================
|
||||
|
||||
// CreateExtractConfig 创建抽取配置
|
||||
func (l *ConfigLoader) CreateExtractConfig(ctx context.Context, ec *model.ExtractConfig) (int64, error) {
|
||||
data, _ := json.Marshal(ec)
|
||||
var m map[string]interface{}
|
||||
json.Unmarshal(data, &m)
|
||||
delete(m, "id")
|
||||
delete(m, "created_at")
|
||||
delete(m, "updated_at")
|
||||
delete(m, "deleted_at")
|
||||
|
||||
result, err := gfdb.DB(ctx).Model(ctx, "report_extract_config").Data(m).Insert()
|
||||
if err != nil {
|
||||
return 0, fmt.Errorf("创建抽取配置失败: %w", err)
|
||||
}
|
||||
id, _ := result.LastInsertId()
|
||||
l.InvalidateCache(ec.BusinessCode, ec.ReportCode)
|
||||
return id, nil
|
||||
}
|
||||
|
||||
// UpdateExtractConfig 更新抽取配置
|
||||
func (l *ConfigLoader) UpdateExtractConfig(ctx context.Context, ec *model.ExtractConfig) error {
|
||||
data, _ := json.Marshal(ec)
|
||||
var m map[string]interface{}
|
||||
json.Unmarshal(data, &m)
|
||||
delete(m, "id")
|
||||
delete(m, "created_at")
|
||||
delete(m, "deleted_at")
|
||||
|
||||
_, err := gfdb.DB(ctx).Model(ctx, "report_extract_config").
|
||||
Where("id", ec.ID).
|
||||
Data(m).
|
||||
Update()
|
||||
if err != nil {
|
||||
return fmt.Errorf("更新抽取配置失败: %w", err)
|
||||
}
|
||||
l.InvalidateCache(ec.BusinessCode, ec.ReportCode)
|
||||
return nil
|
||||
}
|
||||
|
||||
// DeleteExtractConfig 删除抽取配置（软删除）
|
||||
func (l *ConfigLoader) DeleteExtractConfig(ctx context.Context, id int64, businessCode, reportCode string) error {
|
||||
_, err := gfdb.DB(ctx).Model(ctx, "report_extract_config").
|
||||
Where("id", id).
|
||||
Data(map[string]interface{}{
|
||||
"status": model.StatusInactive,
|
||||
"deleted_at": "NOW()",
|
||||
}).
|
||||
Update()
|
||||
if err != nil {
|
||||
return fmt.Errorf("删除抽取配置失败: %w", err)
|
||||
}
|
||||
l.InvalidateCache(businessCode, reportCode)
|
||||
return nil
|
||||
}
|
||||
|
||||
// GetExtractConfigByID 根据ID获取抽取配置
|
||||
func (l *ConfigLoader) GetExtractConfigByID(ctx context.Context, id int64) (*model.ExtractConfig, error) {
|
||||
var ec model.ExtractConfig
|
||||
r, err := gfdb.DB(ctx).Model(ctx, "report_extract_config").
|
||||
Where("id", id).
|
||||
One()
|
||||
if err != nil {
|
||||
if err == sql.ErrNoRows {
|
||||
return nil, fmt.Errorf("抽取配置不存在: id=%d", id)
|
||||
}
|
||||
return nil, err
|
||||
}
|
||||
if r.IsEmpty() {
|
||||
return nil, fmt.Errorf("抽取配置不存在: id=%d", id)
|
||||
}
|
||||
if err = r.Struct(&ec); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return &ec, nil
|
||||
}
|
||||
|
||||
// GetAllBusinesses 获取所有业务配置
|
||||
func (l *ConfigLoader) GetAllBusinesses(ctx context.Context) ([]model.BusinessConfig, error) {
|
||||
var businesses []model.BusinessConfig
|
||||
err := gfdb.DB(ctx).Model(ctx, "report_business_config").
|
||||
Where("status", model.StatusActive).
|
||||
Order("id ASC").
|
||||
Scan(&businesses)
|
||||
return businesses, err
|
||||
}
|
||||
|
||||
// GetAllReports 获取所有报表配置
|
||||
func (l *ConfigLoader) GetAllReports(ctx context.Context, businessCode string) ([]model.ReportConfig, error) {
|
||||
var reports []model.ReportConfig
|
||||
err := gfdb.DB(ctx).Model(ctx, "report_report_config").
|
||||
Where("business_code", businessCode).
|
||||
Where("status", model.StatusActive).
|
||||
Order("id ASC").
|
||||
Scan(&reports)
|
||||
return reports, err
|
||||
}
|
||||
|
||||
// GetReportFields 获取报表可用字段(按角色分类)
|
||||
func (l *ConfigLoader) GetReportFields(ctx context.Context, businessCode, reportCode string) (*model.GetReportFieldsResp, error) {
|
||||
fields, err := l.GetFields(ctx, businessCode, reportCode)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
resp := &model.GetReportFieldsResp{
|
||||
BusinessCode: businessCode,
|
||||
ReportCode: reportCode,
|
||||
Dimensions: []model.FieldConfig{},
|
||||
Indicators: []model.FieldConfig{},
|
||||
Filters: []model.FieldConfig{},
|
||||
}
|
||||
|
||||
for _, f := range fields {
|
||||
switch f.FieldRole {
|
||||
case model.RoleDimension:
|
||||
resp.Dimensions = append(resp.Dimensions, f)
|
||||
case model.RoleIndicator:
|
||||
resp.Indicators = append(resp.Indicators, f)
|
||||
case model.RoleFilter, model.RoleFilterOnly:
|
||||
resp.Filters = append(resp.Filters, f)
|
||||
}
|
||||
}
|
||||
|
||||
return resp, nil
|
||||
}
|
||||
@@ -0,0 +1,210 @@
|
||||
package ddlsync
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"dataengine/common/report/config"
|
||||
"dataengine/common/report/model"
|
||||
|
||||
"gitea.redpowerfuture.com/red-future/common/db/gfdb"
|
||||
"github.com/sirupsen/logrus"
|
||||
)
|
||||
|
||||
// StatTableCreator 统计宽表创建器
|
||||
type StatTableCreator struct {
|
||||
loader *config.ConfigLoader
|
||||
}
|
||||
|
||||
// NewStatTableCreator 创建统计宽表创建器
|
||||
func NewStatTableCreator() *StatTableCreator {
|
||||
return &StatTableCreator{
|
||||
loader: config.GetLoader(),
|
||||
}
|
||||
}
|
||||
|
||||
// AutoCreateStatTable 根据配置自动创建统计宽表
|
||||
func (c *StatTableCreator) AutoCreateStatTable(ctx context.Context, businessCode, reportCode string) (*model.AutoCreateStatTableResp, error) {
|
||||
start := time.Now()
|
||||
logger := logrus.WithFields(logrus.Fields{
|
||||
"businessCode": businessCode,
|
||||
"reportCode": reportCode,
|
||||
})
|
||||
|
||||
// 1. 获取报表配置
|
||||
report, err := c.loader.GetReport(ctx, businessCode, reportCode)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("获取报表配置失败: %w", err)
|
||||
}
|
||||
|
||||
// 2. 获取字段配置
|
||||
fields, err := c.loader.GetFields(ctx, businessCode, reportCode)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("获取字段配置失败: %w", err)
|
||||
}
|
||||
|
||||
if len(fields) == 0 {
|
||||
return nil, fmt.Errorf("报表字段配置为空，请先配置字段")
|
||||
}
|
||||
|
||||
// 3. 构建建表SQL
|
||||
tableName := report.StatTableName
|
||||
sql := c.buildCreateTableSQL(tableName, report, fields)
|
||||
|
||||
logger.Infof("创建统计宽表: %s", tableName)
|
||||
|
||||
// 4. 执行建表
|
||||
_, err = gfdb.DB(ctx).Exec(ctx, sql)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("建表失败: %w", err)
|
||||
}
|
||||
|
||||
// 5. 创建冲突唯一索引
|
||||
if len(report.ConflictKeys) > 0 {
|
||||
indexName := fmt.Sprintf("uq_%s_conflict", tableName)
|
||||
indexCols := strings.Join(report.ConflictKeys, ", ")
|
||||
indexSQL := fmt.Sprintf("CREATE UNIQUE INDEX IF NOT EXISTS %s ON %s (%s)", indexName, tableName, indexCols)
|
||||
if _, err := gfdb.DB(ctx).Exec(ctx, indexSQL); err != nil {
|
||||
logger.Warnf("创建冲突索引失败: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// 6. 添加字段注释
|
||||
for _, f := range fields {
|
||||
if f.FieldName != "" {
|
||||
commentSQL := fmt.Sprintf("COMMENT ON COLUMN %s.%s IS '%s'", tableName, f.FieldCode, strings.ReplaceAll(f.FieldName, "'", "''"))
|
||||
if _, err := gfdb.DB(ctx).Exec(ctx, commentSQL); err != nil {
|
||||
logger.Warnf("添加字段注释失败 [%s]: %v", f.FieldCode, err)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
execTime := time.Since(start).Milliseconds()
|
||||
logger.Infof("统计宽表 %s 创建完成，字段数: %d，耗时: %dms", tableName, len(fields), execTime)
|
||||
|
||||
return &model.AutoCreateStatTableResp{
|
||||
Success: true,
|
||||
TableName: tableName,
|
||||
ColumnCount: len(fields),
|
||||
ExecTimeMs: execTime,
|
||||
}, nil
|
||||
}
|
||||
|
||||
// buildCreateTableSQL 构建建表SQL
|
||||
func (c *StatTableCreator) buildCreateTableSQL(tableName string, report *model.ReportConfig, fields []model.FieldConfig) string {
|
||||
var cols []string
|
||||
|
||||
// 基础字段
|
||||
cols = append(cols, "id BIGSERIAL PRIMARY KEY")
|
||||
cols = append(cols, "tenant_id BIGINT NOT NULL DEFAULT 0")
|
||||
cols = append(cols, "business_code VARCHAR(64) NOT NULL DEFAULT ''")
|
||||
cols = append(cols, "creator VARCHAR(64) DEFAULT ''")
|
||||
cols = append(cols, "created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW()")
|
||||
cols = append(cols, "updater VARCHAR(64) DEFAULT ''")
|
||||
cols = append(cols, "updated_at TIMESTAMP WITH TIME ZONE DEFAULT NOW()")
|
||||
cols = append(cols, "deleted_at TIMESTAMP WITH TIME ZONE")
|
||||
|
||||
// 日期维度字段
|
||||
dateField := report.DateField
|
||||
if dateField == "" {
|
||||
dateField = "stat_date"
|
||||
}
|
||||
cols = append(cols, fmt.Sprintf("%s VARCHAR(16) NOT NULL DEFAULT ''", dateField))
|
||||
|
||||
// 业务字段
|
||||
for _, f := range fields {
|
||||
colDef := c.fieldTypeToColumn(f.FieldCode, f.FieldType)
|
||||
cols = append(cols, colDef)
|
||||
}
|
||||
|
||||
// 原始数据
|
||||
cols = append(cols, "raw_data JSONB DEFAULT '{}'")
|
||||
|
||||
sql := fmt.Sprintf("CREATE TABLE IF NOT EXISTS %s (\n %s\n)", tableName, strings.Join(cols, ",\n "))
|
||||
|
||||
return sql
|
||||
}
|
||||
|
||||
// fieldTypeToColumn 字段类型转PG列类型
|
||||
func (c *StatTableCreator) fieldTypeToColumn(fieldCode, fieldType string) string {
|
||||
switch fieldType {
|
||||
case model.FieldTypeInt, model.FieldTypeFloat:
|
||||
return fmt.Sprintf("%s NUMERIC(20,4) DEFAULT 0", fieldCode)
|
||||
case model.FieldTypeDate:
|
||||
return fmt.Sprintf("%s VARCHAR(16) DEFAULT ''", fieldCode)
|
||||
case model.FieldTypeDatetime:
|
||||
return fmt.Sprintf("%s TIMESTAMP WITH TIME ZONE", fieldCode)
|
||||
case model.FieldTypeJsonb:
|
||||
return fmt.Sprintf("%s JSONB DEFAULT '{}'", fieldCode)
|
||||
default: // STRING
|
||||
return fmt.Sprintf("%s VARCHAR(256) DEFAULT ''", fieldCode)
|
||||
}
|
||||
}
|
||||
|
||||
// DropStatTable 删除统计宽表
|
||||
func (c *StatTableCreator) DropStatTable(ctx context.Context, businessCode, reportCode string) error {
|
||||
report, err := c.loader.GetReport(ctx, businessCode, reportCode)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
sql := fmt.Sprintf("DROP TABLE IF EXISTS %s CASCADE", report.StatTableName)
|
||||
_, err = gfdb.DB(ctx).Exec(ctx, sql)
|
||||
return err
|
||||
}
|
||||
|
||||
// TableExists 检查表是否存在
|
||||
func (c *StatTableCreator) TableExists(ctx context.Context, tableName string) (bool, error) {
|
||||
result, err := gfdb.DB(ctx).GetAll(ctx, "SELECT COUNT(*) FROM pg_tables WHERE tablename = $1", strings.ToLower(tableName))
|
||||
if err != nil {
|
||||
return false, err
|
||||
}
|
||||
if len(result) == 0 {
|
||||
return false, nil
|
||||
}
|
||||
count := result[0]["count"].Int()
|
||||
return count > 0, nil
|
||||
}
|
||||
|
||||
// GetTableColumns 获取表字段列表
|
||||
func (c *StatTableCreator) GetTableColumns(ctx context.Context, tableName string) ([]string, error) {
|
||||
var columns []string
|
||||
sql := `SELECT column_name FROM information_schema.columns WHERE table_name = $1 AND table_schema = 'public' ORDER BY ordinal_position`
|
||||
rows, err := gfdb.DB(ctx).GetAll(ctx, sql, strings.ToLower(tableName))
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
for _, row := range rows.List() {
|
||||
if col, ok := row["column_name"].(string); ok {
|
||||
columns = append(columns, col)
|
||||
}
|
||||
}
|
||||
return columns, nil
|
||||
}
|
||||
|
||||
// AlterTableAddColumns 为已存在的表添加新字段
|
||||
func (c *StatTableCreator) AlterTableAddColumns(ctx context.Context, tableName string, newFields []model.FieldConfig) error {
|
||||
existingCols, err := c.GetTableColumns(ctx, tableName)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
existingMap := make(map[string]bool)
|
||||
for _, col := range existingCols {
|
||||
existingMap[col] = true
|
||||
}
|
||||
|
||||
for _, f := range newFields {
|
||||
if existingMap[f.FieldCode] {
|
||||
continue
|
||||
}
|
||||
colDef := c.fieldTypeToColumn(f.FieldCode, f.FieldType)
|
||||
sql := fmt.Sprintf("ALTER TABLE %s ADD COLUMN IF NOT EXISTS %s", tableName, colDef)
|
||||
if _, err := gfdb.DB(ctx).Exec(ctx, sql); err != nil {
|
||||
return fmt.Errorf("添加字段 %s 失败: %w", f.FieldCode, err)
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
@@ -0,0 +1,666 @@
|
||||
package report
|
||||
|
||||
// ============================================================
|
||||
// 通用报表引擎 - 完整调用示例
|
||||
// ============================================================
|
||||
//
|
||||
// 包名: dataengine/common/report
|
||||
// 入口: report.GetService() → *ReportService
|
||||
//
|
||||
// 接口一览:
|
||||
// ┌──────────────┬─────────────────────────────────────────────────┐
|
||||
// │ 分类 │ 接口 │
|
||||
// ├──────────────┼─────────────────────────────────────────────────┤
|
||||
// │ 配置 CRUD │ SaveBusiness / DelBusiness / GetBusiness │
|
||||
// │ │ SaveReport / DelReport / GetReport │
|
||||
// │ │ SaveField / DelField / GetField │
|
||||
// │ │ SaveExtractConfig / DelExtractConfig / Get.. │
|
||||
// ├──────────────┼─────────────────────────────────────────────────┤
|
||||
// │ 数据抽取 │ ExtractDailyData / AutoCreateStatTable │
|
||||
// ├──────────────┼─────────────────────────────────────────────────┤
|
||||
// │ 报表查询 │ QueryReportByUserSelect │
|
||||
// ├──────────────┼─────────────────────────────────────────────────┤
|
||||
// │ 辅助查询 │ GetAllBusinesses / GetAllReports │
|
||||
// │ │ GetReportFields / GetExtractConfigs │
|
||||
// └──────────────┴─────────────────────────────────────────────────┘
|
||||
//
|
||||
// ============================================================
|
||||
|
||||
// ============================================================================
|
||||
// 场景一：新平台零代码接入（快手电商为例）
|
||||
// ============================================================================
|
||||
//
|
||||
// 背景：快手订单数据已通过数据引擎同步到 kuaishou_order_list 表（每条订单一行）。
|
||||
// 需求：按店铺+天聚合订单数据 → 自动建统计宽表 → 抽取 → 前端自由查询报表。
|
||||
//
|
||||
// 全程只调 API 不写 SQL，前端管理后台可直接操作。
|
||||
|
||||
/*
|
||||
package example
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"time"
|
||||
|
||||
"dataengine/common/report"
|
||||
"dataengine/common/report/model"
|
||||
)
|
||||
|
||||
func KuaishouExample() {
|
||||
ctx := context.Background()
|
||||
svc := report.GetService()
|
||||
|
||||
// ─── Step 1: 注册业务 ───────────────────────────────────────────
|
||||
// 一个业务就是一个数据源平台（快手/抖音/淘宝/...）
|
||||
_, _ = svc.SaveBusiness(ctx, &model.SaveBusinessReq{
|
||||
BusinessCode: "KUAISHOU", // 唯一标识，后续所有接口都用它
|
||||
BusinessName: "快手电商",
|
||||
Description: "快手平台电商业务线",
|
||||
Status: model.StatusActive,
|
||||
Operator: "admin",
|
||||
})
|
||||
|
||||
// ─── Step 2: 注册报表 ───────────────────────────────────────────
|
||||
// 一个业务可以有多个报表（店铺日报、商品日报、主播日报...）
|
||||
_, _ = svc.SaveReport(ctx, &model.SaveReportReq{
|
||||
BusinessCode: "KUAISHOU",
|
||||
ReportCode: "shop_daily_report", // 报表唯一编码
|
||||
ReportName: "快手店铺日报",
|
||||
StatTableName: "stat_kuaishou_shop_daily", // 统计宽表名（自动创建）
|
||||
DateField: "stat_date", // 日期字段
|
||||
ConflictKeys: []string{"shop_id", "stat_date"}, // 唯一约束（upsert 依据）
|
||||
Operator: "admin",
|
||||
})
|
||||
|
||||
// ─── Step 3: 配置字段（前端可选择的所有维度/指标/筛选） ─────────
|
||||
// 这是前端"自定义报表"的数据源 —— 前端建好字段后，
|
||||
// 用户选择哪些维度、哪些指标、怎么筛选，实时查。
|
||||
|
||||
// 3a. 维度字段：分组依据，不可聚合
|
||||
dimensions := []model.SaveFieldReq{
|
||||
{BusinessCode: "KUAISHOU", ReportCode: "shop_daily_report",
|
||||
FieldCode: "shop_id", FieldName: "店铺ID", FieldType: "STRING",
|
||||
FieldRole: "DIMENSION", IsSortable: true, SortOrder: 1, GroupName: "店铺", Operator: "admin"},
|
||||
{BusinessCode: "KUAISHOU", ReportCode: "shop_daily_report",
|
||||
FieldCode: "shop_name", FieldName: "店铺名称", FieldType: "STRING",
|
||||
FieldRole: "DIMENSION", IsSortable: true, SortOrder: 2, GroupName: "店铺", Operator: "admin"},
|
||||
{BusinessCode: "KUAISHOU", ReportCode: "shop_daily_report",
|
||||
FieldCode: "stat_date", FieldName: "统计日期", FieldType: "DATE",
|
||||
FieldRole: "DIMENSION", IsSortable: true, SortOrder: 3, GroupName: "时间", Operator: "admin"},
|
||||
}
|
||||
|
||||
// 3b. 指标字段：聚合度量，前端可选 SUM/COUNT/AVG/MAX/MIN
|
||||
indicators := []model.SaveFieldReq{
|
||||
{BusinessCode: "KUAISHOU", ReportCode: "shop_daily_report",
|
||||
FieldCode: "order_count", FieldName: "订单数", FieldType: "INT",
|
||||
FieldRole: "INDICATOR", IsAggregatable: true, DefaultAggregate: "SUM",
|
||||
ValidAggregates: []string{"SUM", "COUNT", "AVG", "MAX", "MIN"},
|
||||
SortOrder: 10, GroupName: "订单", Operator: "admin"},
|
||||
{BusinessCode: "KUAISHOU", ReportCode: "shop_daily_report",
|
||||
FieldCode: "order_amount", FieldName: "订单金额(元)", FieldType: "FLOAT",
|
||||
FieldRole: "INDICATOR", IsAggregatable: true, DefaultAggregate: "SUM",
|
||||
ValidAggregates: []string{"SUM", "AVG", "MAX", "MIN"},
|
||||
SortOrder: 11, GroupName: "金额", Unit: "元", Operator: "admin"},
|
||||
{BusinessCode: "KUAISHOU", ReportCode: "shop_daily_report",
|
||||
FieldCode: "paid_amount", FieldName: "实付金额(元)", FieldType: "FLOAT",
|
||||
FieldRole: "INDICATOR", IsAggregatable: true, DefaultAggregate: "SUM",
|
||||
ValidAggregates: []string{"SUM", "AVG"},
|
||||
SortOrder: 12, GroupName: "金额", Unit: "元", Operator: "admin"},
|
||||
{BusinessCode: "KUAISHOU", ReportCode: "shop_daily_report",
|
||||
FieldCode: "refund_amount", FieldName: "退款金额(元)", FieldType: "FLOAT",
|
||||
FieldRole: "INDICATOR", IsAggregatable: true, DefaultAggregate: "SUM",
|
||||
ValidAggregates: []string{"SUM", "AVG"},
|
||||
SortOrder: 13, GroupName: "退款", Unit: "元", Operator: "admin"},
|
||||
{BusinessCode: "KUAISHOU", ReportCode: "shop_daily_report",
|
||||
FieldCode: "buyer_count", FieldName: "下单买家数", FieldType: "INT",
|
||||
FieldRole: "INDICATOR", IsAggregatable: true, DefaultAggregate: "COUNT",
|
||||
ValidAggregates: []string{"COUNT"},
|
||||
SortOrder: 14, GroupName: "用户", Operator: "admin"},
|
||||
// 衍生指标：退款率 = 退款金额 / 订单金额 * 100
|
||||
{BusinessCode: "KUAISHOU", ReportCode: "shop_daily_report",
|
||||
FieldCode: "refund_rate", FieldName: "退款率", FieldType: "FLOAT",
|
||||
FieldRole: "INDICATOR", IsAggregatable: true, DefaultAggregate: "AVG",
|
||||
Expression: "{refund_amount} / NULLIF({order_amount}, 0) * 100",
|
||||
ExpressionType: "CALCULATED", FormatPattern: "#,##0.00",
|
||||
Unit: "%", SortOrder: 20, GroupName: "退款", Operator: "admin"},
|
||||
}
|
||||
|
||||
// 3c. 筛选字段：纯筛选，不出现在 SELECT 中
|
||||
filters := []model.SaveFieldReq{
|
||||
{BusinessCode: "KUAISHOU", ReportCode: "shop_daily_report",
|
||||
FieldCode: "order_status", FieldName: "订单状态", FieldType: "STRING",
|
||||
FieldRole: "FILTER", IsFilterable: true,
|
||||
FilterOperators: []string{"=", "IN"},
|
||||
SortOrder: 30, GroupName: "筛选", Operator: "admin"},
|
||||
}
|
||||
|
||||
for _, f := range dimensions {
|
||||
_, _ = svc.SaveField(ctx, &f)
|
||||
}
|
||||
for _, f := range indicators {
|
||||
_, _ = svc.SaveField(ctx, &f)
|
||||
}
|
||||
for _, f := range filters {
|
||||
_, _ = svc.SaveField(ctx, &f)
|
||||
}
|
||||
|
||||
// ─── Step 4: 配置数据抽取规则 ────────────────────────────────────
|
||||
// 关键：AGGREGATE 模式，从源表聚合到统计宽表
|
||||
_, _ = svc.SaveExtractConfig(ctx, &model.SaveExtractConfigReq{
|
||||
BusinessCode: "KUAISHOU",
|
||||
ReportCode: "shop_daily_report",
|
||||
ExtractCode: "extract_shop_daily",
|
||||
ExtractName: "快手店铺订单按天聚合",
|
||||
SourceTableName: "kuaishou_order_list", // ← 源表（订单明细）
|
||||
SourceTableAlias: "o",
|
||||
TargetTableName: "stat_kuaishou_shop_daily", // ← 目标（统计宽表）
|
||||
ExtractType: model.ExtractTypeIncremental, // 增量抽取
|
||||
ExtractMode: model.ExtractModeAggregate, // ← 聚合模式
|
||||
ExtractKeyField: "created_at", // 增量依据字段
|
||||
GroupByFields: []string{"shop_id"}, // ← GROUP BY
|
||||
FilterExpression: "o.order_status != 'CANCELLED'", // 过滤取消订单
|
||||
// 字段映射：源表字段 → 目标表字段 + 聚合函数
|
||||
FieldMappings: []model.FieldMapping{
|
||||
{SourceField: "shop_id", TargetField: "shop_id", FieldType: "STRING"},
|
||||
{SourceField: "shop_name", TargetField: "shop_name", FieldType: "STRING"},
|
||||
{SourceField: "id", TargetField: "order_count", FieldType: "INT", AggregateFunction: "COUNT"},
|
||||
{SourceField: "order_amount", TargetField: "order_amount", FieldType: "FLOAT", AggregateFunction: "SUM"},
|
||||
{SourceField: "paid_amount", TargetField: "paid_amount", FieldType: "FLOAT", AggregateFunction: "SUM"},
|
||||
{SourceField: "refund_amount",TargetField: "refund_amount",FieldType: "FLOAT", AggregateFunction: "SUM"},
|
||||
{SourceField: "buyer_id", TargetField: "buyer_count", FieldType: "INT", AggregateFunction: "COUNT"},
|
||||
},
|
||||
BatchSize: 1000,
|
||||
Operator: "admin",
|
||||
})
|
||||
|
||||
// ─── Step 5: 每天定时抽取（cron/k8s CronJob 中调用） ─────────
|
||||
// ExtractDailyData 内部：
|
||||
// 1. 检测 stat_kuaishou_shop_daily 表是否存在
|
||||
// 2. 不存在 → 根据 FieldConfig 自动 CREATE TABLE
|
||||
// 3. 从 kuaishou_order_list 按 AGGREGATE 模式抽取当天数据
|
||||
// 4. UPSERT 到统计宽表
|
||||
today := time.Now().Format("2006-01-02")
|
||||
resp, _ := svc.ExtractDailyData(ctx, "KUAISHOU", "shop_daily_report", today, "cron")
|
||||
fmt.Printf("[%s] 抽取完成: 总%d 成功%d 失败%d 耗时%dms\n",
|
||||
today, resp.TotalCount, resp.SuccessCount, resp.FailCount, resp.ExecTimeMs)
|
||||
|
||||
// 实际抽取生成的 SQL（AGGREGATE 模式）:
|
||||
//
|
||||
// INSERT INTO stat_kuaishou_shop_daily (...)
|
||||
// SELECT ROW_NUMBER() OVER () AS id,
|
||||
// '2026-06-10' AS stat_date,
|
||||
// o.shop_id, o.shop_name,
|
||||
// COUNT(o.id) AS order_count,
|
||||
// SUM(o.order_amount) AS order_amount,
|
||||
// SUM(o.paid_amount) AS paid_amount,
|
||||
// SUM(o.refund_amount) AS refund_amount,
|
||||
// COUNT(o.buyer_id) AS buyer_count
|
||||
// FROM kuaishou_order_list o
|
||||
// WHERE o.created_at::date = '2026-06-10'
|
||||
// GROUP BY o.shop_id
|
||||
|
||||
// ─── Step 6: 前端查询 ──────────────────────────────────────────
|
||||
|
||||
// 6a. 前端先拉取可用字段列表
|
||||
fields, _ := svc.GetReportFields(ctx, "KUAISHOU", "shop_daily_report")
|
||||
// 返回:
|
||||
// dimensions: [shop_id, shop_name, stat_date]
|
||||
// indicators: [order_count, order_amount, paid_amount, refund_amount, buyer_count, refund_rate]
|
||||
// filters: [order_status]
|
||||
// 前端据此渲染选择器面板
|
||||
|
||||
// 6b. 用户选择: 维度=店铺, 指标=金额+订单数, 时间=近7天, 排名=TOP10
|
||||
top10Req := &model.UserSelectQueryReq{
|
||||
BusinessCode: "KUAISHOU",
|
||||
ReportCode: "shop_daily_report",
|
||||
Dimensions: []string{"shop_id", "shop_name"},
|
||||
Indicators: []model.IndicatorSelect{
|
||||
{FieldCode: "order_amount", Aggregate: "SUM", Alias: "total_amount"},
|
||||
{FieldCode: "order_count", Aggregate: "SUM", Alias: "total_orders"},
|
||||
{FieldCode: "paid_amount", Aggregate: "SUM", Alias: "total_paid"},
|
||||
{FieldCode: "refund_rate", Aggregate: "AVG", Alias: "avg_refund_rate"},
|
||||
},
|
||||
TimeRange: &model.TimeRange{
|
||||
StartDate: time.Now().AddDate(0, 0, -7).Format("2006-01-02"),
|
||||
EndDate: today,
|
||||
},
|
||||
OrderBy: []model.OrderCondition{{FieldCode: "total_amount", Direction: "DESC"}},
|
||||
Page: 1,
|
||||
PageSize: 10,
|
||||
}
|
||||
top10Resp, _ := svc.QueryReportByUserSelect(ctx, top10Req)
|
||||
fmt.Printf("TOP10 销售额排行 (总数=%d):\n", top10Resp.Total)
|
||||
for i, row := range top10Resp.List {
|
||||
b, _ := json.Marshal(row)
|
||||
fmt.Printf(" %d. %s\n", i+1, string(b))
|
||||
}
|
||||
// 实际生成 SQL:
|
||||
// SELECT o.shop_id, o.shop_name,
|
||||
// SUM(o.order_amount) AS total_amount,
|
||||
// SUM(o.order_count) AS total_orders,
|
||||
// SUM(o.paid_amount) AS total_paid,
|
||||
// AVG(refund_amount / NULLIF(order_amount,0) * 100) AS avg_refund_rate
|
||||
// FROM stat_kuaishou_shop_daily o
|
||||
// WHERE stat_date BETWEEN '2026-06-03' AND '2026-06-10'
|
||||
// GROUP BY o.shop_id, o.shop_name
|
||||
// ORDER BY total_amount DESC
|
||||
// LIMIT 10 OFFSET 0
|
||||
|
||||
// 6c. 用户切换维度: 每日趋势（聚合所有店铺）
|
||||
trendReq := &model.UserSelectQueryReq{
|
||||
BusinessCode: "KUAISHOU",
|
||||
ReportCode: "shop_daily_report",
|
||||
Dimensions: []string{"stat_date"},
|
||||
Indicators: []model.IndicatorSelect{
|
||||
{FieldCode: "order_amount", Aggregate: "SUM", Alias: "daily_amount"},
|
||||
{FieldCode: "order_count", Aggregate: "SUM", Alias: "daily_orders"},
|
||||
{FieldCode: "refund_amount",Aggregate: "SUM", Alias: "daily_refund"},
|
||||
},
|
||||
TimeRange: &model.TimeRange{
|
||||
StartDate: time.Now().AddDate(0, 0, -30).Format("2006-01-02"),
|
||||
EndDate: today,
|
||||
},
|
||||
TimeGroup: "day",
|
||||
OrderBy: []model.OrderCondition{{FieldCode: "stat_date", Direction: "ASC"}},
|
||||
PageSize: 100,
|
||||
}
|
||||
trendResp, _ := svc.QueryReportByUserSelect(ctx, trendReq)
|
||||
fmt.Printf("30天趋势 (共%d行):\n", trendResp.Total)
|
||||
|
||||
// 6d. 加筛选条件: 只看活跃订单，金额>10000
|
||||
filteredReq := &model.UserSelectQueryReq{
|
||||
BusinessCode: "KUAISHOU",
|
||||
ReportCode: "shop_daily_report",
|
||||
Dimensions: []string{"shop_id", "shop_name"},
|
||||
Indicators: []model.IndicatorSelect{
|
||||
{FieldCode: "order_amount", Aggregate: "SUM", Alias: "total_amount"},
|
||||
},
|
||||
Filters: []model.FilterCondition{
|
||||
{FieldCode: "order_status", Operator: "=", Value: "ACTIVE"},
|
||||
{FieldCode: "total_amount", Operator: ">=", Value: 10000}, // 指标别名也可筛选
|
||||
},
|
||||
OrderBy: []model.OrderCondition{{FieldCode: "total_amount", Direction: "DESC"}},
|
||||
PageSize: 20,
|
||||
}
|
||||
_ = filteredReq
|
||||
|
||||
// 6e. 按周汇总趋势
|
||||
weeklyReq := &model.UserSelectQueryReq{
|
||||
BusinessCode: "KUAISHOU",
|
||||
ReportCode: "shop_daily_report",
|
||||
Dimensions: []string{"shop_id"},
|
||||
Indicators: []model.IndicatorSelect{
|
||||
{FieldCode: "order_amount", Aggregate: "SUM", Alias: "weekly_amount"},
|
||||
},
|
||||
TimeGroup: "week",
|
||||
OrderBy: []model.OrderCondition{{FieldCode: "weekly_amount", Direction: "DESC"}},
|
||||
PageSize: 50,
|
||||
}
|
||||
_ = weeklyReq
|
||||
}
|
||||
|
||||
|
||||
// ============================================================================
|
||||
// 场景二：已有配置管理（CRUD 二次开发参考）
|
||||
// ============================================================================
|
||||
|
||||
func CRUDExample() {
|
||||
ctx := context.Background()
|
||||
svc := report.GetService()
|
||||
|
||||
// ── 业务 CRUD ────────────────────────────────────────────
|
||||
|
||||
// 新增
|
||||
result, _ := svc.SaveBusiness(ctx, &model.SaveBusinessReq{
|
||||
BusinessCode: "DOUYIN", BusinessName: "抖音电商",
|
||||
Operator: "admin",
|
||||
})
|
||||
businessId := result.ID
|
||||
|
||||
// 修改（传 ID 即修改）
|
||||
result, _ = svc.SaveBusiness(ctx, &model.SaveBusinessReq{
|
||||
ID: &businessId, BusinessCode: "DOUYIN",
|
||||
BusinessName: "抖音电商（新版）", Operator: "admin",
|
||||
})
|
||||
|
||||
// 查询
|
||||
biz, _ := svc.GetBusiness(ctx, businessId)
|
||||
fmt.Printf("%s: %s\n", biz.BusinessCode, biz.BusinessName)
|
||||
|
||||
// 删除
|
||||
svc.DeleteBusiness(ctx, businessId)
|
||||
|
||||
// 全部列表
|
||||
allBiz, _ := svc.GetAllBusinesses(ctx)
|
||||
for _, b := range allBiz {
|
||||
fmt.Printf("- %s (%s)\n", b.BusinessCode, b.BusinessName)
|
||||
}
|
||||
|
||||
// ── 报表 CRUD ────────────────────────────────────────────
|
||||
|
||||
reportResult, _ := svc.SaveReport(ctx, &model.SaveReportReq{
|
||||
BusinessCode: "DOUYIN",
|
||||
ReportCode: "shop_daily_report",
|
||||
ReportName: "抖音店铺日报",
|
||||
StatTableName: "stat_douyin_shop_daily",
|
||||
ConflictKeys: []string{"shop_id", "stat_date"},
|
||||
Operator: "admin",
|
||||
})
|
||||
reportId := reportResult.ID
|
||||
|
||||
rpt, _ := svc.GetReport(ctx, reportId)
|
||||
svc.DeleteReport(ctx, reportId)
|
||||
_ = rpt
|
||||
|
||||
// ── 字段 CRUD ────────────────────────────────────────────
|
||||
|
||||
// 新增字段（id 不传 = 新增）
|
||||
fieldResult, _ := svc.SaveField(ctx, &model.SaveFieldReq{
|
||||
BusinessCode: "DOUYIN", ReportCode: "shop_daily_report",
|
||||
FieldCode: "order_amount", FieldName: "订单金额",
|
||||
FieldType: "FLOAT", FieldRole: "INDICATOR",
|
||||
IsAggregatable: true, DefaultAggregate: "SUM",
|
||||
ValidAggregates: []string{"SUM", "AVG", "MAX", "MIN"},
|
||||
SortOrder: 10, GroupName: "金额", Operator: "admin",
|
||||
})
|
||||
fieldId := fieldResult.ID
|
||||
|
||||
// 修改字段（传 id = 更新）
|
||||
svc.SaveField(ctx, &model.SaveFieldReq{
|
||||
ID: &fieldId,
|
||||
FieldName: "订单金额（元）",
|
||||
Operator: "admin",
|
||||
// ... 只传要修改的字段，未传的保持原值
|
||||
})
|
||||
|
||||
f, _ := svc.GetField(ctx, fieldId)
|
||||
svc.DeleteField(ctx, fieldId)
|
||||
_ = f
|
||||
|
||||
// ── 抽取配置 CRUD ────────────────────────────────────────
|
||||
|
||||
ecResult, _ := svc.SaveExtractConfig(ctx, &model.SaveExtractConfigReq{
|
||||
BusinessCode: "DOUYIN", ReportCode: "shop_daily_report",
|
||||
ExtractCode: "extract_daily", ExtractName: "按天聚合抽取",
|
||||
SourceTableName: "douyin_order_list", SourceTableAlias: "o",
|
||||
TargetTableName: "stat_douyin_shop_daily",
|
||||
ExtractMode: "AGGREGATE",
|
||||
ExtractKeyField: "created_at",
|
||||
GroupByFields: []string{"shop_id"},
|
||||
FieldMappings: []model.FieldMapping{
|
||||
{SourceField: "id", TargetField: "order_count", FieldType: "INT", AggregateFunction: "COUNT"},
|
||||
{SourceField: "order_amount", TargetField: "order_amount", FieldType: "FLOAT", AggregateFunction: "SUM"},
|
||||
},
|
||||
Operator: "admin",
|
||||
})
|
||||
ecId := ecResult.ID
|
||||
|
||||
ec, _ := svc.GetExtractConfig(ctx, ecId)
|
||||
allEc, _ := svc.GetExtractConfigs(ctx, "DOUYIN", "shop_daily_report")
|
||||
svc.DeleteExtractConfig(ctx, ecId)
|
||||
_, _ = ec, allEc
|
||||
}
|
||||
|
||||
|
||||
// ============================================================================
|
||||
// 场景三：任意平台接入的通用模式（零硬编码）
|
||||
// ============================================================================
|
||||
//
|
||||
// 假设要接入淘宝平台，taobao_order_list 表已有数据。
|
||||
// 全程不写任何代码，只需调 CRUD API。
|
||||
|
||||
func GenericPlatformExample() {
|
||||
ctx := context.Background()
|
||||
svc := report.GetService()
|
||||
|
||||
// 1. 前端注册业务
|
||||
svc.SaveBusiness(ctx, &model.SaveBusinessReq{
|
||||
BusinessCode: "TAOBAO", BusinessName: "淘宝电商",
|
||||
Operator: "admin",
|
||||
})
|
||||
|
||||
// 2. 注册报表 + 指定统计宽表名（以后表名不再手工管理）
|
||||
svc.SaveReport(ctx, &model.SaveReportReq{
|
||||
BusinessCode: "TAOBAO",
|
||||
ReportCode: "shop_daily_report",
|
||||
ReportName: "淘宝店铺日报",
|
||||
StatTableName: "stat_taobao_shop_daily",
|
||||
ConflictKeys: []string{"shop_id", "stat_date"},
|
||||
Operator: "admin",
|
||||
})
|
||||
|
||||
// 3. 前端选择统计维度（自由组合，随时增删改）
|
||||
svc.SaveField(ctx, &model.SaveFieldReq{
|
||||
BusinessCode: "TAOBAO", ReportCode: "shop_daily_report",
|
||||
FieldCode: "shop_id", FieldName: "店铺ID",
|
||||
FieldType: "STRING", FieldRole: "DIMENSION",
|
||||
SortOrder: 1, GroupName: "店铺", Operator: "admin",
|
||||
})
|
||||
svc.SaveField(ctx, &model.SaveFieldReq{
|
||||
BusinessCode: "TAOBAO", ReportCode: "shop_daily_report",
|
||||
FieldCode: "shop_name", FieldName: "店铺名称",
|
||||
FieldType: "STRING", FieldRole: "DIMENSION",
|
||||
SortOrder: 2, GroupName: "店铺", Operator: "admin",
|
||||
})
|
||||
svc.SaveField(ctx, &model.SaveFieldReq{
|
||||
BusinessCode: "TAOBAO", ReportCode: "shop_daily_report",
|
||||
FieldCode: "stat_date", FieldName: "统计日期",
|
||||
FieldType: "DATE", FieldRole: "DIMENSION",
|
||||
SortOrder: 3, GroupName: "时间", Operator: "admin",
|
||||
})
|
||||
|
||||
// 4. 前端选择统计指标（自由组合，随时增删改）
|
||||
svc.SaveField(ctx, &model.SaveFieldReq{
|
||||
BusinessCode: "TAOBAO", ReportCode: "shop_daily_report",
|
||||
FieldCode: "order_count", FieldName: "订单数",
|
||||
FieldType: "INT", FieldRole: "INDICATOR",
|
||||
IsAggregatable: true, DefaultAggregate: "SUM",
|
||||
ValidAggregates: []string{"SUM", "COUNT", "AVG"},
|
||||
SortOrder: 10, GroupName: "订单", Operator: "admin",
|
||||
})
|
||||
svc.SaveField(ctx, &model.SaveFieldReq{
|
||||
BusinessCode: "TAOBAO", ReportCode: "shop_daily_report",
|
||||
FieldCode: "order_amount", FieldName: "订单金额",
|
||||
FieldType: "FLOAT", FieldRole: "INDICATOR",
|
||||
IsAggregatable: true, DefaultAggregate: "SUM",
|
||||
ValidAggregates: []string{"SUM", "AVG", "MAX", "MIN"},
|
||||
SortOrder: 11, GroupName: "金额", Unit: "元", Operator: "admin",
|
||||
})
|
||||
svc.SaveField(ctx, &model.SaveFieldReq{
|
||||
BusinessCode: "TAOBAO", ReportCode: "shop_daily_report",
|
||||
FieldCode: "buyer_count", FieldName: "下单买家数",
|
||||
FieldType: "INT", FieldRole: "INDICATOR",
|
||||
IsAggregatable: true, DefaultAggregate: "COUNT",
|
||||
ValidAggregates: []string{"COUNT"},
|
||||
SortOrder: 12, GroupName: "用户", Operator: "admin",
|
||||
})
|
||||
|
||||
// 5. 配置抽取规则
|
||||
svc.SaveExtractConfig(ctx, &model.SaveExtractConfigReq{
|
||||
BusinessCode: "TAOBAO", ReportCode: "shop_daily_report",
|
||||
ExtractCode: "extract_daily", ExtractName: "淘宝按天聚合",
|
||||
SourceTableName: "taobao_order_list", SourceTableAlias: "o",
|
||||
TargetTableName: "stat_taobao_shop_daily",
|
||||
ExtractMode: "AGGREGATE",
|
||||
ExtractKeyField: "created_at",
|
||||
GroupByFields: []string{"shop_id"},
|
||||
FieldMappings: []model.FieldMapping{
|
||||
{SourceField: "shop_id", TargetField: "shop_id", FieldType: "STRING"},
|
||||
{SourceField: "shop_name", TargetField: "shop_name", FieldType: "STRING"},
|
||||
{SourceField: "id", TargetField: "order_count", FieldType: "INT", AggregateFunction: "COUNT"},
|
||||
{SourceField: "order_amount", TargetField: "order_amount", FieldType: "FLOAT", AggregateFunction: "SUM"},
|
||||
{SourceField: "buyer_id", TargetField: "buyer_count", FieldType: "INT", AggregateFunction: "COUNT"},
|
||||
},
|
||||
BatchSize: 1000,
|
||||
Operator: "admin",
|
||||
})
|
||||
|
||||
// 6. 定时任务每天执行
|
||||
svc.ExtractDailyData(ctx, "TAOBAO", "shop_daily_report", "2026-06-10", "cron")
|
||||
|
||||
// 7. 前端实时查询
|
||||
req := &model.UserSelectQueryReq{
|
||||
BusinessCode: "TAOBAO", ReportCode: "shop_daily_report",
|
||||
Dimensions: []string{"shop_id", "shop_name"},
|
||||
Indicators: []model.IndicatorSelect{
|
||||
{FieldCode: "order_amount", Aggregate: "SUM", Alias: "total"},
|
||||
{FieldCode: "order_count", Aggregate: "SUM", Alias: "orders"},
|
||||
},
|
||||
TimeRange: &model.TimeRange{StartDate: "2026-06-01", EndDate: "2026-06-10"},
|
||||
Page: 1, PageSize: 20,
|
||||
}
|
||||
resp, _ := svc.QueryReportByUserSelect(ctx, req)
|
||||
fmt.Printf("查询结果: 共%d条, 耗时%dms\n", resp.Total, resp.ExecTimeMs)
|
||||
|
||||
// 平台接入完毕。全程零代码改动。
|
||||
}
|
||||
|
||||
|
||||
// ============================================================================
|
||||
// 场景四：在外部业务服务（如 goview-report）中调用
|
||||
// ============================================================================
|
||||
|
||||
func ExternalServiceExample() {
|
||||
ctx := context.Background()
|
||||
svc := report.GetService()
|
||||
|
||||
// 直接用，不需要初始化，内部自动懒加载和建表。
|
||||
today := time.Now().Format("2006-01-02")
|
||||
|
||||
// 按天抽取
|
||||
svc.ExtractDailyData(ctx, "KUAISHOU", "shop_daily_report", today, "cron")
|
||||
|
||||
// 实时查询
|
||||
svc.QueryReportByUserSelect(ctx, &model.UserSelectQueryReq{
|
||||
BusinessCode: "KUAISHOU", ReportCode: "shop_daily_report",
|
||||
Dimensions: []string{"shop_id", "shop_name"},
|
||||
Indicators: []model.IndicatorSelect{
|
||||
{FieldCode: "order_amount", Aggregate: "SUM", Alias: "total_amount"},
|
||||
},
|
||||
TimeRange: &model.TimeRange{StartDate: "2026-06-01", EndDate: today},
|
||||
Page: 1, PageSize: 20,
|
||||
})
|
||||
}
|
||||
|
||||
|
||||
// ============================================================================
|
||||
// 场景五：Direct 模式（逐行抽取，不做聚合）
|
||||
// ============================================================================
|
||||
//
|
||||
// 适用于源表已经是一行一条统计记录的场景（如已预聚合的报表源表）。
|
||||
|
||||
func DirectModeExample() {
|
||||
ctx := context.Background()
|
||||
svc := report.GetService()
|
||||
|
||||
svc.SaveBusiness(ctx, &model.SaveBusinessReq{
|
||||
BusinessCode: "HDWL", BusinessName: "HDWL业务", Operator: "admin",
|
||||
})
|
||||
svc.SaveReport(ctx, &model.SaveReportReq{
|
||||
BusinessCode: "HDWL", ReportCode: "shop_daily_stat",
|
||||
ReportName: "HDWL店铺日报", StatTableName: "stat_hdwl_shop_daily",
|
||||
ConflictKeys: []string{"report_date"},
|
||||
Operator: "admin",
|
||||
})
|
||||
|
||||
// Direct 模式：不聚合，逐行映射
|
||||
svc.SaveExtractConfig(ctx, &model.SaveExtractConfigReq{
|
||||
BusinessCode: "HDWL", ReportCode: "shop_daily_stat",
|
||||
ExtractCode: "extract_direct", ExtractName: "直接逐行抽取",
|
||||
SourceTableName: "hdwl_daily_summary", SourceTableAlias: "s",
|
||||
TargetTableName: "stat_hdwl_shop_daily",
|
||||
ExtractMode: "DIRECT", // ← DIRECT 模式
|
||||
ExtractKeyField: "report_date",
|
||||
FieldMappings: []model.FieldMapping{
|
||||
{SourceField: "shop_id", TargetField: "shop_id", FieldType: "STRING"},
|
||||
{SourceField: "shop_name", TargetField: "shop_name", FieldType: "STRING"},
|
||||
{SourceField: "total_sale", TargetField: "sale_amount", FieldType: "FLOAT"},
|
||||
{SourceField: "total_orders", TargetField: "order_count", FieldType: "INT"},
|
||||
},
|
||||
Operator: "admin",
|
||||
})
|
||||
|
||||
svc.ExtractDailyData(ctx, "HDWL", "shop_daily_stat", "2026-06-10", "cron")
|
||||
}
|
||||
|
||||
|
||||
// ============================================================================
|
||||
// 场景六：前端交互流程参考
|
||||
// ============================================================================
|
||||
//
|
||||
// 前端代码调用示例（伪代码，展示 API 调用顺序）：
|
||||
//
|
||||
// // 1. 页面加载 → 获取业务列表 → 渲染下拉框
|
||||
// GET /api/report/businesses
|
||||
// 返回: [{ businessCode: "KUAISHOU", businessName: "快手电商" }, ...]
|
||||
//
|
||||
// // 2. 用户选择业务 → 获取报表列表
|
||||
// GET /api/report/reports?businessCode=KUAISHOU
|
||||
// 返回: [{ reportCode: "shop_daily_report", reportName: "快手店铺日报" }, ...]
|
||||
//
|
||||
// // 3. 用户选择报表 → 获取可用字段 → 渲染维度/指标/筛选选择器
|
||||
// GET /api/report/fields?businessCode=KUAISHOU&reportCode=shop_daily_report
|
||||
// 返回: { dimensions: [...], indicators: [...], filters: [...] }
|
||||
//
|
||||
// // 4. 用户选好条件 → 查询
|
||||
// POST /api/report/query
|
||||
// Body: {
|
||||
// "businessCode": "KUAISHOU", "reportCode": "shop_daily_report",
|
||||
// "dimensions": ["shop_id", "shop_name"],
|
||||
// "indicators": [
|
||||
// {"fieldCode": "order_amount", "aggregate": "SUM", "alias": "total"},
|
||||
// {"fieldCode": "order_count", "aggregate": "SUM", "alias": "count"}
|
||||
// ],
|
||||
// "timeRange": {"startDate": "2026-06-01", "endDate": "2026-06-10"},
|
||||
// "orderBy": [{"fieldCode": "total", "direction": "DESC"}],
|
||||
// "page": 1, "pageSize": 20
|
||||
// }
|
||||
// 返回: { list: [...], total: 152, page: 1, totalPages: 8, execTimeMs: 45 }
|
||||
//
|
||||
// // 5. 翻页/换维度/换指标 → 重新调 Step 4
|
||||
//
|
||||
// 管理后台（用户可自行维护配置）：
|
||||
//
|
||||
// // 新增业务
|
||||
// POST /api/report/business/save
|
||||
// Body: { "businessCode": "TAOBAO", "businessName": "淘宝电商" }
|
||||
//
|
||||
// // 新增/修改字段（用户自定义统计维度）
|
||||
// POST /api/report/field/save
|
||||
// Body: { "businessCode": "TAOBAO", "reportCode": "shop_daily_report",
|
||||
// "fieldCode": "category", "fieldName": "商品类目",
|
||||
// "fieldType": "STRING", "fieldRole": "DIMENSION" }
|
||||
//
|
||||
// // 配置抽取规则
|
||||
// POST /api/report/extractConfig/save
|
||||
// Body: { "businessCode": "TAOBAO", "reportCode": "shop_daily_report",
|
||||
// "extractCode": "extract_daily", "sourceTableName": "taobao_order_list",
|
||||
// "extractMode": "AGGREGATE", "groupByFields": ["shop_id"],
|
||||
// "fieldMappings": [...] }
|
||||
|
||||
|
||||
// ============================================================================
|
||||
// Direct vs AGGREGATE 模式对比
|
||||
// ============================================================================
|
||||
//
|
||||
// ┌──────────┬──────────────────────────────────────────────┐
|
||||
// │ 模式 │ 行为 │
|
||||
// ├──────────┼──────────────────────────────────────────────┤
|
||||
// │ DIRECT │ 逐行映射，1:1 从源表复制到目标表 │
|
||||
// │ │ 适用：源表已是一行一统计 │
|
||||
// │ │ SQL: SELECT ... FROM source │
|
||||
// ├──────────┼──────────────────────────────────────────────┤
|
||||
// │ AGGREGATE│ GROUP BY + 聚合函数，N:1 聚合 │
|
||||
// │ │ 适用：源表是明细表（如订单行） │
|
||||
// │ │ SQL: SELECT ... SUM() COUNT() ... │
|
||||
// │ │ FROM source GROUP BY groupByFields │
|
||||
// └──────────┴──────────────────────────────────────────────┘
|
||||
//
|
||||
// 可聚合函数: SUM / COUNT / AVG / MAX / MIN
|
||||
// 字段角色: DIMENSION(维度) / INDICATOR(指标) / FILTER(筛选) / FILTER_ONLY
|
||||
// 字段类型: STRING / INT / FLOAT / DATE / DATETIME / JSONB
|
||||
// 操作符: = / != / > / < / >= / <= / IN / LIKE / BETWEEN
|
||||
// 时间分组: day / week / month / quarter
|
||||
*/
|
||||
@@ -0,0 +1,192 @@
|
||||
package executor
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"math"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"dataengine/common/report/builder"
|
||||
"dataengine/common/report/model"
|
||||
|
||||
"gitea.redpowerfuture.com/red-future/common/db/gfdb"
|
||||
"github.com/sirupsen/logrus"
|
||||
)
|
||||
|
||||
// QueryExecutor SQL执行器
|
||||
type QueryExecutor struct {
|
||||
sqlBuilder *builder.SQLBuilder
|
||||
}
|
||||
|
||||
// NewQueryExecutor 创建查询执行器
|
||||
func NewQueryExecutor() *QueryExecutor {
|
||||
return &QueryExecutor{
|
||||
sqlBuilder: builder.NewSQLBuilder(),
|
||||
}
|
||||
}
|
||||
|
||||
// QueryReportByUserSelect 根据用户选择实时查询报表数据（核心接口）
|
||||
func (e *QueryExecutor) QueryReportByUserSelect(ctx context.Context, req *model.UserSelectQueryReq) (*model.UserSelectQueryResp, error) {
|
||||
start := time.Now()
|
||||
logger := logrus.WithFields(logrus.Fields{
|
||||
"businessCode": req.BusinessCode,
|
||||
"reportCode": req.ReportCode,
|
||||
})
|
||||
|
||||
// 1. 参数校验
|
||||
if err := e.validateReq(req); err != nil {
|
||||
return nil, fmt.Errorf("参数校验失败: %w", err)
|
||||
}
|
||||
|
||||
// 2. 构建查询SQL
|
||||
querySQL, queryArgs, metadata, err := e.sqlBuilder.BuildQuerySQL(ctx, req)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("构建查询SQL失败: %w", err)
|
||||
}
|
||||
|
||||
logger.Debugf("查询SQL: %s args: %v", querySQL, queryArgs)
|
||||
|
||||
// 3. 获取记录总数
|
||||
total, err := e.executeCount(ctx, metadata, queryArgs)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("查询总数失败: %w", err)
|
||||
}
|
||||
|
||||
// 4. 分页查询
|
||||
if req.Page < 1 {
|
||||
req.Page = 1
|
||||
}
|
||||
if req.PageSize < 1 {
|
||||
req.PageSize = 20
|
||||
}
|
||||
if req.PageSize > 1000 {
|
||||
req.PageSize = 1000
|
||||
}
|
||||
|
||||
pagedSQL := e.sqlBuilder.AddLimit(querySQL, req.Page, req.PageSize)
|
||||
logger.Debugf("分页SQL: %s", pagedSQL)
|
||||
|
||||
dataResult, err := gfdb.DB(ctx).GetAll(ctx, pagedSQL, queryArgs...)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("查询数据失败: %w", err)
|
||||
}
|
||||
|
||||
var list []map[string]interface{}
|
||||
if dataResult.Len() > 0 {
|
||||
list = dataResult.List()
|
||||
}
|
||||
|
||||
// 计算总页数
|
||||
totalPages := int(math.Ceil(float64(total) / float64(req.PageSize)))
|
||||
|
||||
execTime := time.Since(start).Milliseconds()
|
||||
|
||||
resp := &model.UserSelectQueryResp{
|
||||
List: list,
|
||||
Total: total,
|
||||
Page: req.Page,
|
||||
PageSize: req.PageSize,
|
||||
TotalPages: totalPages,
|
||||
Sql: querySQL,
|
||||
ExecTimeMs: execTime,
|
||||
}
|
||||
|
||||
logger.Infof("报表查询完成, 总数:%d 耗时:%dms", total, execTime)
|
||||
|
||||
return resp, nil
|
||||
}
|
||||
|
||||
// executeCount 执行总数统计
|
||||
func (e *QueryExecutor) executeCount(ctx context.Context, metadata map[string]interface{}, args []interface{}) (int64, error) {
|
||||
countSql, ok := metadata["countSql"].(string)
|
||||
if !ok || countSql == "" {
|
||||
return 0, nil
|
||||
}
|
||||
|
||||
result, err := gfdb.DB(ctx).GetAll(ctx, countSql, args...)
|
||||
if err != nil {
|
||||
return 0, fmt.Errorf("统计总数失败: %w", err)
|
||||
}
|
||||
|
||||
var total int64
|
||||
if result.Len() > 0 {
|
||||
for k, v := range result.List()[0] {
|
||||
_ = k
|
||||
if n, ok := v.(int64); ok {
|
||||
total = n
|
||||
} else {
|
||||
total = result[0]["count"].Int64()
|
||||
}
|
||||
break
|
||||
}
|
||||
}
|
||||
return total, nil
|
||||
}
|
||||
|
||||
// validateReq 校验请求参数
|
||||
func (e *QueryExecutor) validateReq(req *model.UserSelectQueryReq) error {
|
||||
if req.BusinessCode == "" {
|
||||
return fmt.Errorf("业务编码不能为空")
|
||||
}
|
||||
if req.ReportCode == "" {
|
||||
return fmt.Errorf("报表编码不能为空")
|
||||
}
|
||||
if len(req.Indicators) == 0 {
|
||||
return fmt.Errorf("必须选择至少一个指标")
|
||||
}
|
||||
|
||||
// 校验指标
|
||||
for i, ind := range req.Indicators {
|
||||
if ind.FieldCode == "" {
|
||||
return fmt.Errorf("第%d个指标字段编码不能为空", i+1)
|
||||
}
|
||||
if ind.Aggregate != "" {
|
||||
agg := strings.ToUpper(ind.Aggregate)
|
||||
validAggs := map[string]bool{
|
||||
"SUM": true,
|
||||
"COUNT": true,
|
||||
"AVG": true,
|
||||
"MAX": true,
|
||||
"MIN": true,
|
||||
}
|
||||
if !validAggs[agg] {
|
||||
return fmt.Errorf("不支持的聚合方式: %s", ind.Aggregate)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// 校验时间分组
|
||||
if req.TimeGroup != "" {
|
||||
validGroups := map[string]bool{
|
||||
"day": true,
|
||||
"week": true,
|
||||
"month": true,
|
||||
"quarter": true,
|
||||
}
|
||||
if !validGroups[req.TimeGroup] {
|
||||
return fmt.Errorf("不支持的时间分组: %s", req.TimeGroup)
|
||||
}
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// RawQuery 执行原始SQL查询（用于特殊场景）
|
||||
func (e *QueryExecutor) RawQuery(ctx context.Context, sql string, args ...interface{}) ([]map[string]interface{}, error) {
|
||||
result, err := gfdb.DB(ctx).GetAll(ctx, sql, args...)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("原始查询失败: %w", err)
|
||||
}
|
||||
|
||||
if result.Len() > 0 {
|
||||
return result.List(), nil
|
||||
}
|
||||
return nil, nil
|
||||
}
|
||||
|
||||
// ExecRaw 执行原始SQL（INSERT/UPDATE/DELETE）
|
||||
func (e *QueryExecutor) ExecRaw(ctx context.Context, sql string, args ...interface{}) error {
|
||||
_, err := gfdb.DB(ctx).Exec(ctx, sql, args...)
|
||||
return err
|
||||
}
|
||||
@@ -0,0 +1,644 @@
|
||||
package extract
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"dataengine/common/report/config"
|
||||
"dataengine/common/report/model"
|
||||
|
||||
"gitea.redpowerfuture.com/red-future/common/db/gfdb"
|
||||
"github.com/gogf/gf/v2/util/gconv"
|
||||
"github.com/sirupsen/logrus"
|
||||
)
|
||||
|
||||
// DailyExtractor 天级数据抽取器
|
||||
type DailyExtractor struct {
|
||||
loader *config.ConfigLoader
|
||||
}
|
||||
|
||||
// NewDailyExtractor 创建抽取器
|
||||
func NewDailyExtractor() *DailyExtractor {
|
||||
return &DailyExtractor{
|
||||
loader: config.GetLoader(),
|
||||
}
|
||||
}
|
||||
|
||||
// ExtractDailyData 按天抽取数据（业务层定时任务调用）
|
||||
func (e *DailyExtractor) ExtractDailyData(ctx context.Context, businessCode, reportCode, statDate, executor string) (*model.ExtractDailyDataResp, error) {
|
||||
start := time.Now()
|
||||
logger := logrus.WithFields(logrus.Fields{
|
||||
"businessCode": businessCode,
|
||||
"reportCode": reportCode,
|
||||
"statDate": statDate,
|
||||
})
|
||||
|
||||
// 1. 获取报表配置
|
||||
report, err := e.loader.GetReport(ctx, businessCode, reportCode)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("获取报表配置失败: %w", err)
|
||||
}
|
||||
|
||||
// 2. 获取抽取配置
|
||||
extractConfigs, err := e.loader.GetExtractConfigs(ctx, businessCode, reportCode)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("获取抽取配置失败: %w", err)
|
||||
}
|
||||
|
||||
if len(extractConfigs) == 0 {
|
||||
return nil, fmt.Errorf("没有可用的抽取配置")
|
||||
}
|
||||
|
||||
// 3. 获取字段配置
|
||||
fieldMap, err := e.loader.GetFieldMap(ctx, businessCode, reportCode)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("获取字段配置失败: %w", err)
|
||||
}
|
||||
|
||||
// 4. 确保统计宽表存在
|
||||
if err := e.ensureStatTableExists(ctx, report, fieldMap); err != nil {
|
||||
return nil, fmt.Errorf("确保统计宽表存在失败: %w", err)
|
||||
}
|
||||
|
||||
totalCount := 0
|
||||
successCount := 0
|
||||
failCount := 0
|
||||
var lastErr error
|
||||
|
||||
// 5. 遍历每个抽取配置
|
||||
for _, ec := range extractConfigs {
|
||||
// 检查幂等性
|
||||
exLog, err := e.loader.GetExtractLog(ctx, businessCode, reportCode, ec.ExtractCode, statDate)
|
||||
if err != nil {
|
||||
logger.Errorf("获取抽取记录失败: %v", err)
|
||||
}
|
||||
|
||||
if exLog != nil && exLog.Status == model.ExtractStatusSuccess {
|
||||
logger.Infof("抽取配置 %s 日期 %s 已完成，跳过", ec.ExtractCode, statDate)
|
||||
continue
|
||||
}
|
||||
|
||||
// 创建抽取记录
|
||||
extractLog := &model.ExtractLog{
|
||||
BusinessCode: businessCode,
|
||||
ReportCode: reportCode,
|
||||
ExtractCode: ec.ExtractCode,
|
||||
StatDate: statDate,
|
||||
ExtractType: ec.ExtractType,
|
||||
Status: model.ExtractStatusRunning,
|
||||
Executor: executor,
|
||||
StartTime: &start,
|
||||
}
|
||||
_ = e.loader.CreateExtractLog(ctx, extractLog)
|
||||
|
||||
// 执行抽取
|
||||
c, s, f, err := e.executeExtract(ctx, &ec, report, fieldMap, statDate)
|
||||
totalCount += c
|
||||
successCount += s
|
||||
failCount += f
|
||||
|
||||
// 更新抽取记录
|
||||
now := time.Now()
|
||||
extractLog.EndTime = &now
|
||||
extractLog.TotalCount = c
|
||||
extractLog.SuccessCount = s
|
||||
extractLog.FailCount = f
|
||||
|
||||
if err != nil {
|
||||
extractLog.Status = model.ExtractStatusFailed
|
||||
extractLog.ErrorMessage = err.Error()
|
||||
lastErr = err
|
||||
logger.Errorf("抽取配置 %s 执行失败: %v", ec.ExtractCode, err)
|
||||
} else {
|
||||
extractLog.Status = model.ExtractStatusSuccess
|
||||
logger.Infof("抽取配置 %s 完成, 总数:%d 成功:%d 失败:%d", ec.ExtractCode, c, s, f)
|
||||
}
|
||||
|
||||
if updateErr := e.loader.UpdateExtractLog(ctx, extractLog); updateErr != nil {
|
||||
logger.Errorf("更新抽取记录失败: %v", updateErr)
|
||||
}
|
||||
}
|
||||
|
||||
execTime := time.Since(start).Milliseconds()
|
||||
logger.Infof("按天抽取完成, 总数:%d 成功:%d 失败:%d 耗时:%dms", totalCount, successCount, failCount, execTime)
|
||||
|
||||
resp := &model.ExtractDailyDataResp{
|
||||
Success: lastErr == nil,
|
||||
TotalCount: totalCount,
|
||||
SuccessCount: successCount,
|
||||
FailCount: failCount,
|
||||
ExecTimeMs: execTime,
|
||||
}
|
||||
|
||||
if lastErr != nil {
|
||||
resp.ErrorMsg = lastErr.Error()
|
||||
}
|
||||
|
||||
return resp, nil
|
||||
}
|
||||
|
||||
// executeExtract 执行单个抽取配置
|
||||
func (e *DailyExtractor) executeExtract(ctx context.Context, ec *model.ExtractConfig, report *model.ReportConfig, fieldMap map[string]*model.FieldConfig, statDate string) (total, success, fail int, err error) {
|
||||
logger := logrus.WithField("extractCode", ec.ExtractCode)
|
||||
|
||||
// 1. 构建抽取SQL
|
||||
extractSQL, whereArgs, err := e.buildExtractSQL(ctx, ec, report, statDate)
|
||||
if err != nil {
|
||||
return 0, 0, 0, fmt.Errorf("构建抽取SQL失败: %w", err)
|
||||
}
|
||||
|
||||
logger.Debugf("抽取SQL: %s", extractSQL)
|
||||
|
||||
// 2. 分批抽取
|
||||
batchSize := ec.BatchSize
|
||||
if batchSize <= 0 {
|
||||
batchSize = 1000
|
||||
}
|
||||
|
||||
offset := 0
|
||||
for {
|
||||
// 添加分页
|
||||
pagedSQL := fmt.Sprintf("%s LIMIT %d OFFSET %d", extractSQL, batchSize, offset)
|
||||
args := append(whereArgs)
|
||||
|
||||
rows, queryErr := gfdb.DB(ctx).GetAll(ctx, pagedSQL, args...)
|
||||
if queryErr != nil {
|
||||
return total, success, fail, fmt.Errorf("抽取查询失败: %w", queryErr)
|
||||
}
|
||||
|
||||
batchCount := rows.Len()
|
||||
if batchCount == 0 {
|
||||
break
|
||||
}
|
||||
|
||||
// 3. 应用转换规则（仅 DIRECT 模式需注入审计字段，AGGREGATE 模式已由SQL处理）
|
||||
dataList := rows.List()
|
||||
if ec.ExtractMode != model.ExtractModeAggregate {
|
||||
for i := range dataList {
|
||||
e.applyTransformRules(ec, dataList[i])
|
||||
dataList[i]["tenant_id"] = 1
|
||||
dataList[i]["business_code"] = ec.BusinessCode
|
||||
}
|
||||
}
|
||||
|
||||
// 4. 写入统计宽表
|
||||
c, _, writeErr := e.batchUpsert(ctx, report.StatTableName, report.ConflictKeys, dataList)
|
||||
if writeErr != nil {
|
||||
logger.Errorf("批量写入失败 (offset=%d): %v", offset, writeErr)
|
||||
fail += batchCount
|
||||
} else {
|
||||
success += c
|
||||
}
|
||||
|
||||
total += batchCount
|
||||
offset += batchSize
|
||||
|
||||
if batchCount < batchSize {
|
||||
break
|
||||
}
|
||||
}
|
||||
|
||||
return total, success, fail, nil
|
||||
}
|
||||
|
||||
// buildExtractSQL 构建抽取SQL
|
||||
func (e *DailyExtractor) buildExtractSQL(ctx context.Context, ec *model.ExtractConfig, report *model.ReportConfig, statDate string) (string, []interface{}, error) {
|
||||
var args []interface{}
|
||||
|
||||
sourceTable := ec.SourceTableName
|
||||
if ec.SourceTableAlias != "" {
|
||||
sourceTable = ec.SourceTableAlias
|
||||
} else {
|
||||
sourceTable = "s"
|
||||
}
|
||||
|
||||
// 日期字段
|
||||
dateField := report.DateField
|
||||
if dateField == "" {
|
||||
dateField = "stat_date"
|
||||
}
|
||||
|
||||
// 判断抽取模式
|
||||
mode := ec.ExtractMode
|
||||
if mode == "" {
|
||||
mode = model.ExtractModeDirect
|
||||
}
|
||||
|
||||
if mode == model.ExtractModeAggregate {
|
||||
return e.buildAggregateExtractSQL(ec, report, sourceTable, dateField, statDate)
|
||||
}
|
||||
|
||||
// === 默认 DIRECT 模式：逐行抽取 ===
|
||||
return e.buildDirectExtractSQL(ec, report, sourceTable, dateField, statDate), args, nil
|
||||
}
|
||||
|
||||
// buildDirectExtractSQL 逐行抽取模式SQL（直接映射，不做聚合）
|
||||
func (e *DailyExtractor) buildDirectExtractSQL(ec *model.ExtractConfig, report *model.ReportConfig, sourceTable, dateField, statDate string) string {
|
||||
var selectParts []string
|
||||
|
||||
// 基础审计字段（常量注入）
|
||||
selectParts = append(selectParts, "0 AS id")
|
||||
selectParts = append(selectParts, "1 AS tenant_id")
|
||||
selectParts = append(selectParts, fmt.Sprintf("'%s' AS business_code", ec.BusinessCode))
|
||||
selectParts = append(selectParts, "'system' AS creator")
|
||||
selectParts = append(selectParts, "NOW() AS created_at")
|
||||
selectParts = append(selectParts, "'system' AS updater")
|
||||
selectParts = append(selectParts, "NOW() AS updated_at")
|
||||
selectParts = append(selectParts, "NULL::TIMESTAMP AS deleted_at")
|
||||
|
||||
// 日期字段
|
||||
selectParts = append(selectParts, fmt.Sprintf("'%s' AS %s", statDate, dateField))
|
||||
|
||||
// 原始数据
|
||||
selectParts = append(selectParts, "'{}'::JSONB AS raw_data")
|
||||
|
||||
// 字段映射
|
||||
for _, mapping := range ec.FieldMappings {
|
||||
targetField := mapping.TargetField
|
||||
sourceField := mapping.SourceField
|
||||
|
||||
var expr string
|
||||
if mapping.TransformRule != nil {
|
||||
expr = e.applyTransformExpr(mapping.TransformRule, fmt.Sprintf("%s.%s", sourceTable, sourceField))
|
||||
} else {
|
||||
expr = fmt.Sprintf("%s.%s", sourceTable, sourceField)
|
||||
}
|
||||
|
||||
if mapping.DefaultValue != nil {
|
||||
expr = fmt.Sprintf("COALESCE(%s, '%v')", expr, mapping.DefaultValue)
|
||||
}
|
||||
|
||||
selectParts = append(selectParts, fmt.Sprintf("%s AS %s", expr, targetField))
|
||||
}
|
||||
|
||||
// FROM + JOIN
|
||||
fromClause := e.buildFromClause(ec, sourceTable)
|
||||
|
||||
// JOIN 字段映射
|
||||
selectParts = append(selectParts, e.buildJoinFieldSelects(ec)...)
|
||||
|
||||
// WHERE
|
||||
whereClause := e.buildWhereClause(ec, sourceTable, statDate)
|
||||
|
||||
return fmt.Sprintf("SELECT %s FROM %s %s", strings.Join(selectParts, ", "), fromClause, whereClause)
|
||||
}
|
||||
|
||||
// buildAggregateExtractSQL 聚合抽取模式SQL（GROUP BY + SUM/COUNT/AVG）
|
||||
func (e *DailyExtractor) buildAggregateExtractSQL(ec *model.ExtractConfig, report *model.ReportConfig, sourceTable, dateField, statDate string) (string, []interface{}, error) {
|
||||
var selectParts []string
|
||||
var groupByParts []string
|
||||
var args []interface{}
|
||||
|
||||
// 基础审计字段（聚合模式下用常量）
|
||||
selectParts = append(selectParts, "ROW_NUMBER() OVER () AS id") // 伪自增ID
|
||||
selectParts = append(selectParts, "1 AS tenant_id")
|
||||
selectParts = append(selectParts, fmt.Sprintf("'%s' AS business_code", ec.BusinessCode))
|
||||
selectParts = append(selectParts, "'system' AS creator")
|
||||
selectParts = append(selectParts, "NOW() AS created_at")
|
||||
selectParts = append(selectParts, "'system' AS updater")
|
||||
selectParts = append(selectParts, "NOW() AS updated_at")
|
||||
selectParts = append(selectParts, "NULL::TIMESTAMP AS deleted_at")
|
||||
|
||||
// 日期字段（常量）
|
||||
selectParts = append(selectParts, fmt.Sprintf("'%s' AS %s", statDate, dateField))
|
||||
|
||||
// 原始数据
|
||||
selectParts = append(selectParts, "'{}'::JSONB AS raw_data")
|
||||
|
||||
// GroupByFields 集合（快速查找）
|
||||
gbySet := make(map[string]bool)
|
||||
for _, gbf := range ec.GroupByFields {
|
||||
gbySet[gbf] = true
|
||||
}
|
||||
|
||||
// 添加 GroupBy 字段到 SELECT 和 GROUP BY
|
||||
for _, gbf := range ec.GroupByFields {
|
||||
selectParts = append(selectParts, fmt.Sprintf("%s.%s", sourceTable, gbf))
|
||||
groupByParts = append(groupByParts, fmt.Sprintf("%s.%s", sourceTable, gbf))
|
||||
}
|
||||
|
||||
// 字段映射：根据 AggregateFunction 决定聚合方式
|
||||
for _, mapping := range ec.FieldMappings {
|
||||
targetField := mapping.TargetField
|
||||
sourceField := mapping.SourceField
|
||||
|
||||
// 构建源表达式
|
||||
var sourceExpr string
|
||||
if mapping.TransformRule != nil {
|
||||
sourceExpr = e.applyTransformExpr(mapping.TransformRule, fmt.Sprintf("%s.%s", sourceTable, sourceField))
|
||||
} else {
|
||||
sourceExpr = fmt.Sprintf("%s.%s", sourceTable, sourceField)
|
||||
}
|
||||
|
||||
// 判断是否需要聚合
|
||||
aggFunc := strings.ToUpper(mapping.AggregateFunction)
|
||||
if aggFunc != "" && !gbySet[sourceField] {
|
||||
// 聚合字段：SUM(s.xxx) / COUNT(s.xxx) / AVG(s.xxx)
|
||||
expr := fmt.Sprintf("%s(%s)", aggFunc, sourceExpr)
|
||||
if mapping.DefaultValue != nil {
|
||||
expr = fmt.Sprintf("COALESCE(%s, %v)", expr, mapping.DefaultValue)
|
||||
}
|
||||
selectParts = append(selectParts, fmt.Sprintf("%s AS %s", expr, targetField))
|
||||
} else if gbySet[sourceField] {
|
||||
// GroupBy 字段不需要重复加入 SELECT（已通过 groupByFields 处理）
|
||||
continue
|
||||
} else {
|
||||
// 非聚合字段，也未在 GroupBy 中 → 用 MAX/MIN 取值（兼容 PG only_full_group_by）
|
||||
expr := fmt.Sprintf("MAX(%s)", sourceExpr)
|
||||
if mapping.DefaultValue != nil {
|
||||
expr = fmt.Sprintf("COALESCE(%s, %v)", expr, mapping.DefaultValue)
|
||||
}
|
||||
selectParts = append(selectParts, fmt.Sprintf("%s AS %s", expr, targetField))
|
||||
}
|
||||
}
|
||||
|
||||
// FROM + JOIN
|
||||
fromClause := e.buildFromClause(ec, sourceTable)
|
||||
|
||||
// WHERE
|
||||
whereClause := e.buildWhereClause(ec, sourceTable, statDate)
|
||||
|
||||
// 组合 SQL
|
||||
sql := fmt.Sprintf("SELECT %s FROM %s %s",
|
||||
strings.Join(selectParts, ", "),
|
||||
fromClause,
|
||||
whereClause)
|
||||
|
||||
// GROUP BY
|
||||
if len(groupByParts) > 0 {
|
||||
sql += " GROUP BY " + strings.Join(groupByParts, ", ")
|
||||
}
|
||||
|
||||
return sql, args, nil
|
||||
}
|
||||
|
||||
// buildFromClause 构建FROM + JOIN子句
|
||||
func (e *DailyExtractor) buildFromClause(ec *model.ExtractConfig, sourceTable string) string {
|
||||
fromClause := fmt.Sprintf("%s %s", ec.SourceTableName, sourceTable)
|
||||
|
||||
for _, join := range ec.JoinConfigs {
|
||||
joinType := "LEFT JOIN"
|
||||
jType := strings.ToUpper(join.JoinType)
|
||||
if jType == "INNER" {
|
||||
joinType = "INNER JOIN"
|
||||
} else if jType == "RIGHT" {
|
||||
joinType = "RIGHT JOIN"
|
||||
}
|
||||
|
||||
joinAlias := join.JoinAlias
|
||||
if joinAlias == "" {
|
||||
joinAlias = join.JoinTable
|
||||
}
|
||||
|
||||
fromClause += fmt.Sprintf(" %s %s %s ON %s", joinType, join.JoinTable, joinAlias, join.JoinCondition)
|
||||
}
|
||||
|
||||
return fromClause
|
||||
}
|
||||
|
||||
// buildJoinFieldSelects 构建JOIN表的字段映射SELECT部分
|
||||
func (e *DailyExtractor) buildJoinFieldSelects(ec *model.ExtractConfig) []string {
|
||||
var parts []string
|
||||
for _, join := range ec.JoinConfigs {
|
||||
joinAlias := join.JoinAlias
|
||||
if joinAlias == "" {
|
||||
joinAlias = join.JoinTable
|
||||
}
|
||||
for _, jm := range join.FieldMappings {
|
||||
targetField := jm.TargetField
|
||||
sourceExpr := fmt.Sprintf("%s.%s", joinAlias, jm.SourceField)
|
||||
if jm.TransformRule != nil {
|
||||
sourceExpr = e.applyTransformExpr(jm.TransformRule, sourceExpr)
|
||||
}
|
||||
parts = append(parts, fmt.Sprintf("%s AS %s", sourceExpr, targetField))
|
||||
}
|
||||
}
|
||||
return parts
|
||||
}
|
||||
|
||||
// buildWhereClause 构建WHERE子句
|
||||
func (e *DailyExtractor) buildWhereClause(ec *model.ExtractConfig, sourceTable, statDate string) string {
|
||||
var whereConditions []string
|
||||
|
||||
// 日期范围（增量抽取）
|
||||
if ec.ExtractType == model.ExtractTypeIncremental && ec.ExtractKeyField != "" {
|
||||
dateCondition := fmt.Sprintf("%s.%s::date = '%s'", sourceTable, ec.ExtractKeyField, statDate)
|
||||
whereConditions = append(whereConditions, dateCondition)
|
||||
}
|
||||
|
||||
// 自定义过滤条件
|
||||
if ec.FilterExpression != "" {
|
||||
whereConditions = append(whereConditions, ec.FilterExpression)
|
||||
}
|
||||
|
||||
if len(whereConditions) == 0 {
|
||||
return ""
|
||||
}
|
||||
return "WHERE " + strings.Join(whereConditions, " AND ")
|
||||
}
|
||||
|
||||
// applyTransformExpr 应用转换表达式
|
||||
func (e *DailyExtractor) applyTransformExpr(rule *model.TransformRule, sourceExpr string) string {
|
||||
switch rule.RuleType {
|
||||
case "CALCULATE":
|
||||
if rule.Expression != "" {
|
||||
return strings.ReplaceAll(rule.Expression, "{source}", sourceExpr)
|
||||
}
|
||||
case "FORMAT":
|
||||
if rule.Format != "" {
|
||||
return fmt.Sprintf("TO_CHAR(%s, '%s')", sourceExpr, rule.Format)
|
||||
}
|
||||
case "MAPPING":
|
||||
// 在代码中运行时做映射
|
||||
return sourceExpr
|
||||
}
|
||||
return sourceExpr
|
||||
}
|
||||
|
||||
// applyTransformRules 应用运行时转换规则（映射等代码转换）
|
||||
func (e *DailyExtractor) applyTransformRules(ec *model.ExtractConfig, row map[string]interface{}) {
|
||||
for _, rule := range ec.TransformRules {
|
||||
if rule.RuleType != "MAPPING" {
|
||||
continue
|
||||
}
|
||||
|
||||
sourceField := rule.Expression // 存储源字段名
|
||||
targetField := rule.RuleCode // 存储目标字段名
|
||||
|
||||
if sourceVal, ok := row[sourceField]; ok {
|
||||
strVal := gconv.String(sourceVal)
|
||||
if mapped, exists := rule.Mapping[strVal]; exists {
|
||||
row[targetField] = mapped
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// ensureStatTableExists 确保统计宽表存在
|
||||
func (e *DailyExtractor) ensureStatTableExists(ctx context.Context, report *model.ReportConfig, fieldMap map[string]*model.FieldConfig) error {
|
||||
tableName := report.StatTableName
|
||||
|
||||
// 检查表是否存在
|
||||
result, err := gfdb.DB(ctx).GetAll(ctx, "SELECT COUNT(*) FROM pg_tables WHERE tablename = $1", strings.ToLower(tableName))
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
count := 0
|
||||
if len(result) > 0 {
|
||||
count = result[0]["count"].Int()
|
||||
}
|
||||
|
||||
if count == 0 {
|
||||
// 需要建表
|
||||
return e.createStatTable(ctx, report, fieldMap)
|
||||
}
|
||||
|
||||
logrus.Infof("统计宽表 %s 已存在", tableName)
|
||||
return nil
|
||||
}
|
||||
|
||||
// createStatTable 创建统计宽表
|
||||
func (e *DailyExtractor) createStatTable(ctx context.Context, report *model.ReportConfig, fieldMap map[string]*model.FieldConfig) error {
|
||||
var cols []string
|
||||
|
||||
// 标准审计字段
|
||||
cols = append(cols, "id BIGSERIAL PRIMARY KEY")
|
||||
cols = append(cols, "tenant_id BIGINT NOT NULL DEFAULT 0")
|
||||
cols = append(cols, "business_code VARCHAR(64) NOT NULL DEFAULT ''")
|
||||
cols = append(cols, "creator VARCHAR(64) DEFAULT ''")
|
||||
cols = append(cols, "created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW()")
|
||||
cols = append(cols, "updater VARCHAR(64) DEFAULT ''")
|
||||
cols = append(cols, "updated_at TIMESTAMP WITH TIME ZONE DEFAULT NOW()")
|
||||
cols = append(cols, "deleted_at TIMESTAMP WITH TIME ZONE")
|
||||
|
||||
// 日期字段
|
||||
dateField := report.DateField
|
||||
if dateField == "" {
|
||||
dateField = "stat_date"
|
||||
}
|
||||
cols = append(cols, fmt.Sprintf("%s VARCHAR(16) NOT NULL DEFAULT ''", dateField))
|
||||
|
||||
// 业务字段
|
||||
for _, fc := range fieldMap {
|
||||
fc := fc
|
||||
colType := fieldTypeToPG(fc.FieldType)
|
||||
cols = append(cols, fmt.Sprintf("%s %s", fc.FieldCode, colType))
|
||||
}
|
||||
|
||||
// 原始数据
|
||||
cols = append(cols, "raw_data JSONB DEFAULT '{}'")
|
||||
|
||||
tableName := report.StatTableName
|
||||
sql := fmt.Sprintf("CREATE TABLE IF NOT EXISTS %s (\n %s\n)", tableName, strings.Join(cols, ",\n "))
|
||||
|
||||
logrus.Infof("创建统计宽表: %s", tableName)
|
||||
|
||||
if _, err := gfdb.DB(ctx).Exec(ctx, sql); err != nil {
|
||||
return fmt.Errorf("建表失败: %w", err)
|
||||
}
|
||||
|
||||
// 冲突唯一索引
|
||||
if len(report.ConflictKeys) > 0 {
|
||||
indexName := fmt.Sprintf("uq_%s_conflict", tableName)
|
||||
indexCols := strings.Join(report.ConflictKeys, ", ")
|
||||
indexSQL := fmt.Sprintf("CREATE UNIQUE INDEX IF NOT EXISTS %s ON %s (%s)", indexName, tableName, indexCols)
|
||||
if _, err := gfdb.DB(ctx).Exec(ctx, indexSQL); err != nil {
|
||||
logrus.Warnf("创建冲突索引失败: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// 字段注释
|
||||
for _, fc := range fieldMap {
|
||||
fc := fc
|
||||
if fc.FieldName != "" {
|
||||
escaped := strings.ReplaceAll(fc.FieldName, "'", "''")
|
||||
commentSQL := fmt.Sprintf("COMMENT ON COLUMN %s.%s IS '%s'", tableName, fc.FieldCode, escaped)
|
||||
if _, err := gfdb.DB(ctx).Exec(ctx, commentSQL); err != nil {
|
||||
logrus.Warnf("添加字段注释失败 [%s.%s]: %v", tableName, fc.FieldCode, err)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// batchUpsert 批量upsert写入
|
||||
func (e *DailyExtractor) batchUpsert(ctx context.Context, tableName string, conflictKeys []string, rows []map[string]interface{}) (int, []string, error) {
|
||||
if len(rows) == 0 {
|
||||
return 0, nil, nil
|
||||
}
|
||||
|
||||
now := time.Now()
|
||||
for i := range rows {
|
||||
if rows[i] == nil {
|
||||
rows[i] = make(map[string]interface{})
|
||||
}
|
||||
rows[i]["updated_at"] = now
|
||||
}
|
||||
|
||||
batchSize := 100
|
||||
total := 0
|
||||
var allColumns []string
|
||||
|
||||
for i := 0; i < len(rows); i += batchSize {
|
||||
end := i + batchSize
|
||||
if end > len(rows) {
|
||||
end = len(rows)
|
||||
}
|
||||
batch := rows[i:end]
|
||||
|
||||
m := gfdb.DB(ctx).Model(ctx, tableName).Data(batch)
|
||||
if len(conflictKeys) > 0 {
|
||||
keys := make([]interface{}, len(conflictKeys))
|
||||
for j, k := range conflictKeys {
|
||||
keys[j] = k
|
||||
}
|
||||
m = m.OnConflict(keys...)
|
||||
}
|
||||
|
||||
_, err := m.Save()
|
||||
if err != nil {
|
||||
logrus.Errorf("批量写入 %s 失败: %v", tableName, err)
|
||||
// 逐条重试
|
||||
for _, row := range batch {
|
||||
mm := gfdb.DB(ctx).Model(ctx, tableName).Data(row)
|
||||
if len(conflictKeys) > 0 {
|
||||
keys := make([]interface{}, len(conflictKeys))
|
||||
for j, k := range conflictKeys {
|
||||
keys[j] = k
|
||||
}
|
||||
mm = mm.OnConflict(keys...)
|
||||
}
|
||||
if _, e := mm.Save(); e != nil {
|
||||
logrus.Errorf("逐条写入失败: %v", e)
|
||||
} else {
|
||||
total++
|
||||
}
|
||||
}
|
||||
} else {
|
||||
total += len(batch)
|
||||
}
|
||||
}
|
||||
|
||||
return total, allColumns, nil
|
||||
}
|
||||
|
||||
// fieldTypeToPG 字段类型转PG类型
|
||||
func fieldTypeToPG(fieldType string) string {
|
||||
switch fieldType {
|
||||
case model.FieldTypeInt:
|
||||
return "NUMERIC(20,0) DEFAULT 0"
|
||||
case model.FieldTypeFloat:
|
||||
return "NUMERIC(20,4) DEFAULT 0"
|
||||
case model.FieldTypeDate:
|
||||
return "VARCHAR(16) DEFAULT ''"
|
||||
case model.FieldTypeDatetime:
|
||||
return "TIMESTAMP WITH TIME ZONE"
|
||||
case model.FieldTypeJsonb:
|
||||
return "JSONB DEFAULT '{}'"
|
||||
default:
|
||||
return "VARCHAR(256) DEFAULT ''"
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,424 @@
|
||||
package model
|
||||
|
||||
import (
|
||||
"time"
|
||||
)
|
||||
|
||||
// ============================================================
|
||||
// 实体定义
|
||||
// ============================================================
|
||||
|
||||
// BusinessConfig 业务配置
|
||||
type BusinessConfig struct {
|
||||
ID int64 `orm:"id" json:"id"`
|
||||
TenantId uint64 `orm:"tenant_id" json:"tenant_id"`
|
||||
BusinessCode string `orm:"business_code" json:"businessCode"`
|
||||
BusinessName string `orm:"business_name" json:"businessName"`
|
||||
Description string `orm:"description" json:"description"`
|
||||
Status string `orm:"status" json:"status"`
|
||||
Config map[string]interface{} `orm:"config" json:"config"`
|
||||
Creator string `orm:"creator" json:"creator"`
|
||||
CreatedAt *time.Time `orm:"created_at" json:"createdAt"`
|
||||
Updater string `orm:"updater" json:"updater"`
|
||||
UpdatedAt *time.Time `orm:"updated_at" json:"updatedAt"`
|
||||
DeletedAt *time.Time `orm:"deleted_at" json:"deletedAt"`
|
||||
}
|
||||
|
||||
// ReportConfig 报表配置
|
||||
type ReportConfig struct {
|
||||
ID int64 `orm:"id" json:"id"`
|
||||
TenantId uint64 `orm:"tenant_id" json:"tenant_id"`
|
||||
BusinessCode string `orm:"business_code" json:"businessCode"`
|
||||
ReportCode string `orm:"report_code" json:"reportCode"`
|
||||
ReportName string `orm:"report_name" json:"reportName"`
|
||||
Description string `orm:"description" json:"description"`
|
||||
Status string `orm:"status" json:"status"`
|
||||
StatTableName string `orm:"stat_table_name" json:"statTableName"`
|
||||
StatTableComment string `orm:"stat_table_comment" json:"statTableComment"`
|
||||
DateField string `orm:"date_field" json:"dateField"`
|
||||
PrimaryKeys []string `orm:"primary_keys" json:"primaryKeys"`
|
||||
ConflictKeys []string `orm:"conflict_keys" json:"conflictKeys"`
|
||||
Config map[string]interface{} `orm:"config" json:"config"`
|
||||
Creator string `orm:"creator" json:"creator"`
|
||||
CreatedAt *time.Time `orm:"created_at" json:"createdAt"`
|
||||
Updater string `orm:"updater" json:"updater"`
|
||||
UpdatedAt *time.Time `orm:"updated_at" json:"updatedAt"`
|
||||
DeletedAt *time.Time `orm:"deleted_at" json:"deletedAt"`
|
||||
}
|
||||
|
||||
// FieldConfig 字段配置
|
||||
type FieldConfig struct {
|
||||
ID int64 `orm:"id" json:"id"`
|
||||
TenantId uint64 `orm:"tenant_id" json:"tenant_id"`
|
||||
BusinessCode string `orm:"business_code" json:"businessCode"`
|
||||
ReportCode string `orm:"report_code" json:"reportCode"`
|
||||
FieldCode string `orm:"field_code" json:"fieldCode"`
|
||||
FieldName string `orm:"field_name" json:"fieldName"`
|
||||
FieldType string `orm:"field_type" json:"fieldType"`
|
||||
DataType string `orm:"data_type" json:"dataType"`
|
||||
FieldRole string `orm:"field_role" json:"fieldRole"`
|
||||
IsAggregatable bool `orm:"is_aggregatable" json:"isAggregatable"`
|
||||
IsFilterable bool `orm:"is_filterable" json:"isFilterable"`
|
||||
IsQueryable bool `orm:"is_queryable" json:"isQueryable"`
|
||||
IsSortable bool `orm:"is_sortable" json:"isSortable"`
|
||||
DefaultAggregate string `orm:"default_aggregate" json:"defaultAggregate"`
|
||||
ValidAggregates []string `orm:"valid_aggregates" json:"validAggregates"`
|
||||
FilterOperators []string `orm:"filter_operators" json:"filterOperators"`
|
||||
Expression string `orm:"expression" json:"expression"`
|
||||
ExpressionType string `orm:"expression_type" json:"expressionType"`
|
||||
FormatPattern string `orm:"format_pattern" json:"formatPattern"`
|
||||
Unit string `orm:"unit" json:"unit"`
|
||||
DictCode string `orm:"dict_code" json:"dictCode"`
|
||||
SortOrder int `orm:"sort_order" json:"sortOrder"`
|
||||
GroupName string `orm:"group_name" json:"groupName"`
|
||||
Status string `orm:"status" json:"status"`
|
||||
Creator string `orm:"creator" json:"creator"`
|
||||
CreatedAt *time.Time `orm:"created_at" json:"createdAt"`
|
||||
Updater string `orm:"updater" json:"updater"`
|
||||
UpdatedAt *time.Time `orm:"updated_at" json:"updatedAt"`
|
||||
DeletedAt *time.Time `orm:"deleted_at" json:"deletedAt"`
|
||||
}
|
||||
|
||||
// ExtractConfig 抽取配置
|
||||
type ExtractConfig struct {
|
||||
ID int64 `orm:"id" json:"id"`
|
||||
TenantId uint64 `orm:"tenant_id" json:"tenant_id"`
|
||||
BusinessCode string `orm:"business_code" json:"businessCode"`
|
||||
ReportCode string `orm:"report_code" json:"reportCode"`
|
||||
ExtractCode string `orm:"extract_code" json:"extractCode"`
|
||||
ExtractName string `orm:"extract_name" json:"extractName"`
|
||||
SourceTableName string `orm:"source_table_name" json:"sourceTableName"`
|
||||
SourceTableAlias string `orm:"source_table_alias" json:"sourceTableAlias"`
|
||||
TargetTableName string `orm:"target_table_name" json:"targetTableName"`
|
||||
IsEnabled bool `orm:"is_enabled" json:"isEnabled"`
|
||||
ExtractType string `orm:"extract_type" json:"extractType"`
|
||||
ExtractMode string `orm:"extract_mode" json:"extractMode"`
|
||||
ExtractKeyField string `orm:"extract_key_field" json:"extractKeyField"`
|
||||
ExtractKeyFormat string `orm:"extract_key_format" json:"extractKeyFormat"`
|
||||
GroupByFields []string `orm:"group_by_fields" json:"groupByFields"`
|
||||
FilterExpression string `orm:"filter_expression" json:"filterExpression"`
|
||||
JoinConfigs []JoinConfig `orm:"join_configs" json:"joinConfigs"`
|
||||
FieldMappings []FieldMapping `orm:"field_mappings" json:"fieldMappings"`
|
||||
TransformRules []TransformRule `orm:"transform_rules" json:"transformRules"`
|
||||
BatchSize int `orm:"batch_size" json:"batchSize"`
|
||||
Status string `orm:"status" json:"status"`
|
||||
Creator string `orm:"creator" json:"creator"`
|
||||
CreatedAt *time.Time `orm:"created_at" json:"createdAt"`
|
||||
Updater string `orm:"updater" json:"updater"`
|
||||
UpdatedAt *time.Time `orm:"updated_at" json:"updatedAt"`
|
||||
DeletedAt *time.Time `orm:"deleted_at" json:"deletedAt"`
|
||||
}
|
||||
|
||||
// ExtractLog 抽取记录
|
||||
type ExtractLog struct {
|
||||
ID int64 `orm:"id" json:"id"`
|
||||
BusinessCode string `orm:"business_code" json:"businessCode"`
|
||||
ReportCode string `orm:"report_code" json:"reportCode"`
|
||||
ExtractCode string `orm:"extract_code" json:"extractCode"`
|
||||
StatDate string `orm:"stat_date" json:"statDate"`
|
||||
ExtractType string `orm:"extract_type" json:"extractType"`
|
||||
Status string `orm:"status" json:"status"`
|
||||
TotalCount int `orm:"total_count" json:"totalCount"`
|
||||
SuccessCount int `orm:"success_count" json:"successCount"`
|
||||
FailCount int `orm:"fail_count" json:"failCount"`
|
||||
StartTime *time.Time `orm:"start_time" json:"startTime"`
|
||||
EndTime *time.Time `orm:"end_time" json:"endTime"`
|
||||
ErrorMessage string `orm:"error_message" json:"errorMessage"`
|
||||
Executor string `orm:"executor" json:"executor"`
|
||||
CreatedAt *time.Time `orm:"created_at" json:"createdAt"`
|
||||
UpdatedAt *time.Time `orm:"updated_at" json:"updatedAt"`
|
||||
}
|
||||
|
||||
// ============================================================
|
||||
// 辅助结构
|
||||
// ============================================================
|
||||
|
||||
// JoinConfig 关联配置
|
||||
type JoinConfig struct {
|
||||
JoinTable string `json:"joinTable"`
|
||||
JoinAlias string `json:"joinAlias"`
|
||||
JoinType string `json:"joinType"` // LEFT/RIGHT/INNER
|
||||
JoinCondition string `json:"joinCondition"`
|
||||
FieldMappings []FieldMapping `json:"fieldMappings"`
|
||||
}
|
||||
|
||||
// FieldMapping 字段映射
|
||||
type FieldMapping struct {
|
||||
SourceField string `json:"sourceField"`
|
||||
TargetField string `json:"targetField"`
|
||||
FieldType string `json:"fieldType"`
|
||||
AggregateFunction string `json:"aggregateFunction"`
|
||||
DefaultValue interface{} `json:"defaultValue"`
|
||||
TransformRule *TransformRule `json:"transformRule,omitempty"`
|
||||
}
|
||||
|
||||
// TransformRule 转换规则
|
||||
type TransformRule struct {
|
||||
RuleCode string `json:"ruleCode"`
|
||||
RuleType string `json:"ruleType"` // DIRECT/MAPPING/FORMAT/CALCULATE
|
||||
Expression string `json:"expression"`
|
||||
Format string `json:"format"`
|
||||
Mapping map[string]interface{} `json:"mapping"`
|
||||
}
|
||||
|
||||
// ============================================================
|
||||
// 前端请求/响应结构体
|
||||
// ============================================================
|
||||
|
||||
// UserSelectQueryReq 用户选择查询请求
|
||||
type UserSelectQueryReq struct {
|
||||
BusinessCode string `json:"businessCode" v:"required" dc:"业务编码"`
|
||||
ReportCode string `json:"reportCode" v:"required" dc:"报表编码"`
|
||||
Dimensions []string `json:"dimensions" dc:"统计维度列表，如 shop_id/anchor_id/date"`
|
||||
Indicators []IndicatorSelect `json:"indicators" dc:"统计指标列表(含聚合方式)"`
|
||||
Filters []FilterCondition `json:"filters" dc:"筛选条件列表"`
|
||||
TimeRange *TimeRange `json:"timeRange" dc:"时间范围"`
|
||||
TimeGroup string `json:"timeGroup" dc:"时间分组: day/week/month/quarter"`
|
||||
OrderBy []OrderCondition `json:"orderBy" dc:"排序条件"`
|
||||
Page int `json:"page" dc:"页码" d:"1"`
|
||||
PageSize int `json:"pageSize" dc:"每页条数" d:"20"`
|
||||
}
|
||||
|
||||
// IndicatorSelect 指标选择
|
||||
type IndicatorSelect struct {
|
||||
FieldCode string `json:"fieldCode" dc:"字段编码"`
|
||||
Aggregate string `json:"aggregate" dc:"聚合方式: SUM/COUNT/AVG/MAX/MIN"`
|
||||
Alias string `json:"alias" dc:"别名"`
|
||||
}
|
||||
|
||||
// FilterCondition 筛选条件
|
||||
type FilterCondition struct {
|
||||
FieldCode string `json:"fieldCode" dc:"字段编码"`
|
||||
Operator string `json:"operator" dc:"操作符: =/!=/>/</>=/<=/IN/LIKE/BETWEEN"`
|
||||
Value interface{} `json:"value" dc:"值"`
|
||||
Value2 interface{} `json:"value2" dc:"第二个值(BETWEEN时使用)"`
|
||||
}
|
||||
|
||||
// TimeRange 时间范围
|
||||
type TimeRange struct {
|
||||
StartDate string `json:"startDate" dc:"开始日期 yyyy-MM-dd"`
|
||||
EndDate string `json:"endDate" dc:"结束日期 yyyy-MM-dd"`
|
||||
}
|
||||
|
||||
// OrderCondition 排序条件
|
||||
type OrderCondition struct {
|
||||
FieldCode string `json:"fieldCode" dc:"字段编码"`
|
||||
Direction string `json:"direction" dc:"排序方向: ASC/DESC"`
|
||||
}
|
||||
|
||||
// UserSelectQueryResp 用户选择查询响应
|
||||
type UserSelectQueryResp struct {
|
||||
List []map[string]interface{} `json:"list" dc:"数据列表"`
|
||||
Total int64 `json:"total" dc:"总数"`
|
||||
Page int `json:"page" dc:"当前页"`
|
||||
PageSize int `json:"pageSize" dc:"每页条数"`
|
||||
TotalPages int `json:"totalPages" dc:"总页数"`
|
||||
Sql string `json:"sql,omitempty" dc:"执行的SQL(调试用)"`
|
||||
ExecTimeMs int64 `json:"execTimeMs" dc:"执行耗时(毫秒)"`
|
||||
}
|
||||
|
||||
// ExtractDailyDataReq 按天抽取数据请求
|
||||
type ExtractDailyDataReq struct {
|
||||
BusinessCode string `json:"businessCode" v:"required" dc:"业务编码"`
|
||||
ReportCode string `json:"reportCode" v:"required" dc:"报表编码"`
|
||||
StatDate string `json:"statDate" v:"required" dc:"统计日期 yyyy-MM-dd"`
|
||||
Executor string `json:"executor" dc:"执行人"`
|
||||
}
|
||||
|
||||
// ExtractDailyDataResp 按天抽取数据响应
|
||||
type ExtractDailyDataResp struct {
|
||||
Success bool `json:"success" dc:"是否成功"`
|
||||
TotalCount int `json:"totalCount" dc:"总记录数"`
|
||||
SuccessCount int `json:"successCount" dc:"成功记录数"`
|
||||
FailCount int `json:"failCount" dc:"失败记录数"`
|
||||
ExecTimeMs int64 `json:"execTimeMs" dc:"执行耗时(毫秒)"`
|
||||
ErrorMsg string `json:"errorMsg" dc:"错误信息"`
|
||||
}
|
||||
|
||||
// AutoCreateStatTableReq 自动创建统计宽表请求
|
||||
type AutoCreateStatTableReq struct {
|
||||
BusinessCode string `json:"businessCode" v:"required" dc:"业务编码"`
|
||||
ReportCode string `json:"reportCode" v:"required" dc:"报表编码"`
|
||||
Creator string `json:"creator" dc:"创建人"`
|
||||
}
|
||||
|
||||
// AutoCreateStatTableResp 自动创建统计宽表响应
|
||||
type AutoCreateStatTableResp struct {
|
||||
Success bool `json:"success" dc:"是否成功"`
|
||||
TableName string `json:"tableName" dc:"创建的表名"`
|
||||
ColumnCount int `json:"columnCount" dc:"字段数量"`
|
||||
ExecTimeMs int64 `json:"execTimeMs" dc:"执行耗时(毫秒)"`
|
||||
}
|
||||
|
||||
// GetReportFieldsResp 获取报表可用字段响应
|
||||
type GetReportFieldsResp struct {
|
||||
BusinessCode string `json:"businessCode" dc:"业务编码"`
|
||||
ReportCode string `json:"reportCode" dc:"报表编码"`
|
||||
Dimensions []FieldConfig `json:"dimensions" dc:"维度字段列表"`
|
||||
Indicators []FieldConfig `json:"indicators" dc:"指标字段列表"`
|
||||
Filters []FieldConfig `json:"filters" dc:"筛选字段列表"`
|
||||
}
|
||||
|
||||
// ============================================================
|
||||
// 配置 CRUD 请求/响应
|
||||
// ============================================================
|
||||
|
||||
// SaveBusinessReq 保存业务配置请求（新增/修改合一）
|
||||
type SaveBusinessReq struct {
|
||||
ID *int64 `json:"id"` // 有值为更新，无值为新增
|
||||
BusinessCode string `json:"businessCode" v:"required" dc:"业务编码"`
|
||||
BusinessName string `json:"businessName" v:"required" dc:"业务名称"`
|
||||
Description string `json:"description" dc:"描述"`
|
||||
Status string `json:"status" dc:"状态 ACTIVE/INACTIVE" d:"ACTIVE"`
|
||||
Config map[string]interface{} `json:"config" dc:"扩展配置"`
|
||||
Operator string `json:"operator" dc:"操作人"`
|
||||
}
|
||||
|
||||
// SaveReportReq 保存报表配置请求
|
||||
type SaveReportReq struct {
|
||||
ID *int64 `json:"id"`
|
||||
BusinessCode string `json:"businessCode" v:"required" dc:"业务编码"`
|
||||
ReportCode string `json:"reportCode" v:"required" dc:"报表编码"`
|
||||
ReportName string `json:"reportName" v:"required" dc:"报表名称"`
|
||||
Description string `json:"description" dc:"描述"`
|
||||
Status string `json:"status" dc:"状态" d:"ACTIVE"`
|
||||
StatTableName string `json:"statTableName" v:"required" dc:"统计宽表名"`
|
||||
StatTableComment string `json:"statTableComment" dc:"统计宽表注释"`
|
||||
DateField string `json:"dateField" dc:"日期字段" d:"stat_date"`
|
||||
PrimaryKeys []string `json:"primaryKeys" dc:"主键字段"`
|
||||
ConflictKeys []string `json:"conflictKeys" dc:"冲突键（唯一索引）"`
|
||||
Config map[string]interface{} `json:"config" dc:"扩展配置"`
|
||||
Operator string `json:"operator" dc:"操作人"`
|
||||
}
|
||||
|
||||
// SaveFieldReq 保存字段配置请求
|
||||
type SaveFieldReq struct {
|
||||
ID *int64 `json:"id"`
|
||||
BusinessCode string `json:"businessCode" v:"required" dc:"业务编码"`
|
||||
ReportCode string `json:"reportCode" v:"required" dc:"报表编码"`
|
||||
FieldCode string `json:"fieldCode" v:"required" dc:"字段编码"`
|
||||
FieldName string `json:"fieldName" v:"required" dc:"字段名称"`
|
||||
FieldType string `json:"fieldType" v:"required" dc:"字段类型 STRING/INT/FLOAT/DATE/DATETIME/JSONB"`
|
||||
DataType string `json:"dataType" dc:"数据存储类型" d:"STRING"`
|
||||
FieldRole string `json:"fieldRole" v:"required" dc:"字段角色 DIMENSION/INDICATOR/FILTER/FILTER_ONLY"`
|
||||
IsAggregatable bool `json:"isAggregatable" dc:"是否可聚合"`
|
||||
IsFilterable bool `json:"isFilterable" dc:"是否可筛选" d:"true"`
|
||||
IsQueryable bool `json:"isQueryable" dc:"是否可查询" d:"true"`
|
||||
IsSortable bool `json:"isSortable" dc:"是否可排序" d:"true"`
|
||||
DefaultAggregate string `json:"defaultAggregate" dc:"默认聚合方式"`
|
||||
ValidAggregates []string `json:"validAggregates" dc:"可选聚合列表"`
|
||||
FilterOperators []string `json:"filterOperators" dc:"可选操作符列表"`
|
||||
Expression string `json:"expression" dc:"表达式（衍生字段）"`
|
||||
ExpressionType string `json:"expressionType" dc:"表达式类型 DIRECT/CALCULATED"`
|
||||
FormatPattern string `json:"formatPattern" dc:"格式化模板"`
|
||||
Unit string `json:"unit" dc:"单位"`
|
||||
DictCode string `json:"dictCode" dc:"字典编码"`
|
||||
SortOrder int `json:"sortOrder" dc:"排序"`
|
||||
GroupName string `json:"groupName" dc:"分组名称"`
|
||||
Status string `json:"status" dc:"状态" d:"ACTIVE"`
|
||||
Operator string `json:"operator" dc:"操作人"`
|
||||
}
|
||||
|
||||
// SaveExtractConfigReq 保存抽取配置请求
|
||||
type SaveExtractConfigReq struct {
|
||||
ID *int64 `json:"id"`
|
||||
BusinessCode string `json:"businessCode" v:"required" dc:"业务编码"`
|
||||
ReportCode string `json:"reportCode" v:"required" dc:"报表编码"`
|
||||
ExtractCode string `json:"extractCode" v:"required" dc:"抽取编码"`
|
||||
ExtractName string `json:"extractName" v:"required" dc:"抽取名称"`
|
||||
SourceTableName string `json:"sourceTableName" v:"required" dc:"源表名"`
|
||||
SourceTableAlias string `json:"sourceTableAlias" dc:"源表别名"`
|
||||
TargetTableName string `json:"targetTableName" v:"required" dc:"目标表名"`
|
||||
IsEnabled bool `json:"isEnabled" dc:"是否启用" d:"true"`
|
||||
ExtractType string `json:"extractType" dc:"抽取类型 FULL/INCREMENTAL" d:"INCREMENTAL"`
|
||||
ExtractMode string `json:"extractMode" dc:"抽取模式 DIRECT/AGGREGATE" d:"DIRECT"`
|
||||
ExtractKeyField string `json:"extractKeyField" dc:"抽取关键字段（增量依据）"`
|
||||
ExtractKeyFormat string `json:"extractKeyFormat" dc:"关键字段格式"`
|
||||
GroupByFields []string `json:"groupByFields" dc:"GROUP BY 字段列表"`
|
||||
FilterExpression string `json:"filterExpression" dc:"过滤表达式"`
|
||||
JoinConfigs []JoinConfig `json:"joinConfigs" dc:"JOIN配置"`
|
||||
FieldMappings []FieldMapping `json:"fieldMappings" dc:"字段映射列表"`
|
||||
TransformRules []TransformRule `json:"transformRules" dc:"转换规则列表"`
|
||||
BatchSize int `json:"batchSize" dc:"批处理大小" d:"1000"`
|
||||
Status string `json:"status" dc:"状态" d:"ACTIVE"`
|
||||
Operator string `json:"operator" dc:"操作人"`
|
||||
}
|
||||
|
||||
// IdReq 通用 ID 请求
|
||||
type IdReq struct {
|
||||
ID int64 `json:"id" v:"required" dc:"主键ID"`
|
||||
}
|
||||
|
||||
// SaveResult 写操作通用返回
|
||||
type SaveResult struct {
|
||||
Success bool `json:"success"`
|
||||
ID int64 `json:"id"`
|
||||
Message string `json:"message"`
|
||||
}
|
||||
|
||||
// DeleteResult 删除结果
|
||||
type DeleteResult struct {
|
||||
Success bool `json:"success"`
|
||||
Message string `json:"message"`
|
||||
}
|
||||
|
||||
// GetExtractConfigsReq 获取抽取配置列表请求
|
||||
type GetExtractConfigsReq struct {
|
||||
BusinessCode string `json:"businessCode" v:"required" dc:"业务编码"`
|
||||
ReportCode string `json:"reportCode" v:"required" dc:"报表编码"`
|
||||
}
|
||||
|
||||
// ============================================================
|
||||
// 常量定义
|
||||
// ============================================================
|
||||
|
||||
const (
|
||||
// 状态
|
||||
StatusActive = "ACTIVE"
|
||||
StatusInactive = "INACTIVE"
|
||||
|
||||
// 字段角色
|
||||
RoleDimension = "DIMENSION"
|
||||
RoleIndicator = "INDICATOR"
|
||||
RoleFilter = "FILTER"
|
||||
RoleFilterOnly = "FILTER_ONLY"
|
||||
|
||||
// 字段类型
|
||||
FieldTypeString = "STRING"
|
||||
FieldTypeInt = "INT"
|
||||
FieldTypeFloat = "FLOAT"
|
||||
FieldTypeDate = "DATE"
|
||||
FieldTypeDatetime = "DATETIME"
|
||||
FieldTypeJsonb = "JSONB"
|
||||
|
||||
// 聚合方式
|
||||
AggregateSum = "SUM"
|
||||
AggregateCount = "COUNT"
|
||||
AggregateAvg = "AVG"
|
||||
AggregateMax = "MAX"
|
||||
AggregateMin = "MIN"
|
||||
|
||||
// 操作符
|
||||
OperatorEq = "="
|
||||
OperatorNe = "!="
|
||||
OperatorGt = ">"
|
||||
OperatorLt = "<"
|
||||
OperatorGe = ">="
|
||||
OperatorLe = "<="
|
||||
OperatorIn = "IN"
|
||||
OperatorLike = "LIKE"
|
||||
OperatorBetween = "BETWEEN"
|
||||
|
||||
// 抽取类型
|
||||
ExtractTypeFull = "FULL"
|
||||
ExtractTypeIncremental = "INCREMENTAL"
|
||||
|
||||
// 抽取模式
|
||||
ExtractModeDirect = "DIRECT" // 逐行抽取（默认，源表每行 → 宽表一行）
|
||||
ExtractModeAggregate = "AGGREGATE" // 聚合抽取（按 GROUP BY 聚合，SUM/COUNT/AVG）
|
||||
|
||||
// 抽取状态
|
||||
ExtractStatusRunning = "RUNNING"
|
||||
ExtractStatusSuccess = "SUCCESS"
|
||||
ExtractStatusFailed = "FAILED"
|
||||
)
|
||||
@@ -0,0 +1,146 @@
|
||||
package report
|
||||
|
||||
import (
|
||||
"context"
|
||||
|
||||
"gitea.redpowerfuture.com/red-future/common/db/gfdb"
|
||||
)
|
||||
|
||||
// initTables 初始化系统表
|
||||
func initTables(ctx context.Context) error {
|
||||
ddls := []string{
|
||||
// 业务配置表
|
||||
`CREATE TABLE IF NOT EXISTS report_business_config (
|
||||
id BIGSERIAL PRIMARY KEY,
|
||||
tenant_id BIGINT NOT NULL DEFAULT 0,
|
||||
business_code VARCHAR(64) NOT NULL,
|
||||
business_name VARCHAR(128) NOT NULL,
|
||||
description TEXT DEFAULT '',
|
||||
status VARCHAR(16) NOT NULL DEFAULT 'ACTIVE',
|
||||
config JSONB DEFAULT '{}',
|
||||
creator VARCHAR(64) DEFAULT '',
|
||||
created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
|
||||
updater VARCHAR(64) DEFAULT '',
|
||||
updated_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
|
||||
deleted_at TIMESTAMP WITH TIME ZONE
|
||||
)`,
|
||||
|
||||
// 报表配置表
|
||||
`CREATE TABLE IF NOT EXISTS report_report_config (
|
||||
id BIGSERIAL PRIMARY KEY,
|
||||
tenant_id BIGINT NOT NULL DEFAULT 0,
|
||||
business_code VARCHAR(64) NOT NULL,
|
||||
report_code VARCHAR(64) NOT NULL,
|
||||
report_name VARCHAR(128) NOT NULL,
|
||||
description TEXT DEFAULT '',
|
||||
status VARCHAR(16) NOT NULL DEFAULT 'ACTIVE',
|
||||
stat_table_name VARCHAR(128) NOT NULL,
|
||||
stat_table_comment VARCHAR(256) DEFAULT '',
|
||||
date_field VARCHAR(64) DEFAULT 'stat_date',
|
||||
primary_keys JSONB DEFAULT '["id"]'::jsonb,
|
||||
conflict_keys JSONB DEFAULT '["stat_date"]'::jsonb,
|
||||
config JSONB DEFAULT '{}',
|
||||
creator VARCHAR(64) DEFAULT '',
|
||||
created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
|
||||
updater VARCHAR(64) DEFAULT '',
|
||||
updated_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
|
||||
deleted_at TIMESTAMP WITH TIME ZONE,
|
||||
CONSTRAINT uk_business_report_code UNIQUE (tenant_id, business_code, report_code)
|
||||
)`,
|
||||
|
||||
// 字段配置表
|
||||
`CREATE TABLE IF NOT EXISTS report_field_config (
|
||||
id BIGSERIAL PRIMARY KEY,
|
||||
tenant_id BIGINT NOT NULL DEFAULT 0,
|
||||
business_code VARCHAR(64) NOT NULL,
|
||||
report_code VARCHAR(64) NOT NULL,
|
||||
field_code VARCHAR(64) NOT NULL,
|
||||
field_name VARCHAR(128) NOT NULL,
|
||||
field_type VARCHAR(32) NOT NULL,
|
||||
data_type VARCHAR(32) NOT NULL DEFAULT 'STRING',
|
||||
field_role VARCHAR(32) NOT NULL,
|
||||
is_aggregatable BOOLEAN DEFAULT FALSE,
|
||||
is_filterable BOOLEAN DEFAULT TRUE,
|
||||
is_queryable BOOLEAN DEFAULT TRUE,
|
||||
is_sortable BOOLEAN DEFAULT TRUE,
|
||||
default_aggregate VARCHAR(32) DEFAULT '',
|
||||
valid_aggregates JSONB DEFAULT '[]'::jsonb,
|
||||
filter_operators JSONB DEFAULT '["=","!=",">","<",">=","<=","IN","LIKE","BETWEEN"]'::jsonb,
|
||||
expression VARCHAR(512) DEFAULT '',
|
||||
expression_type VARCHAR(32) DEFAULT '',
|
||||
format_pattern VARCHAR(64) DEFAULT '',
|
||||
unit VARCHAR(32) DEFAULT '',
|
||||
dict_code VARCHAR(64) DEFAULT '',
|
||||
sort_order INT DEFAULT 0,
|
||||
group_name VARCHAR(64) DEFAULT '',
|
||||
status VARCHAR(16) NOT NULL DEFAULT 'ACTIVE',
|
||||
creator VARCHAR(64) DEFAULT '',
|
||||
created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
|
||||
updater VARCHAR(64) DEFAULT '',
|
||||
updated_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
|
||||
deleted_at TIMESTAMP WITH TIME ZONE,
|
||||
CONSTRAINT uk_business_report_field_code UNIQUE (tenant_id, business_code, report_code, field_code)
|
||||
)`,
|
||||
|
||||
// 抽取配置表
|
||||
`CREATE TABLE IF NOT EXISTS report_extract_config (
|
||||
id BIGSERIAL PRIMARY KEY,
|
||||
tenant_id BIGINT NOT NULL DEFAULT 0,
|
||||
business_code VARCHAR(64) NOT NULL,
|
||||
report_code VARCHAR(64) NOT NULL,
|
||||
extract_code VARCHAR(64) NOT NULL,
|
||||
extract_name VARCHAR(128) NOT NULL,
|
||||
source_table_name VARCHAR(128) NOT NULL,
|
||||
source_table_alias VARCHAR(64) DEFAULT '',
|
||||
target_table_name VARCHAR(128) NOT NULL,
|
||||
is_enabled BOOLEAN DEFAULT TRUE,
|
||||
extract_type VARCHAR(32) NOT NULL DEFAULT 'FULL',
|
||||
extract_mode VARCHAR(32) NOT NULL DEFAULT 'DIRECT',
|
||||
extract_key_field VARCHAR(64) DEFAULT '',
|
||||
extract_key_format VARCHAR(64) DEFAULT '',
|
||||
group_by_fields JSONB DEFAULT '[]'::jsonb,
|
||||
filter_expression TEXT DEFAULT '',
|
||||
join_configs JSONB DEFAULT '[]'::jsonb,
|
||||
field_mappings JSONB DEFAULT '[]'::jsonb,
|
||||
transform_rules JSONB DEFAULT '[]'::jsonb,
|
||||
batch_size INT DEFAULT 1000,
|
||||
status VARCHAR(16) NOT NULL DEFAULT 'ACTIVE',
|
||||
creator VARCHAR(64) DEFAULT '',
|
||||
created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
|
||||
updater VARCHAR(64) DEFAULT '',
|
||||
updated_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
|
||||
deleted_at TIMESTAMP WITH TIME ZONE,
|
||||
CONSTRAINT uk_business_report_extract_code UNIQUE (tenant_id, business_code, report_code, extract_code)
|
||||
)`,
|
||||
|
||||
// 抽取记录表
|
||||
`CREATE TABLE IF NOT EXISTS report_extract_log (
|
||||
id BIGSERIAL PRIMARY KEY,
|
||||
tenant_id BIGINT NOT NULL DEFAULT 0,
|
||||
business_code VARCHAR(64) NOT NULL,
|
||||
report_code VARCHAR(64) NOT NULL,
|
||||
extract_code VARCHAR(64) NOT NULL,
|
||||
stat_date VARCHAR(16) NOT NULL,
|
||||
extract_type VARCHAR(32) NOT NULL,
|
||||
status VARCHAR(16) NOT NULL DEFAULT 'RUNNING',
|
||||
total_count INT DEFAULT 0,
|
||||
success_count INT DEFAULT 0,
|
||||
fail_count INT DEFAULT 0,
|
||||
start_time TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
|
||||
end_time TIMESTAMP WITH TIME ZONE,
|
||||
error_message TEXT DEFAULT '',
|
||||
executor VARCHAR(64) DEFAULT '',
|
||||
created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
|
||||
updated_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
|
||||
CONSTRAINT uk_extract_keys UNIQUE (tenant_id, business_code, report_code, extract_code, stat_date)
|
||||
)`,
|
||||
}
|
||||
|
||||
for _, ddl := range ddls {
|
||||
if _, err := gfdb.DB(ctx).Exec(ctx, ddl); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
@@ -17,6 +17,7 @@ sync:
|
||||
sync_interval_minutes: 60 # 自动同步间隔（分钟）
|
||||
compensation_interval_seconds: 300 # 补偿调度器扫描间隔（秒）
|
||||
auto_sync_enabled: false # 是否启用自动同步
|
||||
sync_timeout_minutes: 120 # 单次同步超时（分钟），全量超大表可适当调大
|
||||
|
||||
# Database.
|
||||
database:
|
||||
|
||||
@@ -1,13 +1,10 @@
|
||||
package api_feature
|
||||
|
||||
// AppStatus 应用状态
|
||||
type AppStatus string
|
||||
// ApplicationStatus 应用状态（暂未使用，保留供未来扩展）
|
||||
type ApplicationStatus string
|
||||
|
||||
const (
|
||||
AppStatusActive AppStatus = "active" // 启用
|
||||
AppStatusInactive AppStatus = "inactive" // 停用
|
||||
ApplicationStatusPending ApplicationStatus = "pending"
|
||||
ApplicationStatusApproved ApplicationStatus = "approved"
|
||||
ApplicationStatusRejected ApplicationStatus = "rejected"
|
||||
)
|
||||
|
||||
func (s AppStatus) String() string {
|
||||
return string(s)
|
||||
}
|
||||
|
||||
@@ -1,17 +1,10 @@
|
||||
package api_feature
|
||||
|
||||
// AppType 应用类型
|
||||
type AppType string
|
||||
// ApplicationType 应用类型（暂未使用，保留供未来扩展）
|
||||
type ApplicationType string
|
||||
|
||||
const (
|
||||
AppTypeWeb AppType = "web" // Web应用
|
||||
AppTypeMobile AppType = "mobile" // 移动应用
|
||||
AppTypeMiniApp AppType = "mini_app" // 小程序
|
||||
AppTypeH5 AppType = "h5" // H5应用
|
||||
AppTypeDesktop AppType = "desktop" // 桌面应用
|
||||
AppTypeThirdParty AppType = "third_party" // 第三方应用
|
||||
ApplicationTypeAPI ApplicationType = "api"
|
||||
ApplicationTypeWebhook ApplicationType = "webhook"
|
||||
ApplicationTypeSDK ApplicationType = "sdk"
|
||||
)
|
||||
|
||||
func (s AppType) String() string {
|
||||
return string(s)
|
||||
}
|
||||
|
||||
@@ -1,16 +1,11 @@
|
||||
package api_feature
|
||||
|
||||
// FetchStatus 数据获取状态
|
||||
// FetchStatus 数据拉取状态（暂未使用，保留供未来扩展）
|
||||
type FetchStatus string
|
||||
|
||||
const (
|
||||
FetchStatusPending FetchStatus = "pending" // 待执行
|
||||
FetchStatusRunning FetchStatus = "running" // 执行中
|
||||
FetchStatusSuccess FetchStatus = "success" // 成功
|
||||
FetchStatusFailed FetchStatus = "failed" // 失败
|
||||
FetchStatusRateLimit FetchStatus = "rate_limit" // 触发限流
|
||||
FetchStatusPending FetchStatus = "pending"
|
||||
FetchStatusRunning FetchStatus = "running"
|
||||
FetchStatusCompleted FetchStatus = "completed"
|
||||
FetchStatusFailed FetchStatus = "failed"
|
||||
)
|
||||
|
||||
func (f FetchStatus) String() string {
|
||||
return string(f)
|
||||
}
|
||||
|
||||
@@ -1,14 +1,9 @@
|
||||
package api_feature
|
||||
|
||||
// LimitType 限流类型
|
||||
// LimitType 限流类型（暂未使用，保留供未来扩展）
|
||||
type LimitType string
|
||||
|
||||
const (
|
||||
LimitTypeApp LimitType = "app" // 应用维度限流
|
||||
LimitTypeTenant LimitType = "tenant" // 租户维度限流
|
||||
LimitTypeApi LimitType = "api" // 接口维度限流
|
||||
LimitTypeRate LimitType = "rate"
|
||||
LimitTypeBurst LimitType = "burst"
|
||||
)
|
||||
|
||||
func (l LimitType) String() string {
|
||||
return string(l)
|
||||
}
|
||||
|
||||
@@ -1,13 +1,9 @@
|
||||
package api_feature
|
||||
|
||||
// MappingStatus 映射状态
|
||||
// MappingStatus 字段映射状态（暂未使用，保留供未来扩展）
|
||||
type MappingStatus string
|
||||
|
||||
const (
|
||||
MappingStatusActive MappingStatus = "active" // 启用
|
||||
MappingStatusInactive MappingStatus = "inactive" // 停用
|
||||
MappingStatusMapped MappingStatus = "mapped"
|
||||
MappingStatusUnmapped MappingStatus = "unmapped"
|
||||
)
|
||||
|
||||
func (s MappingStatus) String() string {
|
||||
return string(s)
|
||||
}
|
||||
|
||||
@@ -1,21 +1,11 @@
|
||||
package api_feature
|
||||
|
||||
// SyncPlatform 同步平台类型
|
||||
type SyncPlatform string
|
||||
// PlatformType 平台类型（暂未使用，保留供未来扩展）
|
||||
type PlatformType string
|
||||
|
||||
const (
|
||||
PlatformTaobao SyncPlatform = "taobao" // 淘宝
|
||||
PlatformJD SyncPlatform = "jd" // 京东
|
||||
PlatformKuaishou SyncPlatform = "kuaishou" // 快手
|
||||
PlatformDouyin SyncPlatform = "douyin" // 抖音
|
||||
PlatformXhs SyncPlatform = "xhs" // 小红书
|
||||
PlatformPdd SyncPlatform = "pdd" // 拼多多
|
||||
PlatformXianyu SyncPlatform = "xianyu" // 闲鱼
|
||||
PlatformTmall SyncPlatform = "tmall" // 天猫
|
||||
PlatformWechat SyncPlatform = "wechat" // 微信
|
||||
PlatformCustom SyncPlatform = "custom" // 自定义平台
|
||||
PlatformTypeAds PlatformType = "ads"
|
||||
PlatformTypeEcom PlatformType = "ecom"
|
||||
PlatformTypeSocial PlatformType = "social"
|
||||
PlatformTypeDingTalk PlatformType = "dingtalk"
|
||||
)
|
||||
|
||||
func (s SyncPlatform) String() string {
|
||||
return string(s)
|
||||
}
|
||||
|
||||
@@ -1,16 +1,13 @@
|
||||
package api_feature
|
||||
|
||||
// TransformType 转换类型
|
||||
// TransformType 字段转换类型（暂未使用，保留供未来扩展）
|
||||
type TransformType string
|
||||
|
||||
const (
|
||||
TransformTypeFixed TransformType = "fixed" // 固定值
|
||||
TransformTypeMapping TransformType = "mapping" // 值映射
|
||||
TransformTypeRegex TransformType = "regex" // 正则转换
|
||||
TransformTypeFunction TransformType = "function" // 函数转换
|
||||
TransformTypeScript TransformType = "script" // 脚本转换
|
||||
TransformTypeDirect TransformType = "direct"
|
||||
TransformTypeFormatDate TransformType = "formatDate"
|
||||
TransformTypeMapValue TransformType = "mapValue"
|
||||
TransformTypeConcat TransformType = "concat"
|
||||
TransformTypeCalc TransformType = "calc"
|
||||
TransformTypeRegex TransformType = "regex"
|
||||
)
|
||||
|
||||
func (t TransformType) String() string {
|
||||
return string(t)
|
||||
}
|
||||
|
||||
+9
-22
@@ -1,6 +1,12 @@
|
||||
package dict
|
||||
|
||||
// 转换类型
|
||||
// 注意：请不要在此文件添加与 consts/api-feature 包重复的类型定义。
|
||||
// PlatformStatus 和 ApiMethod 等类型已在 dataengine/consts/api-feature 中定义，
|
||||
// 请使用 api_feature.PlatformStatus / api_feature.ApiMethod 替代。
|
||||
|
||||
// Package dict 仅提供该包特有时使用。
|
||||
|
||||
// 转换类型（暂未使用，保留供未来扩展）
|
||||
const (
|
||||
TransformTypeDirect = "direct" // 直接映射
|
||||
TransformTypeFormatDate = "formatDate" // 日期格式化
|
||||
@@ -10,7 +16,7 @@ const (
|
||||
TransformTypeRegex = "regex" // 正则提取
|
||||
)
|
||||
|
||||
// 业务域
|
||||
// 业务域（暂未使用，保留供未来扩展）
|
||||
const (
|
||||
BusinessDomainUser = "user" // 用户
|
||||
BusinessDomainOrder = "order" // 订单
|
||||
@@ -18,7 +24,7 @@ const (
|
||||
BusinessDomainPayment = "payment" // 支付
|
||||
)
|
||||
|
||||
// 字段类型
|
||||
// 字段类型（暂未使用，保留供未来扩展）
|
||||
const (
|
||||
FieldTypeString = "string"
|
||||
FieldTypeNumber = "number"
|
||||
@@ -30,22 +36,3 @@ const (
|
||||
FieldTypeTime = "time"
|
||||
FieldTypeDateTime = "datetime"
|
||||
)
|
||||
|
||||
// 平台状态
|
||||
type PlatformStatus string
|
||||
|
||||
const (
|
||||
PlatformStatusActive PlatformStatus = "active"
|
||||
PlatformStatusInactive PlatformStatus = "inactive"
|
||||
)
|
||||
|
||||
// 接口方法
|
||||
type ApiMethod string
|
||||
|
||||
const (
|
||||
ApiMethodGET ApiMethod = "GET"
|
||||
ApiMethodPOST ApiMethod = "POST"
|
||||
ApiMethodPUT ApiMethod = "PUT"
|
||||
ApiMethodDELETE ApiMethod = "DELETE"
|
||||
ApiMethodPATCH ApiMethod = "PATCH"
|
||||
)
|
||||
|
||||
@@ -0,0 +1,41 @@
|
||||
package public
|
||||
|
||||
import (
|
||||
"context"
|
||||
|
||||
dto "dataengine/model/dto/public"
|
||||
svc "dataengine/service/public"
|
||||
|
||||
"github.com/gogf/gf/v2/frame/g"
|
||||
)
|
||||
|
||||
type publicQueryController struct{}
|
||||
|
||||
// PublicQuery 公共查询控制器
|
||||
var PublicQuery = new(publicQueryController)
|
||||
|
||||
// QueryReq 查询请求（包含空结构用于路径）
|
||||
type ClearCacheReq struct {
|
||||
g.Meta `path:"/public/cache/clear" method:"delete" tags:"公共查询" summary:"清除查询缓存"`
|
||||
}
|
||||
|
||||
// Query 执行公共查询
|
||||
func (c *publicQueryController) Query(ctx context.Context, req *dto.QueryReq) (res *dto.QueryRes, err error) {
|
||||
return svc.PublicQuery.Query(ctx, req)
|
||||
}
|
||||
|
||||
// GetTableList 获取可查询表列表
|
||||
func (c *publicQueryController) GetTableList(ctx context.Context, req *dto.TableListReq) (res *dto.TableListRes, err error) {
|
||||
return svc.PublicQuery.GetTableList(ctx)
|
||||
}
|
||||
|
||||
// GetColumnList 获取表字段列表
|
||||
func (c *publicQueryController) GetColumnList(ctx context.Context, req *dto.ColumnListReq) (res *dto.ColumnListRes, err error) {
|
||||
return svc.PublicQuery.GetColumnList(ctx, req.Table)
|
||||
}
|
||||
|
||||
// ClearCache 清除查询缓存（管理接口）
|
||||
func (c *publicQueryController) ClearCache(ctx context.Context, req *ClearCacheReq) (res *dto.TableListRes, err error) {
|
||||
svc.PublicQuery.ClearTableCache()
|
||||
return &dto.TableListRes{List: []dto.TableInfo{}}, nil
|
||||
}
|
||||
@@ -0,0 +1,949 @@
|
||||
package report
|
||||
|
||||
import (
|
||||
"github.com/gogf/gf/v2/net/ghttp"
|
||||
)
|
||||
|
||||
// ReportAdminPage 报表引擎管理页面
|
||||
func ReportAdminPage(r *ghttp.Request) {
|
||||
r.Response.Header().Set("Content-Type", "text/html; charset=utf-8")
|
||||
r.Response.Write(reportAdminHTML)
|
||||
r.Exit()
|
||||
}
|
||||
|
||||
var reportAdminHTML = `<!DOCTYPE html>
|
||||
<html lang="zh-CN">
|
||||
<head>
|
||||
<meta charset="UTF-8">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1.0">
|
||||
<title>报表引擎管理</title>
|
||||
<style>
|
||||
* { margin: 0; padding: 0; box-sizing: border-box; }
|
||||
body { font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, sans-serif; background: #f0f2f5; color: #333; }
|
||||
.header { background: #001529; color: #fff; padding: 12px 24px; display: flex; align-items: center; gap: 16px; }
|
||||
.header h1 { font-size: 18px; font-weight: 600; }
|
||||
.header a { color: #91a7ff; text-decoration: none; font-size: 13px; }
|
||||
.container { max-width: 1500px; margin: 0 auto; padding: 16px; }
|
||||
.tabs { display: flex; gap: 0; margin-bottom: 16px; background: #fff; border-radius: 8px; overflow: hidden; box-shadow: 0 1px 4px rgba(0,0,0,0.06); }
|
||||
.tab { padding: 10px 24px; cursor: pointer; font-size: 13px; color: #666; border-bottom: 2px solid transparent; transition: all .2s; }
|
||||
.tab:hover { color: #1890ff; }
|
||||
.tab.active { color: #1890ff; border-bottom-color: #1890ff; font-weight: 600; }
|
||||
.panel { display: none; }
|
||||
.panel.active { display: block; }
|
||||
.toolbar { display: flex; justify-content: space-between; align-items: center; margin-bottom: 12px; flex-wrap: wrap; gap: 8px; }
|
||||
.toolbar h2 { font-size: 15px; color: #1a1a2e; }
|
||||
.toolbar-left { display: flex; align-items: center; gap: 8px; }
|
||||
.toolbar-right { display: flex; gap: 6px; }
|
||||
.btn { display: inline-flex; align-items: center; gap: 4px; padding: 6px 14px; border: none; border-radius: 6px; font-size: 12px; cursor: pointer; transition: all .2s; white-space: nowrap; }
|
||||
.btn-primary { background: #1890ff; color: #fff; }
|
||||
.btn-primary:hover { background: #40a9ff; }
|
||||
.btn-danger { background: #ff4d4f; color: #fff; }
|
||||
.btn-danger:hover { background: #ff7875; }
|
||||
.btn-success { background: #52c41a; color: #fff; }
|
||||
.btn-success:hover { background: #73d13d; }
|
||||
.btn-warning { background: #faad14; color: #fff; }
|
||||
.btn-warning:hover { background: #ffc53d; }
|
||||
.btn-sm { padding: 4px 10px; font-size: 11px; }
|
||||
.btn-outline { background: #fff; border: 1px solid #d9d9d9; color: #333; }
|
||||
.btn-outline:hover { border-color: #1890ff; color: #1890ff; }
|
||||
table { width: 100%; background: #fff; border-radius: 8px; overflow: hidden; box-shadow: 0 1px 4px rgba(0,0,0,0.06); border-collapse: collapse; }
|
||||
th { background: #fafafa; padding: 10px 14px; text-align: left; font-size: 12px; color: #666; font-weight: 600; border-bottom: 1px solid #f0f0f0; }
|
||||
td { padding: 10px 14px; font-size: 12px; border-bottom: 1px solid #f0f0f0; }
|
||||
tr:hover td { background: #fafafa; }
|
||||
.badge { display: inline-block; padding: 2px 8px; border-radius: 10px; font-size: 11px; font-weight: 500; }
|
||||
.badge.active { background: #f6ffed; color: #52c41a; border: 1px solid #b7eb8f; }
|
||||
.badge.inactive { background: #fff2f0; color: #ff4d4f; border: 1px solid #ffccc7; }
|
||||
.badge.info { background: #e6f7ff; color: #1890ff; border: 1px solid #91d5ff; }
|
||||
.actions { display: flex; gap: 4px; }
|
||||
.modal-overlay { display: none; position: fixed; top: 0; left: 0; right: 0; bottom: 0; background: rgba(0,0,0,0.45); z-index: 1000; }
|
||||
.modal-overlay.open { display: block; }
|
||||
.modal { display: none; position: fixed; top: 50%; left: 50%; transform: translate(-50%,-50%); background: #fff; border-radius: 8px; width: 720px; max-width: 95%; max-height: 88vh; overflow-y: auto; z-index: 1001; box-shadow: 0 4px 24px rgba(0,0,0,0.15); }
|
||||
.modal.open { display: block; }
|
||||
.modal-header { padding: 14px 20px; border-bottom: 1px solid #f0f0f0; display: flex; justify-content: space-between; align-items: center; position: sticky; top: 0; background: #fff; z-index: 1; }
|
||||
.modal-header h3 { font-size: 15px; }
|
||||
.modal-close { background: none; border: none; font-size: 20px; cursor: pointer; color: #999; }
|
||||
.modal-body { padding: 20px; }
|
||||
.modal-footer { padding: 12px 20px; border-top: 1px solid #f0f0f0; display: flex; justify-content: flex-end; gap: 8px; position: sticky; bottom: 0; background: #fff; }
|
||||
.form-group { margin-bottom: 12px; }
|
||||
.form-group label { display: block; font-size: 12px; color: #666; margin-bottom: 4px; font-weight: 500; }
|
||||
.form-group input, .form-group select, .form-group textarea { width: 100%; padding: 7px 10px; border: 1px solid #d9d9d9; border-radius: 6px; font-size: 12px; outline: none; transition: border .2s; }
|
||||
.form-group input:focus, .form-group select:focus, .form-group textarea:focus { border-color: #1890ff; box-shadow: 0 0 0 2px rgba(24,144,255,0.1); }
|
||||
.form-group textarea { font-family: "Fira Code", "Consolas", monospace; font-size: 11px; min-height: 60px; resize: vertical; }
|
||||
.form-row { display: grid; grid-template-columns: 1fr 1fr; gap: 10px; }
|
||||
.form-row-3 { display: grid; grid-template-columns: 1fr 1fr 1fr; gap: 10px; }
|
||||
.form-inline { display: flex; gap: 8px; align-items: flex-end; flex-wrap: wrap; }
|
||||
.form-inline .form-group { margin-bottom: 0; flex: 1; min-width: 150px; }
|
||||
.checkbox-group { display: flex; flex-wrap: wrap; gap: 8px; }
|
||||
.checkbox-group label { display: flex; align-items: center; gap: 4px; font-size: 12px; cursor: pointer; font-weight: normal; }
|
||||
.select-biz { padding: 7px 10px; border: 1px solid #d9d9d9; border-radius: 6px; font-size: 12px; }
|
||||
.loading { text-align: center; padding: 40px; color: #999; }
|
||||
.empty { text-align: center; padding: 32px; color: #999; font-size: 13px; }
|
||||
.toast { position: fixed; top: 16px; right: 16px; padding: 10px 20px; border-radius: 6px; color: #fff; font-size: 13px; z-index: 2000; animation: slideIn .3s; }
|
||||
.toast.success { background: #52c41a; }
|
||||
.toast.error { background: #ff4d4f; }
|
||||
.toast.info { background: #1890ff; }
|
||||
@keyframes slideIn { from { transform: translateX(100%); opacity: 0; } to { transform: translateX(0); opacity: 1; } }
|
||||
.result-box { margin-top: 12px; background: #fafafa; border-radius: 6px; padding: 12px; max-height: 400px; overflow: auto; font-size: 12px; font-family: monospace; white-space: pre-wrap; }
|
||||
.query-section { background: #fff; border-radius: 8px; padding: 16px; box-shadow: 0 1px 4px rgba(0,0,0,0.06); margin-bottom: 12px; }
|
||||
.query-section h3 { font-size: 14px; margin-bottom: 10px; color: #333; }
|
||||
.chip { display: inline-flex; align-items: center; gap: 4px; padding: 2px 10px; background: #f0f5ff; border: 1px solid #adc6ff; border-radius: 12px; font-size: 11px; cursor: pointer; }
|
||||
.chip.selected { background: #1890ff; color: #fff; border-color: #1890ff; }
|
||||
.chip-group { display: flex; flex-wrap: wrap; gap: 6px; }
|
||||
.field-tag { display: inline-flex; align-items: center; background: #f6ffed; border: 1px solid #b7eb8f; border-radius: 4px; padding: 2px 8px; font-size: 11px; margin: 2px; }
|
||||
.field-tag .remove { margin-left: 4px; cursor: pointer; color: #ff4d4f; font-weight: bold; }
|
||||
.add-filter-row { display: grid; grid-template-columns: 1fr 1fr 1.5fr 40px; gap: 6px; align-items: end; margin-bottom: 6px; }
|
||||
.section-label { font-size: 12px; color: #999; margin-bottom: 6px; }
|
||||
</style>
|
||||
</head>
|
||||
<body>
|
||||
<div class="header">
|
||||
<h1>📊 报表引擎管理</h1>
|
||||
<a href="/admin">← 返回管理后台</a>
|
||||
</div>
|
||||
<div class="container">
|
||||
|
||||
<div class="tabs">
|
||||
<div class="tab active" onclick="switchTab('business',this)">业务管理</div>
|
||||
<div class="tab" onclick="switchTab('report',this)">报表配置</div>
|
||||
<div class="tab" onclick="switchTab('field',this)">字段配置</div>
|
||||
<div class="tab" onclick="switchTab('extract',this)">抽取配置</div>
|
||||
<div class="tab" onclick="switchTab('query',this)">数据查询</div>
|
||||
</div>
|
||||
|
||||
<!-- ========== 业务管理 ========== -->
|
||||
<div id="tab-business" class="panel active">
|
||||
<div class="toolbar">
|
||||
<div class="toolbar-left"><h2>业务列表</h2></div>
|
||||
<div class="toolbar-right">
|
||||
<button class="btn btn-outline btn-sm" onclick="initTables()">初始化系统表</button>
|
||||
<button class="btn btn-primary btn-sm" onclick="openBizModal()">+ 新建业务</button>
|
||||
</div>
|
||||
</div>
|
||||
<div id="biz-loading" class="loading">加载中...</div>
|
||||
<div id="biz-list"></div>
|
||||
</div>
|
||||
|
||||
<!-- ========== 报表配置 ========== -->
|
||||
<div id="tab-report" class="panel">
|
||||
<div class="toolbar">
|
||||
<div class="toolbar-left">
|
||||
<h2>报表列表</h2>
|
||||
<select class="select-biz" id="report-biz-select" onchange="loadReportList()">
|
||||
<option value="">-- 选择业务 --</option>
|
||||
</select>
|
||||
</div>
|
||||
<div class="toolbar-right">
|
||||
<button class="btn btn-primary btn-sm" onclick="openReportModal()">+ 新建报表</button>
|
||||
</div>
|
||||
</div>
|
||||
<div id="report-loading" class="loading" style="display:none">加载中...</div>
|
||||
<div id="report-list"></div>
|
||||
</div>
|
||||
|
||||
<!-- ========== 字段配置 ========== -->
|
||||
<div id="tab-field" class="panel">
|
||||
<div class="toolbar">
|
||||
<div class="toolbar-left">
|
||||
<h2>字段列表</h2>
|
||||
<select class="select-biz" id="field-biz-select" onchange="onFieldBizChange()">
|
||||
<option value="">-- 选择业务 --</option>
|
||||
</select>
|
||||
<select class="select-biz" id="field-report-select" onchange="loadFieldList()">
|
||||
<option value="">-- 选择报表 --</option>
|
||||
</select>
|
||||
</div>
|
||||
<div class="toolbar-right">
|
||||
<button class="btn btn-primary btn-sm" onclick="openFieldModal()">+ 新建字段</button>
|
||||
</div>
|
||||
</div>
|
||||
<div id="field-loading" class="loading" style="display:none">加载中...</div>
|
||||
<div id="field-list"></div>
|
||||
</div>
|
||||
|
||||
<!-- ========== 抽取配置 ========== -->
|
||||
<div id="tab-extract" class="panel">
|
||||
<div class="toolbar">
|
||||
<div class="toolbar-left">
|
||||
<h2>抽取配置列表</h2>
|
||||
<select class="select-biz" id="extract-biz-select" onchange="onExtractBizChange()">
|
||||
<option value="">-- 选择业务 --</option>
|
||||
</select>
|
||||
<select class="select-biz" id="extract-report-select" onchange="loadExtractList()">
|
||||
<option value="">-- 选择报表 --</option>
|
||||
</select>
|
||||
</div>
|
||||
<div class="toolbar-right">
|
||||
<button class="btn btn-success btn-sm" onclick="execExtract()">执行抽取</button>
|
||||
<button class="btn btn-primary btn-sm" onclick="openExtractModal()">+ 新建抽取配置</button>
|
||||
</div>
|
||||
</div>
|
||||
<div id="extract-loading" class="loading" style="display:none">加载中...</div>
|
||||
<div id="extract-list"></div>
|
||||
</div>
|
||||
|
||||
<!-- ========== 数据查询 ========== -->
|
||||
<div id="tab-query" class="panel">
|
||||
<div class="toolbar">
|
||||
<div class="toolbar-left">
|
||||
<h2>数据查询</h2>
|
||||
<select class="select-biz" id="query-biz-select" onchange="onQueryBizChange()">
|
||||
<option value="">-- 选择业务 --</option>
|
||||
</select>
|
||||
<select class="select-biz" id="query-report-select" onchange="loadQueryFields()">
|
||||
<option value="">-- 选择报表 --</option>
|
||||
</select>
|
||||
</div>
|
||||
</div>
|
||||
<div id="query-content" style="display:none">
|
||||
<div class="query-section">
|
||||
<h3>📊 统计维度</h3>
|
||||
<div id="dimension-chips" class="chip-group"></div>
|
||||
</div>
|
||||
<div class="query-section">
|
||||
<h3>📈 统计指标（点击后选择聚合方式）</h3>
|
||||
<div id="indicator-area"></div>
|
||||
<div id="selected-indicators" style="margin-top:8px"></div>
|
||||
</div>
|
||||
<div class="query-section">
|
||||
<h3>🔍 筛选条件</h3>
|
||||
<div id="filter-area"></div>
|
||||
</div>
|
||||
<div class="query-section">
|
||||
<h3>📅 时间范围</h3>
|
||||
<div class="form-inline">
|
||||
<div class="form-group"><label>开始日期</label><input type="date" id="query-start-date"></div>
|
||||
<div class="form-group"><label>结束日期</label><input type="date" id="query-end-date"></div>
|
||||
<div class="form-group"><label>排序字段</label><input id="query-order-field" placeholder="如 total_amount"></div>
|
||||
<div class="form-group"><label>排序方向</label><select id="query-order-dir"><option value="DESC">降序</option><option value="ASC">升序</option></select></div>
|
||||
<div class="form-group"><label>每页</label><input type="number" id="query-page-size" value="20" style="width:70px"></div>
|
||||
<div><button class="btn btn-primary" onclick="executeQuery()" style="margin-top:22px">查询</button></div>
|
||||
</div>
|
||||
</div>
|
||||
<div id="query-result" class="result-box" style="display:none"></div>
|
||||
</div>
|
||||
<div id="query-empty" class="empty" style="display:none">请先选择业务和报表</div>
|
||||
</div>
|
||||
|
||||
</div><!-- container -->
|
||||
|
||||
<!-- Modal -->
|
||||
<div class="modal-overlay" id="modalOverlay" onclick="closeModal()"></div>
|
||||
<div class="modal" id="modal">
|
||||
<div class="modal-header"><h3 id="modalTitle"></h3><button class="modal-close" onclick="closeModal()">×</button></div>
|
||||
<div class="modal-body" id="modalBody"></div>
|
||||
<div class="modal-footer" id="modalFooter"></div>
|
||||
</div>
|
||||
|
||||
<script>
|
||||
const BASE = '';
|
||||
let state = { businesses: [], currentBiz: null };
|
||||
|
||||
function toast(msg, type) {
|
||||
const t = document.createElement('div');
|
||||
t.className = 'toast ' + (type||'info');
|
||||
t.textContent = msg;
|
||||
document.body.appendChild(t);
|
||||
setTimeout(() => t.remove(), 2500);
|
||||
}
|
||||
|
||||
async function api(method, url, body) {
|
||||
const opt = { method, headers: {} };
|
||||
if (body) { opt.headers['Content-Type'] = 'application/json'; opt.body = JSON.stringify(body); }
|
||||
const resp = await fetch(BASE + url, opt);
|
||||
if (!resp.ok) { const t = await resp.text(); throw new Error(t || resp.statusText); }
|
||||
const json = await resp.json();
|
||||
console.log('[API] ' + method + ' ' + url, 'raw response:', json);
|
||||
// GoFrame MiddlewareHandlerResponse 包裹: {code:0, data: <returnValue>}
|
||||
// returnValue 可能是 {data: {...}} 或 {list: [...]} 等
|
||||
if (json.code !== undefined && json.data !== undefined) {
|
||||
if (json.code !== 0) throw new Error(json.message || '请求失败');
|
||||
console.log('[API] extracted data:', json.data);
|
||||
// 兼容 GoFrame 的 data 可能包含首字母大写的 Go 字段名
|
||||
// 如 getBusinessRes{Data: ...} 在某些版本可能序列化为 {"Data": ...}
|
||||
// 此时尝试统一：若有 Data 字段且无 data 字段，取 Data
|
||||
if (json.data && json.data.Data !== undefined && json.data.data === undefined) {
|
||||
return json.data.Data;
|
||||
}
|
||||
return json.data;
|
||||
}
|
||||
return json;
|
||||
}
|
||||
|
||||
function esc(s) {
|
||||
if (s == null || s === undefined) return '';
|
||||
if (typeof s === 'object') s = JSON.stringify(s);
|
||||
const d = document.createElement('div');
|
||||
d.textContent = String(s);
|
||||
return d.innerHTML;
|
||||
}
|
||||
function jsAttr(s) {
|
||||
// 用于 onclick 等 JS 属性中的字符串值转义（替换反斜杠和单引号）
|
||||
if (s == null || s === undefined) return '';
|
||||
return String(s).replace(/\\/g, '\\\\').replace(/'/g, "\\'");
|
||||
}
|
||||
function v(id) { return document.getElementById(id).value; }
|
||||
|
||||
async function initTables() {
|
||||
try {
|
||||
await api('POST', '/report/initTables');
|
||||
toast('系统表初始化完成', 'success');
|
||||
} catch(e) { toast(e.message, 'error'); }
|
||||
}
|
||||
|
||||
// ==================== 导航 ====================
|
||||
function switchTab(name, tabEl) {
|
||||
document.querySelectorAll('.tab').forEach(t => t.classList.remove('active'));
|
||||
document.querySelectorAll('.panel').forEach(p => p.classList.remove('active'));
|
||||
if (tabEl) tabEl.classList.add('active');
|
||||
document.getElementById('tab-' + name).classList.add('active');
|
||||
if (name === 'business') loadBizList();
|
||||
else if (name === 'report') loadBizSelects().then(() => loadReportList());
|
||||
else if (name === 'field') loadBizSelects().then(() => onFieldBizChange());
|
||||
else if (name === 'extract') loadBizSelects().then(() => onExtractBizChange());
|
||||
else if (name === 'query') loadBizSelects().then(() => { document.getElementById('query-content').style.display='none'; document.getElementById('query-empty').style.display='block'; });
|
||||
}
|
||||
|
||||
// ==================== 业务选择器 ====================
|
||||
async function loadBizSelects() {
|
||||
try {
|
||||
const data = await api('GET', '/report/businesses');
|
||||
state.businesses = data.list || [];
|
||||
const optHtml = '<option value="">-- 选择业务 --</option>' + state.businesses.map(b => '<option value="'+b.businessCode+'">'+esc(b.businessName)+' ('+b.businessCode+')</option>').join('');
|
||||
['report-biz-select','field-biz-select','extract-biz-select','query-biz-select'].forEach(id => {
|
||||
const sel = document.getElementById(id);
|
||||
const oldVal = sel.value;
|
||||
sel.innerHTML = optHtml;
|
||||
if (oldVal) sel.value = oldVal;
|
||||
});
|
||||
} catch(e) { toast(e.message, 'error'); }
|
||||
}
|
||||
|
||||
async function onFieldBizChange() {
|
||||
const bc = v('field-biz-select');
|
||||
const rs = document.getElementById('field-report-select');
|
||||
rs.innerHTML = '<option value="">-- 选择报表 --</option>';
|
||||
if (!bc) return;
|
||||
try {
|
||||
const data = await api('GET', '/report/reports?businessCode=' + bc);
|
||||
rs.innerHTML = '<option value="">-- 选择报表 --</option>' + (data.list||[]).map(r => '<option value="'+r.reportCode+'">'+esc(r.reportName)+' ('+r.reportCode+')</option>').join('');
|
||||
} catch(e) { toast(e.message, 'error'); }
|
||||
}
|
||||
|
||||
async function onExtractBizChange() {
|
||||
const bc = v('extract-biz-select');
|
||||
const rs = document.getElementById('extract-report-select');
|
||||
rs.innerHTML = '<option value="">-- 选择报表 --</option>';
|
||||
if (!bc) return;
|
||||
try {
|
||||
const data = await api('GET', '/report/reports?businessCode=' + bc);
|
||||
rs.innerHTML = '<option value="">-- 选择报表 --</option>' + (data.list||[]).map(r => '<option value="'+r.reportCode+'">'+esc(r.reportName)+' ('+r.reportCode+')</option>').join('');
|
||||
} catch(e) { toast(e.message, 'error'); }
|
||||
}
|
||||
|
||||
// ==================== 业务 CRUD ====================
|
||||
async function loadBizList() {
|
||||
document.getElementById('biz-loading').style.display = 'block';
|
||||
try {
|
||||
const data = await api('GET', '/report/businesses');
|
||||
document.getElementById('biz-loading').style.display = 'none';
|
||||
const list = data.list || [];
|
||||
if (list.length === 0) {
|
||||
document.getElementById('biz-list').innerHTML = '<div class="empty">暂无业务，点击"新建业务"添加</div>';
|
||||
return;
|
||||
}
|
||||
let h = '<table><thead><tr><th>ID</th><th>编码</th><th>名称</th><th>描述</th><th>创建人</th><th>状态</th><th>操作</th></tr></thead><tbody>';
|
||||
for (const b of list) {
|
||||
const sc = b.status === 'ACTIVE' ? 'active' : 'inactive';
|
||||
h += '<tr><td>'+b.id+'</td><td><strong>'+esc(b.businessCode)+'</strong></td><td>'+esc(b.businessName)+'</td>' +
|
||||
'<td style="max-width:200px;overflow:hidden;text-overflow:ellipsis;white-space:nowrap">'+esc(b.description||'-')+'</td>' +
|
||||
'<td>'+esc(b.creator||'-')+'</td>' +
|
||||
'<td><span class="badge '+sc+'">'+(b.status||'ACTIVE')+'</span></td>' +
|
||||
'<td class="actions">' +
|
||||
'<button class="btn btn-sm btn-outline" onclick="openBizModal('+b.id+')">编辑</button>' +
|
||||
'<button class="btn btn-sm btn-danger" onclick="deleteBiz('+b.id+')">删除</button></td></tr>';
|
||||
}
|
||||
h += '</tbody></table>';
|
||||
document.getElementById('biz-list').innerHTML = h;
|
||||
} catch(e) {
|
||||
document.getElementById('biz-loading').style.display = 'none';
|
||||
toast(e.message, 'error');
|
||||
}
|
||||
}
|
||||
|
||||
function bizFormHTML(d) {
|
||||
d = d || {};
|
||||
return '<div class="form-row"><div class="form-group"><label>业务编码 *</label><input id="f-bcode" value="'+esc(d.businessCode||'')+'" placeholder="如 KUAISHOU"></div>' +
|
||||
'<div class="form-group"><label>业务名称 *</label><input id="f-bname" value="'+esc(d.businessName||'')+'" placeholder="如 快手电商"></div></div>' +
|
||||
'<div class="form-group"><label>描述</label><input id="f-bdesc" value="'+esc(d.description||'')+'" placeholder="业务描述"></div>' +
|
||||
'<div class="form-row"><div class="form-group"><label>操作人</label><input id="f-bop" value="'+esc(d.creator||'admin')+'"></div>' +
|
||||
'<div class="form-group"><label>状态</label><select id="f-bs"><option value="ACTIVE"'+((d.status||'ACTIVE')==='ACTIVE'?' selected':'')+'>启用</option><option value="INACTIVE"'+(d.status==='INACTIVE'?' selected':'')+'>停用</option></select></div></div>';
|
||||
}
|
||||
|
||||
async function openBizModal(id) {
|
||||
const title = document.getElementById('modalTitle');
|
||||
const body = document.getElementById('modalBody');
|
||||
const footer = document.getElementById('modalFooter');
|
||||
if (id) {
|
||||
try {
|
||||
const res = await api('GET', '/report/business?id=' + id);
|
||||
const d = res && res.data !== undefined ? res.data : (res && res.Data !== undefined ? res.Data : res);
|
||||
console.log('[BizEdit] id=' + id + ' raw:', res, 'data:', d);
|
||||
if (!d || !d.id) { toast('未获取到业务数据（ID='+id+'）', 'error'); return; }
|
||||
title.textContent = '编辑业务';
|
||||
body.innerHTML = bizFormHTML(d);
|
||||
footer.innerHTML = '<button class="btn btn-outline" onclick="closeModal()">取消</button><button class="btn btn-primary" onclick="saveBiz('+id+')">保存</button>';
|
||||
} catch(e) { toast(e.message, 'error'); return; }
|
||||
} else {
|
||||
title.textContent = '新建业务';
|
||||
body.innerHTML = bizFormHTML(null);
|
||||
footer.innerHTML = '<button class="btn btn-outline" onclick="closeModal()">取消</button><button class="btn btn-primary" onclick="saveBiz(0)">创建</button>';
|
||||
}
|
||||
document.getElementById('modalOverlay').classList.add('open');
|
||||
document.getElementById('modal').classList.add('open');
|
||||
}
|
||||
|
||||
async function saveBiz(id) {
|
||||
const body = {
|
||||
businessCode: v('f-bcode'), businessName: v('f-bname'),
|
||||
description: v('f-bdesc'), status: v('f-bs'), operator: v('f-bop'),
|
||||
};
|
||||
if (id) body.id = id;
|
||||
try {
|
||||
await api('POST', '/report/business/save', body);
|
||||
toast(id ? '更新成功' : '创建成功', 'success');
|
||||
closeModal(); loadBizList(); loadBizSelects();
|
||||
} catch(e) { toast(e.message, 'error'); }
|
||||
}
|
||||
|
||||
async function deleteBiz(id) {
|
||||
if (!confirm('确定删除此业务？')) return;
|
||||
try { await api('DELETE', '/report/business?id='+id); toast('删除成功', 'success'); loadBizList(); loadBizSelects(); }
|
||||
catch(e) { toast(e.message, 'error'); }
|
||||
}
|
||||
|
||||
// ==================== 报表 CRUD ====================
|
||||
async function loadReportList() {
|
||||
const bc = v('report-biz-select');
|
||||
if (!bc) { document.getElementById('report-list').innerHTML = '<div class="empty">请先选择业务</div>'; return; }
|
||||
document.getElementById('report-loading').style.display = 'block';
|
||||
try {
|
||||
const data = await api('GET', '/report/reports?businessCode=' + bc);
|
||||
document.getElementById('report-loading').style.display = 'none';
|
||||
const list = data.list || [];
|
||||
if (list.length === 0) {
|
||||
document.getElementById('report-list').innerHTML = '<div class="empty">暂无报表</div>';
|
||||
return;
|
||||
}
|
||||
let h = '<table><thead><tr><th>ID</th><th>编码</th><th>名称</th><th>统计宽表</th><th>日期字段</th><th>冲突键</th><th>状态</th><th>操作</th></tr></thead><tbody>';
|
||||
for (const r of list) {
|
||||
const sc = r.status === 'ACTIVE' ? 'active' : 'inactive';
|
||||
h += '<tr><td>'+r.id+'</td><td><strong>'+esc(r.reportCode)+'</strong></td><td>'+esc(r.reportName)+'</td>' +
|
||||
'<td style="font-family:monospace;font-size:11px">'+esc(r.statTableName)+'</td>' +
|
||||
'<td>'+esc(r.dateField||'stat_date')+'</td>' +
|
||||
'<td style="font-size:11px">'+esc((r.conflictKeys||[]).join(', '))+'</td>' +
|
||||
'<td><span class="badge '+sc+'">'+(r.status||'ACTIVE')+'</span></td>' +
|
||||
'<td class="actions">' +
|
||||
'<button class="btn btn-sm btn-outline" onclick="openReportModal('+r.id+')">编辑</button>' +
|
||||
'<button class="btn btn-sm btn-danger" onclick="deleteReport('+r.id+')">删除</button></td></tr>';
|
||||
}
|
||||
h += '</tbody></table>';
|
||||
document.getElementById('report-list').innerHTML = h;
|
||||
} catch(e) { document.getElementById('report-loading').style.display = 'none'; toast(e.message, 'error'); }
|
||||
}
|
||||
|
||||
function reportFormHTML(d) {
|
||||
d = d || {};
|
||||
return '<div class="form-group"><label>所属业务</label><input id="f-rbiz" value="'+esc(d.businessCode||v('report-biz-select')||'')+'" readonly style="background:#f5f5f5"></div>' +
|
||||
'<div class="form-row"><div class="form-group"><label>报表编码 *</label><input id="f-rcode" value="'+esc(d.reportCode||'')+'" placeholder="如 shop_daily_report"></div>' +
|
||||
'<div class="form-group"><label>报表名称 *</label><input id="f-rname" value="'+esc(d.reportName||'')+'" placeholder="如 店铺日报"></div></div>' +
|
||||
'<div class="form-group"><label>描述</label><input id="f-rdesc" value="'+esc(d.description||'')+'"></div>' +
|
||||
'<div class="form-row"><div class="form-group"><label>统计宽表名 *</label><input id="f-rtable" value="'+esc(d.statTableName||'')+'" placeholder="如 stat_kuaishou_shop_daily"></div>' +
|
||||
'<div class="form-group"><label>统计宽表注释</label><input id="f-rtcomment" value="'+esc(d.statTableComment||'')+'"></div></div>' +
|
||||
'<div class="form-row"><div class="form-group"><label>日期字段</label><input id="f-rdate" value="'+esc(d.dateField||'stat_date')+'"></div>' +
|
||||
'<div class="form-group"><label>操作人</label><input id="f-rop" value="'+esc(d.creator||'admin')+'"></div></div>' +
|
||||
'<div class="form-group"><label>冲突键（逗号分隔）</label><input id="f-rck" value="'+esc((d.conflictKeys||[]).join(','))+'"></div>' +
|
||||
'<div class="form-group"><label>主键（逗号分隔）</label><input id="f-rpk" value="'+esc((d.primaryKeys||['id']).join(','))+'"></div>' +
|
||||
'<div class="form-row"><div class="form-group"><label>状态</label><select id="f-rs"><option value="ACTIVE"'+((d.status||'ACTIVE')==='ACTIVE'?' selected':'')+'>启用</option><option value="INACTIVE"'+(d.status==='INACTIVE'?' selected':'')+'>停用</option></select></div></div>';
|
||||
}
|
||||
|
||||
async function openReportModal(id) {
|
||||
const title = document.getElementById('modalTitle');
|
||||
const body = document.getElementById('modalBody');
|
||||
const footer = document.getElementById('modalFooter');
|
||||
if (id) {
|
||||
try {
|
||||
const res = await api('GET', '/report/report?id=' + id);
|
||||
const d = res && res.data !== undefined ? res.data : (res && res.Data !== undefined ? res.Data : res);
|
||||
console.log('[ReportEdit] id=' + id + ' raw:', res, 'data:', d);
|
||||
if (!d || !d.id) { toast('未获取到报表数据（ID='+id+'）', 'error'); return; }
|
||||
title.textContent = '编辑报表';
|
||||
body.innerHTML = reportFormHTML(d);
|
||||
footer.innerHTML = '<button class="btn btn-outline" onclick="closeModal()">取消</button><button class="btn btn-primary" onclick="saveReport('+id+')">保存</button>';
|
||||
} catch(e) { toast(e.message, 'error'); return; }
|
||||
} else {
|
||||
title.textContent = '新建报表';
|
||||
body.innerHTML = reportFormHTML(null);
|
||||
footer.innerHTML = '<button class="btn btn-outline" onclick="closeModal()">取消</button><button class="btn btn-primary" onclick="saveReport(0)">创建</button>';
|
||||
}
|
||||
document.getElementById('modalOverlay').classList.add('open');
|
||||
document.getElementById('modal').classList.add('open');
|
||||
}
|
||||
|
||||
async function saveReport(id) {
|
||||
const body = {
|
||||
businessCode: v('f-rbiz'), reportCode: v('f-rcode'), reportName: v('f-rname'),
|
||||
description: v('f-rdesc'), statTableName: v('f-rtable'), statTableComment: v('f-rtcomment'),
|
||||
dateField: v('f-rdate'), operator: v('f-rop'), status: v('f-rs'),
|
||||
conflictKeys: v('f-rck').split(',').map(s => s.trim()).filter(Boolean),
|
||||
primaryKeys: v('f-rpk').split(',').map(s => s.trim()).filter(Boolean),
|
||||
};
|
||||
if (id) body.id = id;
|
||||
try {
|
||||
await api('POST', '/report/report/save', body);
|
||||
toast(id ? '更新成功' : '创建成功', 'success');
|
||||
closeModal(); loadReportList();
|
||||
} catch(e) { toast(e.message, 'error'); }
|
||||
}
|
||||
|
||||
async function deleteReport(id) {
|
||||
if (!confirm('确定删除此报表？')) return;
|
||||
try { await api('DELETE', '/report/report?id='+id); toast('删除成功', 'success'); loadReportList(); }
|
||||
catch(e) { toast(e.message, 'error'); }
|
||||
}
|
||||
|
||||
// ==================== 字段 CRUD ====================
|
||||
async function loadFieldList() {
|
||||
const bc = v('field-biz-select');
|
||||
const rc = v('field-report-select');
|
||||
if (!bc || !rc) { document.getElementById('field-list').innerHTML = '<div class="empty">请先选择业务和报表</div>'; return; }
|
||||
document.getElementById('field-loading').style.display = 'block';
|
||||
try {
|
||||
const data = await api('GET', '/report/fields?businessCode=' + bc + '&reportCode=' + rc);
|
||||
document.getElementById('field-loading').style.display = 'none';
|
||||
const dims = data.dimensions || [], inds = data.indicators || [], fils = data.filters || [];
|
||||
if (dims.length + inds.length + fils.length === 0) {
|
||||
document.getElementById('field-list').innerHTML = '<div class="empty">暂无字段</div>';
|
||||
return;
|
||||
}
|
||||
const allFields = [...dims.map(f => ({...f, _role: '维度'})), ...inds.map(f => ({...f, _role: '指标'})), ...fils.map(f => ({...f, _role: '筛选'}))];
|
||||
let h = '<table><thead><tr><th>ID</th><th>角色</th><th>编码</th><th>名称</th><th>类型</th><th>可聚合</th><th>默认聚合</th><th>分组</th><th>排序</th><th>状态</th><th>操作</th></tr></thead><tbody>';
|
||||
for (const f of allFields) {
|
||||
const sc = f.status === 'ACTIVE' ? 'active' : 'inactive';
|
||||
h += '<tr><td>'+f.id+'</td><td><span class="badge '+(f.fieldRole==='DIMENSION'?'info':f.fieldRole==='INDICATOR'?'success':'')+'">'+f._role+'</span></td>' +
|
||||
'<td><strong>'+esc(f.fieldCode)+'</strong></td><td>'+esc(f.fieldName)+'</td>' +
|
||||
'<td>'+esc(f.fieldType)+'</td>' +
|
||||
'<td>'+(f.isAggregatable ? '✓' : '-')+'</td>' +
|
||||
'<td>'+esc(f.defaultAggregate||'-')+'</td>' +
|
||||
'<td>'+esc(f.groupName||'-')+'</td><td>'+f.sortOrder+'</td>' +
|
||||
'<td><span class="badge '+sc+'">'+(f.status||'ACTIVE')+'</span></td>' +
|
||||
'<td class="actions">' +
|
||||
'<button class="btn btn-sm btn-outline" onclick="openFieldModal('+f.id+')">编辑</button>' +
|
||||
'<button class="btn btn-sm btn-danger" onclick="deleteField('+f.id+')">删除</button></td></tr>';
|
||||
}
|
||||
h += '</tbody></table>';
|
||||
document.getElementById('field-list').innerHTML = h;
|
||||
} catch(e) { document.getElementById('field-loading').style.display = 'none'; toast(e.message, 'error'); }
|
||||
}
|
||||
|
||||
function fieldFormHTML(d) {
|
||||
d = d || {};
|
||||
return '<div class="form-row"><div class="form-group"><label>所属业务</label><input id="f-fbiz" value="'+esc(d.businessCode||v('field-biz-select')||'')+'" readonly style="background:#f5f5f5"></div>' +
|
||||
'<div class="form-group"><label>所属报表</label><input id="f-frep" value="'+esc(d.reportCode||v('field-report-select')||'')+'" readonly style="background:#f5f5f5"></div></div>' +
|
||||
'<div class="form-row-3"><div class="form-group"><label>字段编码 *</label><input id="f-fcode" value="'+esc(d.fieldCode||'')+'" placeholder="如 order_amount"></div>' +
|
||||
'<div class="form-group"><label>字段名称 *</label><input id="f-fname" value="'+esc(d.fieldName||'')+'"></div>' +
|
||||
'<div class="form-group"><label>字段类型 *</label><select id="f-ftype"><option value="STRING"'+(d.fieldType==='STRING'||!d.fieldType?' selected':'')+'>STRING</option><option value="INT"'+(d.fieldType==='INT'?' selected':'')+'>INT</option><option value="FLOAT"'+(d.fieldType==='FLOAT'?' selected':'')+'>FLOAT</option><option value="DATE"'+(d.fieldType==='DATE'?' selected':'')+'>DATE</option><option value="DATETIME"'+(d.fieldType==='DATETIME'?' selected':'')+'>DATETIME</option><option value="JSONB"'+(d.fieldType==='JSONB'?' selected':'')+'>JSONB</option></select></div></div>' +
|
||||
'<div class="form-row-3"><div class="form-group"><label>字段角色 *</label><select id="f-frole"><option value="DIMENSION"'+(d.fieldRole==='DIMENSION'||!d.fieldRole?' selected':'')+'>维度(DIMENSION)</option><option value="INDICATOR"'+(d.fieldRole==='INDICATOR'?' selected':'')+'>指标(INDICATOR)</option><option value="FILTER"'+(d.fieldRole==='FILTER'?' selected':'')+'>筛选(FILTER)</option><option value="FILTER_ONLY"'+(d.fieldRole==='FILTER_ONLY'?' selected':'')+'>仅筛选(FILTER_ONLY)</option></select></div>' +
|
||||
'<div class="form-group"><label>数据存储类型</label><select id="f-fdtype"><option value="STRING"'+(d.dataType==='STRING'||!d.dataType?' selected':'')+'>STRING</option><option value="INT"'+(d.dataType==='INT'?' selected':'')+'>INT</option><option value="FLOAT"'+(d.dataType==='FLOAT'?' selected':'')+'>FLOAT</option><option value="DATE"'+(d.dataType==='DATE'?' selected':'')+'>DATE</option><option value="DATETIME"'+(d.dataType==='DATETIME'?' selected':'')+'>DATETIME</option><option value="JSONB"'+(d.dataType==='JSONB'?' selected':'')+'>JSONB</option></select></div>' +
|
||||
'<div class="form-group"><label>默认聚合</label><select id="f-faggr"><option value=""'+(!d.defaultAggregate?' selected':'')+'>无</option><option value="SUM"'+(d.defaultAggregate==='SUM'?' selected':'')+'>SUM</option><option value="COUNT"'+(d.defaultAggregate==='COUNT'?' selected':'')+'>COUNT</option><option value="AVG"'+(d.defaultAggregate==='AVG'?' selected':'')+'>AVG</option><option value="MAX"'+(d.defaultAggregate==='MAX'?' selected':'')+'>MAX</option><option value="MIN"'+(d.defaultAggregate==='MIN'?' selected':'')+'>MIN</option></select></div></div>' +
|
||||
'<div class="form-row-3"><div class="form-group"><label>分组名称</label><input id="f-fgroup" value="'+esc(d.groupName||'')+'"></div>' +
|
||||
'<div class="form-group"><label>排序号</label><input id="f-fsort" type="number" value="'+(d.sortOrder||0)+'"></div>' +
|
||||
'<div class="form-group"><label>单位</label><input id="f-funit" value="'+esc(d.unit||'')+'"></div></div>' +
|
||||
'<div class="form-group"><label>表达式（衍生字段如 {refund}/{order}*100）</label><input id="f-fexpr" value="'+esc(d.expression||'')+'"></div>' +
|
||||
'<div class="form-row"><div class="form-group"><label>操作人</label><input id="f-fop" value="'+esc(d.creator||'admin')+'"></div>' +
|
||||
'<div class="form-group"><label>状态</label><select id="f-fs"><option value="ACTIVE"'+((d.status||'ACTIVE')==='ACTIVE'?' selected':'')+'>启用</option><option value="INACTIVE"'+(d.status==='INACTIVE'?' selected':'')+'>停用</option></select></div></div>' +
|
||||
'<div class="checkbox-group" style="margin-top:8px"><label><input type="checkbox" id="f-fagg" '+(d.isAggregatable===true?'checked':'')+'> 可聚合</label>' +
|
||||
'<label><input type="checkbox" id="f-ffilter" '+((d.isFilterable===undefined||d.isFilterable===true)?'checked':'')+'> 可筛选</label>' +
|
||||
'<label><input type="checkbox" id="f-fquery" '+((d.isQueryable===undefined||d.isQueryable===true)?'checked':'')+'> 可查询</label>' +
|
||||
'<label><input type="checkbox" id="f-fsortable" '+((d.isSortable===undefined||d.isSortable===true)?'checked':'')+'> 可排序</label></div>';
|
||||
}
|
||||
|
||||
async function openFieldModal(id) {
|
||||
const title = document.getElementById('modalTitle');
|
||||
const body = document.getElementById('modalBody');
|
||||
const footer = document.getElementById('modalFooter');
|
||||
if (id) {
|
||||
try {
|
||||
const res = await api('GET', '/report/field?id=' + id);
|
||||
const d = res && res.data !== undefined ? res.data : (res && res.Data !== undefined ? res.Data : res);
|
||||
console.log('[FieldEdit] id=' + id + ' raw:', res, 'data:', d);
|
||||
if (!d || !d.id) { toast('未获取到字段数据（ID='+id+'）', 'error'); return; }
|
||||
title.textContent = '编辑字段';
|
||||
body.innerHTML = fieldFormHTML(d);
|
||||
footer.innerHTML = '<button class="btn btn-outline" onclick="closeModal()">取消</button><button class="btn btn-primary" onclick="saveField('+id+')">保存</button>';
|
||||
} catch(e) { toast(e.message, 'error'); return; }
|
||||
} else {
|
||||
title.textContent = '新建字段';
|
||||
body.innerHTML = fieldFormHTML(null);
|
||||
footer.innerHTML = '<button class="btn btn-outline" onclick="closeModal()">取消</button><button class="btn btn-primary" onclick="saveField(0)">创建</button>';
|
||||
}
|
||||
document.getElementById('modalOverlay').classList.add('open');
|
||||
document.getElementById('modal').classList.add('open');
|
||||
}
|
||||
|
||||
function fieldVal(id) { const el = document.getElementById(id); return el ? (el.type==='checkbox'?el.checked:el.value) : ''; }
|
||||
|
||||
async function saveField(id) {
|
||||
const body = {
|
||||
businessCode: v('f-fbiz'), reportCode: v('f-frep'),
|
||||
fieldCode: v('f-fcode'), fieldName: v('f-fname'),
|
||||
fieldType: v('f-ftype'), dataType: v('f-fdtype'), fieldRole: v('f-frole'),
|
||||
defaultAggregate: v('f-faggr'), groupName: v('f-fgroup'),
|
||||
sortOrder: parseInt(v('f-fsort'))||0, unit: v('f-funit'),
|
||||
expression: v('f-fexpr'), operator: v('f-fop'), status: v('f-fs'),
|
||||
isAggregatable: fieldVal('f-fagg'),
|
||||
isFilterable: fieldVal('f-ffilter'),
|
||||
isQueryable: fieldVal('f-fquery'),
|
||||
isSortable: fieldVal('f-fsortable'),
|
||||
};
|
||||
if (id) body.id = id;
|
||||
try {
|
||||
await api('POST', '/report/field/save', body);
|
||||
toast(id ? '更新成功' : '创建成功', 'success');
|
||||
closeModal(); loadFieldList();
|
||||
} catch(e) { toast(e.message, 'error'); }
|
||||
}
|
||||
|
||||
async function deleteField(id) {
|
||||
if (!confirm('确定删除此字段？')) return;
|
||||
try { await api('DELETE', '/report/field?id='+id); toast('删除成功', 'success'); loadFieldList(); }
|
||||
catch(e) { toast(e.message, 'error'); }
|
||||
}
|
||||
|
||||
// ==================== 抽取配置 CRUD ====================
|
||||
async function loadExtractList() {
|
||||
const bc = v('extract-biz-select');
|
||||
const rc = v('extract-report-select');
|
||||
if (!bc || !rc) { document.getElementById('extract-list').innerHTML = '<div class="empty">请先选择业务和报表</div>'; return; }
|
||||
document.getElementById('extract-loading').style.display = 'block';
|
||||
try {
|
||||
const data = await api('GET', '/report/extractConfigs?businessCode=' + bc + '&reportCode=' + rc);
|
||||
document.getElementById('extract-loading').style.display = 'none';
|
||||
const list = data.list || [];
|
||||
if (list.length === 0) {
|
||||
document.getElementById('extract-list').innerHTML = '<div class="empty">暂无抽取配置</div>';
|
||||
return;
|
||||
}
|
||||
let h = '<table><thead><tr><th>ID</th><th>编码</th><th>名称</th><th>源表</th><th>目标表</th><th>抽取模式</th><th>抽取类型</th><th>启用</th><th>操作</th></tr></thead><tbody>';
|
||||
for (const ec of list) {
|
||||
h += '<tr><td>'+ec.id+'</td><td><strong>'+esc(ec.extractCode)+'</strong></td><td>'+esc(ec.extractName)+'</td>' +
|
||||
'<td style="font-family:monospace;font-size:11px">'+esc(ec.sourceTableName||'-')+'</td>' +
|
||||
'<td style="font-family:monospace;font-size:11px">'+esc(ec.targetTableName||'-')+'</td>' +
|
||||
'<td><span class="badge '+(ec.extractMode==='AGGREGATE'?'info':'active')+'">'+(ec.extractMode||'DIRECT')+'</span></td>' +
|
||||
'<td><span class="badge active">'+(ec.extractType||'INCREMENTAL')+'</span></td>' +
|
||||
'<td>'+(ec.isEnabled ? '✓' : '✗')+'</td>' +
|
||||
'<td class="actions">' +
|
||||
'<button class="btn btn-sm btn-outline" onclick="openExtractModal('+ec.id+')">编辑</button>' +
|
||||
'<button class="btn btn-sm btn-danger" onclick="deleteExtract('+ec.id+')">删除</button></td></tr>';
|
||||
}
|
||||
h += '</tbody></table>';
|
||||
document.getElementById('extract-list').innerHTML = h;
|
||||
} catch(e) { document.getElementById('extract-loading').style.display = 'none'; toast(e.message, 'error'); }
|
||||
}
|
||||
|
||||
function extractFormHTML(d) {
|
||||
d = d || {};
|
||||
return '<div class="form-row"><div class="form-group"><label>所属业务</label><input id="f-ebiz" value="'+esc(d.businessCode||v('extract-biz-select')||'')+'" readonly style="background:#f5f5f5"></div>' +
|
||||
'<div class="form-group"><label>所属报表</label><input id="f-erep" value="'+esc(d.reportCode||v('extract-report-select')||'')+'" readonly style="background:#f5f5f5"></div></div>' +
|
||||
'<div class="form-row"><div class="form-group"><label>抽取编码 *</label><input id="f-ecode" value="'+esc(d.extractCode||'')+'"></div>' +
|
||||
'<div class="form-group"><label>抽取名称 *</label><input id="f-ename" value="'+esc(d.extractName||'')+'"></div></div>' +
|
||||
'<div class="form-row"><div class="form-group"><label>源表名 *</label><input id="f-esrc" value="'+esc(d.sourceTableName||'')+'"></div>' +
|
||||
'<div class="form-group"><label>源表别名</label><input id="f-esrca" value="'+esc(d.sourceTableAlias||'')+'"></div></div>' +
|
||||
'<div class="form-row"><div class="form-group"><label>目标表名 *</label><input id="f-etgt" value="'+esc(d.targetTableName||'')+'"></div>' +
|
||||
'<div class="form-group"><label>增量依据字段</label><input id="f-ekey" value="'+esc(d.extractKeyField||'')+'"></div></div>' +
|
||||
'<div class="form-row-3"><div class="form-group"><label>抽取模式</label><select id="f-emode"><option value="DIRECT"'+(d.extractMode==='DIRECT'||!d.extractMode?' selected':'')+'>DIRECT</option><option value="AGGREGATE"'+(d.extractMode==='AGGREGATE'?' selected':'')+'>AGGREGATE</option></select></div>' +
|
||||
'<div class="form-group"><label>抽取类型</label><select id="f-etype"><option value="INCREMENTAL"'+(d.extractType==='INCREMENTAL'||!d.extractType?' selected':'')+'>增量(INCREMENTAL)</option><option value="FULL"'+(d.extractType==='FULL'?' selected':'')+'>全量(FULL)</option></select></div>' +
|
||||
'<div class="form-group"><label>批大小</label><input id="f-ebatch" type="number" value="'+(d.batchSize||1000)+'"></div></div>' +
|
||||
'<div class="form-group"><label>GROUP BY 字段（逗号分隔）</label><input id="f-egroup" value="'+esc((d.groupByFields||[]).join(','))+'"></div>' +
|
||||
'<div class="form-group"><label>过滤表达式</label><input id="f-efilter" value="'+esc(d.filterExpression||'')+'" placeholder="如 o.status != \'CANCELLED\'"></div>' +
|
||||
'<div class="form-group"><label>操作人</label><input id="f-eop" value="'+esc(d.creator||'admin')+'"></div>' +
|
||||
'<div class="form-row"><div class="form-group"><label>状态</label><select id="f-es"><option value="ACTIVE"'+((d.status||'ACTIVE')==='ACTIVE'?' selected':'')+'>启用</option><option value="INACTIVE"'+(d.status==='INACTIVE'?' selected':'')+'>停用</option></select></div>' +
|
||||
'<div class="form-group"><label><input type="checkbox" id="f-eenabled" '+((d.isEnabled===undefined||d.isEnabled===true)?'checked':'')+'> 启用抽取</label></div></div>';
|
||||
}
|
||||
|
||||
async function openExtractModal(id) {
|
||||
const title = document.getElementById('modalTitle');
|
||||
const body = document.getElementById('modalBody');
|
||||
const footer = document.getElementById('modalFooter');
|
||||
if (id) {
|
||||
try {
|
||||
const res = await api('GET', '/report/extractConfig?id=' + id);
|
||||
const d = res && res.data !== undefined ? res.data : (res && res.Data !== undefined ? res.Data : res);
|
||||
console.log('[ExtractEdit] id=' + id + ' raw:', res, 'data:', d);
|
||||
if (!d || !d.id) { toast('未获取到抽取配置数据（ID='+id+'）', 'error'); return; }
|
||||
title.textContent = '编辑抽取配置';
|
||||
body.innerHTML = extractFormHTML(d);
|
||||
footer.innerHTML = '<button class="btn btn-outline" onclick="closeModal()">取消</button><button class="btn btn-primary" onclick="saveExtract('+id+')">保存</button>';
|
||||
} catch(e) { toast(e.message, 'error'); return; }
|
||||
} else {
|
||||
title.textContent = '新建抽取配置';
|
||||
body.innerHTML = extractFormHTML(null);
|
||||
footer.innerHTML = '<button class="btn btn-outline" onclick="closeModal()">取消</button><button class="btn btn-primary" onclick="saveExtract(0)">创建</button>';
|
||||
}
|
||||
document.getElementById('modalOverlay').classList.add('open');
|
||||
document.getElementById('modal').classList.add('open');
|
||||
}
|
||||
|
||||
async function saveExtract(id) {
|
||||
const body = {
|
||||
businessCode: v('f-ebiz'), reportCode: v('f-erep'),
|
||||
extractCode: v('f-ecode'), extractName: v('f-ename'),
|
||||
sourceTableName: v('f-esrc'), sourceTableAlias: v('f-esrca'),
|
||||
targetTableName: v('f-etgt'), extractKeyField: v('f-ekey'),
|
||||
extractMode: v('f-emode'), extractType: v('f-etype'),
|
||||
batchSize: parseInt(v('f-ebatch'))||1000,
|
||||
groupByFields: v('f-egroup').split(',').map(s => s.trim()).filter(Boolean),
|
||||
filterExpression: v('f-efilter'), operator: v('f-eop'),
|
||||
status: v('f-es'), isEnabled: document.getElementById('f-eenabled').checked,
|
||||
};
|
||||
if (id) body.id = id;
|
||||
try {
|
||||
await api('POST', '/report/extractConfig/save', body);
|
||||
toast(id ? '更新成功' : '创建成功', 'success');
|
||||
closeModal(); loadExtractList();
|
||||
} catch(e) { toast(e.message, 'error'); }
|
||||
}
|
||||
|
||||
async function deleteExtract(id) {
|
||||
if (!confirm('确定删除此抽取配置？')) return;
|
||||
try { await api('DELETE', '/report/extractConfig?id='+id); toast('删除成功', 'success'); loadExtractList(); }
|
||||
catch(e) { toast(e.message, 'error'); }
|
||||
}
|
||||
|
||||
async function execExtract() {
|
||||
const bc = v('extract-biz-select');
|
||||
const rc = v('extract-report-select');
|
||||
if (!bc || !rc) { toast('请先选择业务和报表', 'error'); return; }
|
||||
const date = prompt('请输入统计日期 (yyyy-MM-dd)：', new Date().toISOString().slice(0,10));
|
||||
if (!date) return;
|
||||
try {
|
||||
const res = await api('POST', '/report/extract', { businessCode: bc, reportCode: rc, statDate: date, executor: 'admin' });
|
||||
toast('抽取完成！成功: ' + res.successCount + ' 失败: ' + res.failCount + ' 耗时: ' + res.execTimeMs + 'ms', res.success ? 'success' : 'error');
|
||||
} catch(e) { toast(e.message, 'error'); }
|
||||
}
|
||||
|
||||
// ==================== 数据查询 ====================
|
||||
let queryCtx = { dimensions: [], indicators: [], filters: [], availableFields: null };
|
||||
|
||||
async function onQueryBizChange() {
|
||||
const bc = v('query-biz-select');
|
||||
const rs = document.getElementById('query-report-select');
|
||||
rs.innerHTML = '<option value="">-- 选择报表 --</option>';
|
||||
queryCtx = { dimensions: [], indicators: [], filters: [], availableFields: null };
|
||||
if (!bc) return;
|
||||
try {
|
||||
const data = await api('GET', '/report/reports?businessCode=' + bc);
|
||||
rs.innerHTML = '<option value="">-- 选择报表 --</option>' + (data.list||[]).map(r => '<option value="'+r.reportCode+'">'+esc(r.reportName)+' ('+r.reportCode+')</option>').join('');
|
||||
} catch(e) { toast(e.message, 'error'); }
|
||||
}
|
||||
|
||||
async function loadQueryFields() {
|
||||
const bc = v('query-biz-select');
|
||||
const rc = v('query-report-select');
|
||||
if (!bc || !rc) {
|
||||
document.getElementById('query-content').style.display = 'none';
|
||||
document.getElementById('query-empty').style.display = 'block';
|
||||
return;
|
||||
}
|
||||
try {
|
||||
const data = await api('GET', '/report/fields?businessCode=' + bc + '&reportCode=' + rc);
|
||||
queryCtx.availableFields = {
|
||||
dimensions: data.dimensions || [],
|
||||
indicators: data.indicators || [],
|
||||
filters: data.filters || []
|
||||
};
|
||||
queryCtx.dimensions = [];
|
||||
queryCtx.indicators = [];
|
||||
queryCtx.filters = [];
|
||||
renderQueryUI();
|
||||
document.getElementById('query-content').style.display = 'block';
|
||||
document.getElementById('query-empty').style.display = 'none';
|
||||
document.getElementById('query-result').style.display = 'none';
|
||||
} catch(e) { toast(e.message, 'error'); }
|
||||
}
|
||||
|
||||
function renderQueryUI() {
|
||||
const f = queryCtx.availableFields;
|
||||
if (!f) return;
|
||||
|
||||
// 维度
|
||||
let dimHtml = '';
|
||||
for (const d of f.dimensions) {
|
||||
const sel = queryCtx.dimensions.includes(d.fieldCode);
|
||||
dimHtml += '<span class="chip'+(sel?' selected':'')+'" onclick="toggleDim(\''+jsAttr(d.fieldCode)+'\')">'+esc(d.fieldName||d.fieldCode)+'</span>';
|
||||
}
|
||||
document.getElementById('dimension-chips').innerHTML = dimHtml;
|
||||
|
||||
// 指标
|
||||
let indHtml = '';
|
||||
for (const d of f.indicators) {
|
||||
const sel = queryCtx.indicators.find(i => i.fieldCode === d.fieldCode);
|
||||
indHtml += '<span class="chip'+(sel?' selected':'')+'" onclick="toggleInd(\''+jsAttr(d.fieldCode)+'\', \''+jsAttr(d.defaultAggregate||'SUM')+'\', \''+jsAttr(d.fieldName||d.fieldCode)+'\')">'+esc(d.fieldName||d.fieldCode)+(d.unit?'('+d.unit+')':'')+'</span>';
|
||||
}
|
||||
document.getElementById('indicator-area').innerHTML = indHtml;
|
||||
renderSelectedIndicators();
|
||||
|
||||
// 筛选
|
||||
let filHtml = '';
|
||||
for (const d of [...f.filters, ...f.indicators, ...f.dimensions]) {
|
||||
filHtml += '<span class="chip" onclick="addFilter(\''+jsAttr(d.fieldCode)+'\', \''+jsAttr(d.fieldName||d.fieldCode)+'\')">'+esc(d.fieldName||d.fieldCode)+'</span>';
|
||||
}
|
||||
document.getElementById('filter-area').innerHTML = filHtml;
|
||||
|
||||
// 默认时间范围
|
||||
if (!document.getElementById('query-start-date').value) {
|
||||
const today = new Date().toISOString().slice(0,10);
|
||||
const weekAgo = new Date(Date.now() - 7*86400000).toISOString().slice(0,10);
|
||||
document.getElementById('query-start-date').value = weekAgo;
|
||||
document.getElementById('query-end-date').value = today;
|
||||
}
|
||||
}
|
||||
|
||||
function toggleDim(code) {
|
||||
const idx = queryCtx.dimensions.indexOf(code);
|
||||
if (idx >= 0) queryCtx.dimensions.splice(idx, 1);
|
||||
else queryCtx.dimensions.push(code);
|
||||
renderQueryUI();
|
||||
}
|
||||
|
||||
function toggleInd(code, defAggr, name) {
|
||||
const idx = queryCtx.indicators.findIndex(i => i.fieldCode === code);
|
||||
if (idx >= 0) { queryCtx.indicators.splice(idx, 1); renderSelectedIndicators(); renderQueryUI(); return; }
|
||||
const aggr = prompt('选择聚合方式 (SUM/COUNT/AVG/MAX/MIN)：', defAggr||'SUM');
|
||||
if (!aggr) return;
|
||||
queryCtx.indicators.push({ fieldCode: code, aggregate: aggr.toUpperCase(), alias: code + '_' + aggr.toLowerCase() });
|
||||
renderSelectedIndicators();
|
||||
renderQueryUI();
|
||||
}
|
||||
|
||||
function renderSelectedIndicators() {
|
||||
if (queryCtx.indicators.length === 0) {
|
||||
document.getElementById('selected-indicators').innerHTML = '';
|
||||
return;
|
||||
}
|
||||
let h = '<div style="font-size:12px;color:#666;margin-bottom:4px">已选指标：</div>';
|
||||
for (const ind of queryCtx.indicators) {
|
||||
h += '<span class="field-tag">'+esc(ind.fieldCode)+' ('+ind.aggregate+') <span class="remove" onclick="removeInd(\''+jsAttr(ind.fieldCode)+'\')">×</span></span>';
|
||||
}
|
||||
document.getElementById('selected-indicators').innerHTML = h;
|
||||
}
|
||||
|
||||
function removeInd(code) {
|
||||
queryCtx.indicators = queryCtx.indicators.filter(i => i.fieldCode !== code);
|
||||
renderSelectedIndicators();
|
||||
renderQueryUI();
|
||||
}
|
||||
|
||||
let filterCounter = 0;
|
||||
|
||||
function addFilter(code, name) {
|
||||
filterCounter++;
|
||||
const area = document.getElementById('filter-area');
|
||||
const ops = ['=','!=','>','<','>=','<=','IN','LIKE','BETWEEN'];
|
||||
const isBetween = ops[ops.length-1];
|
||||
const row = document.createElement('div');
|
||||
row.className = 'add-filter-row';
|
||||
row.id = 'filter-row-'+filterCounter;
|
||||
row.innerHTML = '<select id="fcol-'+filterCounter+'" style="padding:6px 8px;border:1px solid #d9d9d9;border-radius:6px;font-size:11px"><option value="'+esc(code)+'">'+esc(name)+' ('+esc(code)+')</option></select>' +
|
||||
'<select id="fop-'+filterCounter+'" style="padding:6px 8px;border:1px solid #d9d9d9;border-radius:6px;font-size:11px" onchange="onFilterOpChange('+filterCounter+')">'+ops.map(o=>'<option value="'+o+'">'+o+'</option>').join('')+'</select>' +
|
||||
'<input id="fval-'+filterCounter+'" style="padding:6px 8px;border:1px solid #d9d9d9;border-radius:6px;font-size:11px" placeholder="值">' +
|
||||
'<button class="btn btn-sm btn-danger" style="padding:4px 8px" onclick="removeFilter('+filterCounter+')">×</button>';
|
||||
area.appendChild(row);
|
||||
}
|
||||
|
||||
function onFilterOpChange(idx) {
|
||||
const op = document.getElementById('fop-'+idx).value;
|
||||
const valEl = document.getElementById('fval-'+idx);
|
||||
if (op === 'BETWEEN') valEl.placeholder = '值1,值2';
|
||||
else if (op === 'IN') valEl.placeholder = '逗号分隔';
|
||||
else valEl.placeholder = '值';
|
||||
}
|
||||
|
||||
function removeFilter(idx) {
|
||||
const row = document.getElementById('filter-row-'+idx);
|
||||
if (row) row.remove();
|
||||
}
|
||||
|
||||
async function executeQuery() {
|
||||
const bc = v('query-biz-select');
|
||||
const rc = v('query-report-select');
|
||||
if (!bc || !rc) { toast('请先选择业务和报表', 'error'); return; }
|
||||
|
||||
const filters = [];
|
||||
document.querySelectorAll('[id^="filter-row-"]').forEach(row => {
|
||||
const m = row.id.match(/filter-row-(\d+)/);
|
||||
if (!m) return;
|
||||
const cid = m[1];
|
||||
const colEl = document.getElementById('fcol-'+cid);
|
||||
const opEl = document.getElementById('fop-'+cid);
|
||||
const valEl = document.getElementById('fval-'+cid);
|
||||
if (!colEl || !opEl || !valEl || !valEl.value.trim()) return;
|
||||
const op = opEl.value;
|
||||
let val = valEl.value.trim();
|
||||
let val2 = null;
|
||||
if (op === 'BETWEEN') {
|
||||
const parts = val.split(',').map(s => s.trim());
|
||||
if (parts.length >= 2) { val = parts[0]; val2 = parts[1]; }
|
||||
else return;
|
||||
} else if (op === 'IN') {
|
||||
val = val.split(',').map(s => s.trim());
|
||||
}
|
||||
const f = { fieldCode: colEl.value, operator: op, value: val };
|
||||
if (val2 !== null) f.value2 = val2;
|
||||
filters.push(f);
|
||||
});
|
||||
|
||||
const body = {
|
||||
businessCode: bc, reportCode: rc,
|
||||
dimensions: queryCtx.dimensions,
|
||||
indicators: queryCtx.indicators,
|
||||
filters: filters,
|
||||
timeRange: { startDate: v('query-start-date'), endDate: v('query-end-date') },
|
||||
orderBy: v('query-order-field') ? [{ fieldCode: v('query-order-field'), direction: v('query-order-dir') }] : [],
|
||||
page: 1, pageSize: parseInt(v('query-page-size'))||20,
|
||||
};
|
||||
|
||||
const resultBox = document.getElementById('query-result');
|
||||
resultBox.style.display = 'block';
|
||||
resultBox.textContent = '查询中...';
|
||||
try {
|
||||
const res = await api('POST', '/report/query', body);
|
||||
let output = '总记录数: ' + res.total + ' | 执行耗时: ' + res.execTimeMs + 'ms\n';
|
||||
if (res.sql) output += '生成SQL: ' + res.sql + '\n';
|
||||
output += '\n';
|
||||
if (res.list && res.list.length > 0) {
|
||||
// 表头
|
||||
const keys = Object.keys(res.list[0]);
|
||||
output += keys.join('\t') + '\n';
|
||||
output += keys.map(() => '---').join('\t') + '\n';
|
||||
for (const row of res.list) {
|
||||
output += keys.map(k => row[k] != null ? String(row[k]) : '').join('\t') + '\n';
|
||||
}
|
||||
} else {
|
||||
output += '（无数据）';
|
||||
}
|
||||
resultBox.textContent = output;
|
||||
} catch(e) { resultBox.textContent = '查询失败: ' + e.message; }
|
||||
}
|
||||
|
||||
// ==================== 通用 ====================
|
||||
function closeModal() {
|
||||
document.getElementById('modalOverlay').classList.remove('open');
|
||||
document.getElementById('modal').classList.remove('open');
|
||||
}
|
||||
|
||||
// 初始化
|
||||
loadBizList();
|
||||
loadBizSelects();
|
||||
</script>
|
||||
</body>
|
||||
</html>`
|
||||
@@ -0,0 +1,371 @@
|
||||
package report
|
||||
|
||||
import (
|
||||
"context"
|
||||
|
||||
reportSvc "dataengine/common/report"
|
||||
"dataengine/common/report/model"
|
||||
|
||||
"gitea.redpowerfuture.com/red-future/common/beans"
|
||||
"github.com/gogf/gf/v2/frame/g"
|
||||
)
|
||||
|
||||
type report struct{}
|
||||
|
||||
var ReportController = new(report)
|
||||
|
||||
func svc() *reportSvc.ReportService {
|
||||
return reportSvc.GetService()
|
||||
}
|
||||
|
||||
func ctxWithUser(ctx context.Context) context.Context {
|
||||
return context.WithValue(ctx, "user", &beans.User{UserName: "admin", TenantId: 1})
|
||||
}
|
||||
|
||||
// ============================================================
|
||||
// 业务 CRUD
|
||||
// ============================================================
|
||||
|
||||
type listBusinessesReq struct {
|
||||
g.Meta `path:"/businesses" method:"get" tags:"报表引擎" summary:"业务列表"`
|
||||
}
|
||||
|
||||
type listBusinessesRes struct {
|
||||
List []model.BusinessConfig `json:"list"`
|
||||
}
|
||||
|
||||
func (c *report) ListBusinesses(ctx context.Context, req *listBusinessesReq) (*listBusinessesRes, error) {
|
||||
ctx = ctxWithUser(ctx)
|
||||
list, err := svc().GetAllBusinesses(ctx)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return &listBusinessesRes{List: list}, nil
|
||||
}
|
||||
|
||||
type getBusinessReq struct {
|
||||
g.Meta `path:"/business" method:"get" tags:"报表引擎" summary:"获取业务"`
|
||||
ID int64 `json:"id" v:"required"`
|
||||
}
|
||||
|
||||
type getBusinessRes struct {
|
||||
Data *model.BusinessConfig `json:"data"`
|
||||
}
|
||||
|
||||
func (c *report) GetBusiness(ctx context.Context, req *getBusinessReq) (*getBusinessRes, error) {
|
||||
ctx = ctxWithUser(ctx)
|
||||
data, err := svc().GetBusiness(ctx, req.ID)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return &getBusinessRes{Data: data}, nil
|
||||
}
|
||||
|
||||
type saveBusinessReq struct {
|
||||
g.Meta `path:"/business/save" method:"post" tags:"报表引擎" summary:"保存业务"`
|
||||
model.SaveBusinessReq
|
||||
}
|
||||
|
||||
type saveBusinessRes struct {
|
||||
*model.SaveResult
|
||||
}
|
||||
|
||||
func (c *report) SaveBusiness(ctx context.Context, req *saveBusinessReq) (*saveBusinessRes, error) {
|
||||
ctx = ctxWithUser(ctx)
|
||||
result, err := svc().SaveBusiness(ctx, &req.SaveBusinessReq)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return &saveBusinessRes{SaveResult: result}, nil
|
||||
}
|
||||
|
||||
type deleteBusinessReq struct {
|
||||
g.Meta `path:"/business" method:"delete" tags:"报表引擎" summary:"删除业务"`
|
||||
ID int64 `json:"id" v:"required"`
|
||||
}
|
||||
|
||||
type deleteBusinessRes struct {
|
||||
*model.DeleteResult
|
||||
}
|
||||
|
||||
func (c *report) DeleteBusiness(ctx context.Context, req *deleteBusinessReq) (*deleteBusinessRes, error) {
|
||||
ctx = ctxWithUser(ctx)
|
||||
result, err := svc().DeleteBusiness(ctx, req.ID)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return &deleteBusinessRes{DeleteResult: result}, nil
|
||||
}
|
||||
|
||||
// ============================================================
|
||||
// 报表 CRUD
|
||||
// ============================================================
|
||||
|
||||
type listReportsReq struct {
|
||||
g.Meta `path:"/reports" method:"get" tags:"报表引擎" summary:"报表列表"`
|
||||
BusinessCode string `json:"businessCode" v:"required"`
|
||||
}
|
||||
|
||||
type listReportsRes struct {
|
||||
List []model.ReportConfig `json:"list"`
|
||||
}
|
||||
|
||||
func (c *report) ListReports(ctx context.Context, req *listReportsReq) (*listReportsRes, error) {
|
||||
ctx = ctxWithUser(ctx)
|
||||
list, err := svc().GetAllReports(ctx, req.BusinessCode)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return &listReportsRes{List: list}, nil
|
||||
}
|
||||
|
||||
type getReportReq struct {
|
||||
g.Meta `path:"/report" method:"get" tags:"报表引擎" summary:"获取报表"`
|
||||
ID int64 `json:"id" v:"required"`
|
||||
}
|
||||
|
||||
type getReportRes struct {
|
||||
Data *model.ReportConfig `json:"data"`
|
||||
}
|
||||
|
||||
func (c *report) GetReport(ctx context.Context, req *getReportReq) (*getReportRes, error) {
|
||||
ctx = ctxWithUser(ctx)
|
||||
data, err := svc().GetReport(ctx, req.ID)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return &getReportRes{Data: data}, nil
|
||||
}
|
||||
|
||||
type saveReportReq struct {
|
||||
g.Meta `path:"/report/save" method:"post" tags:"报表引擎" summary:"保存报表"`
|
||||
model.SaveReportReq
|
||||
}
|
||||
|
||||
type saveReportRes struct {
|
||||
*model.SaveResult
|
||||
}
|
||||
|
||||
func (c *report) SaveReport(ctx context.Context, req *saveReportReq) (*saveReportRes, error) {
|
||||
ctx = ctxWithUser(ctx)
|
||||
result, err := svc().SaveReport(ctx, &req.SaveReportReq)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return &saveReportRes{SaveResult: result}, nil
|
||||
}
|
||||
|
||||
type deleteReportReq struct {
|
||||
g.Meta `path:"/report" method:"delete" tags:"报表引擎" summary:"删除报表"`
|
||||
ID int64 `json:"id" v:"required"`
|
||||
}
|
||||
|
||||
type deleteReportRes struct {
|
||||
*model.DeleteResult
|
||||
}
|
||||
|
||||
func (c *report) DeleteReport(ctx context.Context, req *deleteReportReq) (*deleteReportRes, error) {
|
||||
ctx = ctxWithUser(ctx)
|
||||
result, err := svc().DeleteReport(ctx, req.ID)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return &deleteReportRes{DeleteResult: result}, nil
|
||||
}
|
||||
|
||||
// ============================================================
|
||||
// 字段 CRUD
|
||||
// ============================================================
|
||||
|
||||
type getReportFieldsReq struct {
|
||||
g.Meta `path:"/fields" method:"get" tags:"报表引擎" summary:"报表字段列表（按角色分组）"`
|
||||
BusinessCode string `json:"businessCode" v:"required"`
|
||||
ReportCode string `json:"reportCode" v:"required"`
|
||||
}
|
||||
|
||||
func (c *report) GetReportFields(ctx context.Context, req *getReportFieldsReq) (*model.GetReportFieldsResp, error) {
|
||||
ctx = ctxWithUser(ctx)
|
||||
return svc().GetReportFields(ctx, req.BusinessCode, req.ReportCode)
|
||||
}
|
||||
|
||||
type getFieldReq struct {
|
||||
g.Meta `path:"/field" method:"get" tags:"报表引擎" summary:"获取字段"`
|
||||
ID int64 `json:"id" v:"required"`
|
||||
}
|
||||
|
||||
type getFieldRes struct {
|
||||
Data *model.FieldConfig `json:"data"`
|
||||
}
|
||||
|
||||
func (c *report) GetField(ctx context.Context, req *getFieldReq) (*getFieldRes, error) {
|
||||
ctx = ctxWithUser(ctx)
|
||||
data, err := svc().GetField(ctx, req.ID)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return &getFieldRes{Data: data}, nil
|
||||
}
|
||||
|
||||
type saveFieldReq struct {
|
||||
g.Meta `path:"/field/save" method:"post" tags:"报表引擎" summary:"保存字段"`
|
||||
model.SaveFieldReq
|
||||
}
|
||||
|
||||
type saveFieldRes struct {
|
||||
*model.SaveResult
|
||||
}
|
||||
|
||||
func (c *report) SaveField(ctx context.Context, req *saveFieldReq) (*saveFieldRes, error) {
|
||||
ctx = ctxWithUser(ctx)
|
||||
result, err := svc().SaveField(ctx, &req.SaveFieldReq)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return &saveFieldRes{SaveResult: result}, nil
|
||||
}
|
||||
|
||||
type deleteFieldReq struct {
|
||||
g.Meta `path:"/field" method:"delete" tags:"报表引擎" summary:"删除字段"`
|
||||
ID int64 `json:"id" v:"required"`
|
||||
}
|
||||
|
||||
type deleteFieldRes struct {
|
||||
*model.DeleteResult
|
||||
}
|
||||
|
||||
func (c *report) DeleteField(ctx context.Context, req *deleteFieldReq) (*deleteFieldRes, error) {
|
||||
ctx = ctxWithUser(ctx)
|
||||
result, err := svc().DeleteField(ctx, req.ID)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return &deleteFieldRes{DeleteResult: result}, nil
|
||||
}
|
||||
|
||||
// ============================================================
|
||||
// 抽取配置 CRUD
|
||||
// ============================================================
|
||||
|
||||
type getExtractConfigsReq struct {
|
||||
g.Meta `path:"/extractConfigs" method:"get" tags:"报表引擎" summary:"抽取配置列表"`
|
||||
BusinessCode string `json:"businessCode" v:"required"`
|
||||
ReportCode string `json:"reportCode" v:"required"`
|
||||
}
|
||||
|
||||
type getExtractConfigsRes struct {
|
||||
List []model.ExtractConfig `json:"list"`
|
||||
}
|
||||
|
||||
func (c *report) GetExtractConfigs(ctx context.Context, req *getExtractConfigsReq) (*getExtractConfigsRes, error) {
|
||||
ctx = ctxWithUser(ctx)
|
||||
list, err := svc().GetExtractConfigs(ctx, req.BusinessCode, req.ReportCode)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return &getExtractConfigsRes{List: list}, nil
|
||||
}
|
||||
|
||||
type getExtractConfigReq struct {
|
||||
g.Meta `path:"/extractConfig" method:"get" tags:"报表引擎" summary:"获取抽取配置"`
|
||||
ID int64 `json:"id" v:"required"`
|
||||
}
|
||||
|
||||
type getExtractConfigRes struct {
|
||||
Data *model.ExtractConfig `json:"data"`
|
||||
}
|
||||
|
||||
func (c *report) GetExtractConfig(ctx context.Context, req *getExtractConfigReq) (*getExtractConfigRes, error) {
|
||||
ctx = ctxWithUser(ctx)
|
||||
data, err := svc().GetExtractConfig(ctx, req.ID)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return &getExtractConfigRes{Data: data}, nil
|
||||
}
|
||||
|
||||
type saveExtractConfigReq struct {
|
||||
g.Meta `path:"/extractConfig/save" method:"post" tags:"报表引擎" summary:"保存抽取配置"`
|
||||
model.SaveExtractConfigReq
|
||||
}
|
||||
|
||||
type saveExtractConfigRes struct {
|
||||
*model.SaveResult
|
||||
}
|
||||
|
||||
func (c *report) SaveExtractConfig(ctx context.Context, req *saveExtractConfigReq) (*saveExtractConfigRes, error) {
|
||||
ctx = ctxWithUser(ctx)
|
||||
result, err := svc().SaveExtractConfig(ctx, &req.SaveExtractConfigReq)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return &saveExtractConfigRes{SaveResult: result}, nil
|
||||
}
|
||||
|
||||
type deleteExtractConfigReq struct {
|
||||
g.Meta `path:"/extractConfig" method:"delete" tags:"报表引擎" summary:"删除抽取配置"`
|
||||
ID int64 `json:"id" v:"required"`
|
||||
}
|
||||
|
||||
type deleteExtractConfigRes struct {
|
||||
*model.DeleteResult
|
||||
}
|
||||
|
||||
func (c *report) DeleteExtractConfig(ctx context.Context, req *deleteExtractConfigReq) (*deleteExtractConfigRes, error) {
|
||||
ctx = ctxWithUser(ctx)
|
||||
result, err := svc().DeleteExtractConfig(ctx, req.ID)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return &deleteExtractConfigRes{DeleteResult: result}, nil
|
||||
}
|
||||
|
||||
// ============================================================
|
||||
// 数据操作
|
||||
// ============================================================
|
||||
|
||||
type extractDataReq struct {
|
||||
g.Meta `path:"/extract" method:"post" tags:"报表引擎" summary:"执行按天数据抽取"`
|
||||
model.ExtractDailyDataReq
|
||||
}
|
||||
|
||||
func (c *report) ExtractData(ctx context.Context, req *extractDataReq) (*model.ExtractDailyDataResp, error) {
|
||||
ctx = ctxWithUser(ctx)
|
||||
return svc().ExtractDailyData(ctx, req.BusinessCode, req.ReportCode, req.StatDate, req.Executor)
|
||||
}
|
||||
|
||||
type autoCreateTableReq struct {
|
||||
g.Meta `path:"/autoCreateTable" method:"post" tags:"报表引擎" summary:"自动创建统计宽表"`
|
||||
model.AutoCreateStatTableReq
|
||||
}
|
||||
|
||||
func (c *report) AutoCreateTable(ctx context.Context, req *autoCreateTableReq) (*model.AutoCreateStatTableResp, error) {
|
||||
ctx = ctxWithUser(ctx)
|
||||
return svc().AutoCreateStatTable(ctx, req.BusinessCode, req.ReportCode)
|
||||
}
|
||||
|
||||
type queryReportReq struct {
|
||||
g.Meta `path:"/query" method:"post" tags:"报表引擎" summary:"用户选择查询"`
|
||||
model.UserSelectQueryReq
|
||||
}
|
||||
|
||||
func (c *report) QueryReport(ctx context.Context, req *queryReportReq) (*model.UserSelectQueryResp, error) {
|
||||
ctx = ctxWithUser(ctx)
|
||||
return svc().QueryReportByUserSelect(ctx, &req.UserSelectQueryReq)
|
||||
}
|
||||
|
||||
type initTablesReq struct {
|
||||
g.Meta `path:"/initTables" method:"post" tags:"报表引擎" summary:"初始化系统表"`
|
||||
}
|
||||
|
||||
type initTablesRes struct {
|
||||
Success bool `json:"success"`
|
||||
Message string `json:"message"`
|
||||
}
|
||||
|
||||
func (c *report) InitTables(ctx context.Context, req *initTablesReq) (*initTablesRes, error) {
|
||||
ctx = ctxWithUser(ctx)
|
||||
if err := svc().InitSystemTables(ctx); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return &initTablesRes{Success: true, Message: "系统表初始化完成"}, nil
|
||||
}
|
||||
@@ -2,11 +2,9 @@ package dict
|
||||
|
||||
import (
|
||||
"context"
|
||||
consts1 "dataengine/consts/api-feature"
|
||||
consts "dataengine/consts/public"
|
||||
dto "dataengine/model/dto/dict"
|
||||
entity "dataengine/model/entity/dict"
|
||||
"strconv"
|
||||
"time"
|
||||
|
||||
"gitea.redpowerfuture.com/red-future/common/db/gfdb"
|
||||
@@ -174,7 +172,7 @@ func (d *datasourcePlatformDao) ExistsByPlatformCode(ctx context.Context, platfo
|
||||
// ListActivePlatforms 获取所有启用的平台
|
||||
func (d *datasourcePlatformDao) ListActivePlatforms(ctx context.Context) (res []entity.DatasourcePlatform, err error) {
|
||||
r, err := gfdb.DB(ctx).Model(ctx, consts.DatasourcePlatformTable).
|
||||
Where(entity.DatasourcePlatformCols.Status, consts1.PlatformStatusActive).
|
||||
Where(entity.DatasourcePlatformCols.Status, "ACTIVE").
|
||||
OrderAsc(entity.DatasourcePlatformCols.PlatformName).
|
||||
All()
|
||||
if err != nil {
|
||||
@@ -197,7 +195,7 @@ func (d *datasourcePlatformDao) GetPlatformStatistics(ctx context.Context) (stat
|
||||
|
||||
// 启用平台数
|
||||
active, err := gfdb.DB(ctx).Model(ctx, consts.DatasourcePlatformTable).
|
||||
Where(entity.DatasourcePlatformCols.Status, consts1.MappingStatusActive).
|
||||
Where(entity.DatasourcePlatformCols.Status, "ACTIVE").
|
||||
Count()
|
||||
if err != nil {
|
||||
return nil, err
|
||||
@@ -228,7 +226,7 @@ func (d *datasourcePlatformDao) BatchUpdateStatus(ctx context.Context, ids []int
|
||||
Data(map[string]interface{}{
|
||||
entity.DatasourcePlatformCols.Status: status,
|
||||
entity.DatasourcePlatformCols.UpdatedBy: updatedBy,
|
||||
entity.DatasourcePlatformCols.UpdatedAt: strconv.FormatInt(time.Now().Unix(), 10),
|
||||
entity.DatasourcePlatformCols.UpdatedAt: time.Now(),
|
||||
}).
|
||||
WhereIn(entity.DatasourcePlatformCols.ID, ids).
|
||||
Update()
|
||||
|
||||
@@ -84,7 +84,9 @@ func (d *apiInterfaceDao) buildListFilter(ctx context.Context, req *dto.ListApiI
|
||||
model.WhereLike(entity.ApiInterfaceCols.Name, "%"+req.Keyword+"%")
|
||||
model.WhereOrLike(entity.ApiInterfaceCols.Code, "%"+req.Keyword+"%")
|
||||
}
|
||||
model.Where(entity.ApiInterfaceCols.PlatformId, req.PlatformId)
|
||||
if req.PlatformId > 0 {
|
||||
model.Where(entity.ApiInterfaceCols.PlatformId, req.PlatformId)
|
||||
}
|
||||
model.Where(entity.ApiInterfaceCols.Name, req.Name)
|
||||
model.Where(entity.ApiInterfaceCols.Code, req.Code)
|
||||
model.Where(entity.ApiInterfaceCols.Method, req.Method)
|
||||
|
||||
Executable
BIN
Binary file not shown.
@@ -3,8 +3,13 @@ package main
|
||||
import (
|
||||
"dataengine/controller/debug"
|
||||
"dataengine/controller/dict"
|
||||
"dataengine/controller/public"
|
||||
reportCtrl "dataengine/controller/report"
|
||||
syncCtrl "dataengine/controller/sync"
|
||||
syncSvc "dataengine/service/sync"
|
||||
"os"
|
||||
"os/signal"
|
||||
"syscall"
|
||||
|
||||
"gitea.redpowerfuture.com/red-future/common/http"
|
||||
"gitea.redpowerfuture.com/red-future/common/jaeger"
|
||||
@@ -12,11 +17,12 @@ import (
|
||||
_ "github.com/gogf/gf/contrib/drivers/pgsql/v2"
|
||||
_ "github.com/gogf/gf/contrib/nosql/redis/v2"
|
||||
"github.com/gogf/gf/v2/frame/g"
|
||||
"golang.org/x/net/context"
|
||||
"github.com/gogf/gf/v2/os/gctx"
|
||||
"github.com/sirupsen/logrus"
|
||||
)
|
||||
|
||||
func main() {
|
||||
ctx := context.Background()
|
||||
ctx := gctx.New()
|
||||
defer jaeger.ShutDown(ctx)
|
||||
|
||||
// 启动自动同步（后台循环执行，首次全量后续增量）
|
||||
@@ -28,10 +34,20 @@ func main() {
|
||||
dict.DatasourcePlatform,
|
||||
// 平台同步引擎
|
||||
syncCtrl.PlatformSyncController,
|
||||
// 公共查询接口
|
||||
public.PublicQuery,
|
||||
// 报表引擎 CRUD API
|
||||
reportCtrl.ReportController,
|
||||
})
|
||||
|
||||
// 管理后台页面
|
||||
g.Server().BindHandler("/admin", debug.DebugController.DebugPage)
|
||||
// 报表引擎管理页面
|
||||
g.Server().BindHandler("/admin/report", reportCtrl.ReportAdminPage)
|
||||
|
||||
select {}
|
||||
// 捕获退出信号，实现优雅关闭
|
||||
quit := make(chan os.Signal, 1)
|
||||
signal.Notify(quit, syscall.SIGINT, syscall.SIGTERM)
|
||||
sig := <-quit
|
||||
logrus.Infof("收到信号 %v，正在退出...", sig)
|
||||
}
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
package api_feature
|
||||
package dict
|
||||
|
||||
import (
|
||||
"dataengine/consts/api-feature"
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
package api_feature
|
||||
package dict
|
||||
|
||||
import (
|
||||
"dataengine/consts/api-feature"
|
||||
|
||||
@@ -0,0 +1,62 @@
|
||||
package public
|
||||
|
||||
import (
|
||||
"github.com/gogf/gf/v2/frame/g"
|
||||
)
|
||||
|
||||
// QueryReq 公共查询请求
|
||||
type QueryReq struct {
|
||||
g.Meta `path:"/public/query" method:"post" tags:"公共查询" summary:"公共数据查询" dc:"支持字段/表名/过滤/分组/分页的通用查询接口"`
|
||||
Table string `json:"table" v:"required" dc:"表名（白名单限制）"`
|
||||
Fields string `json:"fields" dc:"查询字段，逗号分隔，默认 *"`
|
||||
Where map[string]interface{} `json:"where" dc:"过滤条件，支持操作符: =, !=, >, <, >=, <=, like, in, between"`
|
||||
GroupBy string `json:"groupBy" dc:"分组字段，逗号分隔"`
|
||||
OrderBy string `json:"orderBy" dc:"排序字段，如: create_time desc"`
|
||||
Page int `json:"page" dc:"页码，默认1" d:"1"`
|
||||
PageSize int `json:"pageSize" dc:"每页条数，默认20，最大100" d:"20"`
|
||||
}
|
||||
|
||||
// QueryRes 公共查询响应
|
||||
type QueryRes struct {
|
||||
List []map[string]interface{} `json:"list" dc:"数据列表"`
|
||||
Total int64 `json:"total" dc:"总数"`
|
||||
Page int `json:"page" dc:"当前页码"`
|
||||
Size int `json:"size" dc:"每页条数"`
|
||||
}
|
||||
|
||||
// TableListReq 获取可查询表列表请求
|
||||
type TableListReq struct {
|
||||
g.Meta `path:"/public/tables" method:"get" tags:"公共查询" summary:"获取可查询表列表" dc:"获取配置了表结构的同步表列表"`
|
||||
}
|
||||
|
||||
// TableListRes 获取可查询表列表响应
|
||||
type TableListRes struct {
|
||||
List []TableInfo `json:"list" dc:"表列表"`
|
||||
}
|
||||
|
||||
// TableInfo 表信息
|
||||
type TableInfo struct {
|
||||
TableName string `json:"tableName" dc:"表名"`
|
||||
PlatformName string `json:"platformName" dc:"平台名称"`
|
||||
InterfaceName string `json:"interfaceName" dc:"接口名称"`
|
||||
Columns []string `json:"columns" dc:"可用字段列表"`
|
||||
}
|
||||
|
||||
// ColumnListReq 获取表字段列表请求
|
||||
type ColumnListReq struct {
|
||||
g.Meta `path:"/public/tables/{table}/columns" method:"get" tags:"公共查询" summary:"获取表字段列表" dc:"获取指定表的字段列表"`
|
||||
Table string `json:"table" dc:"表名"`
|
||||
}
|
||||
|
||||
// ColumnListRes 获取表字段列表响应
|
||||
type ColumnListRes struct {
|
||||
TableName string `json:"tableName" dc:"表名"`
|
||||
Columns []Column `json:"columns" dc:"字段列表"`
|
||||
}
|
||||
|
||||
// Column 字段信息
|
||||
type Column struct {
|
||||
Name string `json:"name" dc:"字段名"`
|
||||
Type string `json:"type" dc:"字段类型"`
|
||||
Comment string `json:"comment" dc:"字段说明"`
|
||||
}
|
||||
@@ -36,10 +36,10 @@ type DatasourcePlatform struct {
|
||||
// 自定义认证配置 (JSONB)
|
||||
AuthConfig map[string]interface{} `orm:"auth_config" json:"authConfig" description:"自定义认证配置，支持各平台特有的认证方式"`
|
||||
|
||||
// 元数据
|
||||
CreatedBy string `orm:"created_by" json:"createdBy" description:"创建人"`
|
||||
// 元数据（ORM tag 与数据库列名一致：creator/updater）
|
||||
CreatedBy string `orm:"creator" json:"createdBy" description:"创建人"`
|
||||
CreatedAt *time.Time `orm:"created_at" json:"createdAt" description:"创建时间"`
|
||||
UpdatedBy string `orm:"updated_by" json:"updatedBy" description:"更新人"`
|
||||
UpdatedBy string `orm:"updater" json:"updatedBy" description:"更新人"`
|
||||
UpdatedAt *time.Time `orm:"updated_at" json:"updatedAt" description:"更新时间"`
|
||||
Version int `orm:"version" json:"version" description:"版本号(乐观锁)"`
|
||||
}
|
||||
@@ -64,9 +64,9 @@ type DatasourcePlatformCol struct {
|
||||
MaxRetries string
|
||||
RetryDelayMs string
|
||||
AuthConfig string
|
||||
CreatedBy string
|
||||
CreatedBy string // 对应 DB 列 creator
|
||||
CreatedAt string
|
||||
UpdatedBy string
|
||||
UpdatedBy string // 对应 DB 列 updater
|
||||
UpdatedAt string
|
||||
Version string
|
||||
}
|
||||
@@ -91,9 +91,9 @@ var DatasourcePlatformCols = DatasourcePlatformCol{
|
||||
MaxRetries: "max_retries",
|
||||
RetryDelayMs: "retry_delay_ms",
|
||||
AuthConfig: "auth_config",
|
||||
CreatedBy: "created_by",
|
||||
CreatedBy: "creator",
|
||||
CreatedAt: "created_at",
|
||||
UpdatedBy: "updated_by",
|
||||
UpdatedBy: "updater",
|
||||
UpdatedAt: "updated_at",
|
||||
Version: "version",
|
||||
}
|
||||
|
||||
@@ -1,132 +1,17 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"time"
|
||||
|
||||
dao "dataengine/dao/copydata"
|
||||
taskDto "dataengine/model/dto/copydata"
|
||||
syncSvc "dataengine/service/sync"
|
||||
|
||||
"gitea.redpowerfuture.com/red-future/common/beans"
|
||||
_ "github.com/gogf/gf/contrib/drivers/pgsql/v2"
|
||||
"github.com/gogf/gf/v2/frame/g"
|
||||
"github.com/gogf/gf/v2/os/gctx"
|
||||
"github.com/sirupsen/logrus"
|
||||
)
|
||||
|
||||
// CompensationScheduler 通用补偿调度器
|
||||
type CompensationScheduler struct {
|
||||
interval time.Duration
|
||||
}
|
||||
|
||||
// NewCompensationScheduler 创建调度器
|
||||
func NewCompensationScheduler() *CompensationScheduler {
|
||||
ctx := gctx.New()
|
||||
sec := g.Cfg().MustGet(ctx, "sync.compensation_interval_seconds", 300).Int()
|
||||
if sec < 10 {
|
||||
sec = 300
|
||||
}
|
||||
return &CompensationScheduler{interval: time.Duration(sec) * time.Second}
|
||||
}
|
||||
|
||||
// RunOnce 执行一次补偿
|
||||
func (s *CompensationScheduler) RunOnce() {
|
||||
ctx := gctx.New()
|
||||
ctx = context.WithValue(ctx, "user", &beans.User{UserName: "admin", TenantId: 1})
|
||||
|
||||
logrus.Info("=== 开始补偿扫描 ===")
|
||||
|
||||
// 查询所有 failed 状态的任务，不限类型
|
||||
tasks, err := dao.SyncTaskLog.QueryFailedTasks(ctx, &taskDto.QueryFailedTasksReq{
|
||||
Status: []string{"failed"},
|
||||
Limit: 50,
|
||||
})
|
||||
if err != nil {
|
||||
logrus.Errorf("查询失败任务异常: %v", err)
|
||||
return
|
||||
}
|
||||
if len(tasks) == 0 {
|
||||
logrus.Info("当前没有需要补偿的任务")
|
||||
return
|
||||
}
|
||||
|
||||
logrus.Infof("发现 %d 个失败任务", len(tasks))
|
||||
|
||||
for _, task := range tasks {
|
||||
if task.RetryCount >= task.MaxRetry {
|
||||
logrus.Warnf("任务 %s 已达最大重试次数 %d", task.TaskID, task.MaxRetry)
|
||||
dao.SyncTaskLog.Update(ctx, &taskDto.UpdateSyncTaskLogReq{
|
||||
ID: task.Id,
|
||||
Status: "manual_review",
|
||||
ErrorMessage: fmt.Sprintf("已达最大重试次数 %d", task.MaxRetry),
|
||||
})
|
||||
continue
|
||||
}
|
||||
|
||||
platformCode := task.PlatformCode
|
||||
interfaceCode := task.InterfaceCode
|
||||
if platformCode == "" || interfaceCode == "" {
|
||||
logrus.Warnf("任务 %s 缺少 platform_code 或 interface_code，跳过", task.TaskID)
|
||||
continue
|
||||
}
|
||||
|
||||
logrus.Infof("补偿: %s/%s (第 %d 次)", platformCode, interfaceCode, task.RetryCount+1)
|
||||
|
||||
// 更新状态为 retrying
|
||||
retryCount := task.RetryCount + 1
|
||||
dao.SyncTaskLog.Update(ctx, &taskDto.UpdateSyncTaskLogReq{
|
||||
ID: task.Id,
|
||||
Status: "retrying",
|
||||
RetryCount: &retryCount,
|
||||
})
|
||||
|
||||
// 执行补偿（增量同步）
|
||||
_, err := syncSvc.SyncByConfig(ctx, platformCode, interfaceCode, false)
|
||||
if err != nil {
|
||||
logrus.Errorf("补偿失败: %v", err)
|
||||
// 更新状态为 failed，设置下次重试时间
|
||||
backoff := []int{5, 15, 30, 60, 120}
|
||||
waitMin := 5
|
||||
if retryCount <= len(backoff) {
|
||||
waitMin = backoff[retryCount-1]
|
||||
} else {
|
||||
waitMin = backoff[len(backoff)-1]
|
||||
}
|
||||
nextRetry := time.Now().Add(time.Duration(waitMin) * time.Minute)
|
||||
dao.SyncTaskLog.Update(ctx, &taskDto.UpdateSyncTaskLogReq{
|
||||
ID: task.Id,
|
||||
Status: "failed",
|
||||
ErrorMessage: err.Error(),
|
||||
ErrorCode: "COMPENSATION_FAILED",
|
||||
NextRetryTime: nextRetry,
|
||||
})
|
||||
} else {
|
||||
logrus.Infof("补偿成功: %s/%s", platformCode, interfaceCode)
|
||||
now := time.Now()
|
||||
dao.SyncTaskLog.Update(ctx, &taskDto.UpdateSyncTaskLogReq{
|
||||
ID: task.Id,
|
||||
Status: "success",
|
||||
CompletedAt: now,
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
logrus.Info("=== 补偿扫描完成 ===")
|
||||
}
|
||||
|
||||
// Start 启动
|
||||
func (s *CompensationScheduler) Start() {
|
||||
logrus.Infof("补偿调度器启动，间隔: %v", s.interval)
|
||||
s.RunOnce()
|
||||
ticker := time.NewTicker(s.interval)
|
||||
defer ticker.Stop()
|
||||
for range ticker.C {
|
||||
s.RunOnce()
|
||||
}
|
||||
}
|
||||
|
||||
func main() {
|
||||
NewCompensationScheduler().Start()
|
||||
ctx := gctx.New()
|
||||
logrus.Info("独立补偿调度器启动")
|
||||
syncSvc.StartCompensation(ctx)
|
||||
// StartCompensation 内部有永续循环，不会返回
|
||||
select {}
|
||||
}
|
||||
|
||||
@@ -79,9 +79,9 @@ func (s *datasourcePlatformService) List(ctx context.Context, req *dto.ListDatas
|
||||
MaxRetries: item.MaxRetries,
|
||||
RetryDelayMs: item.RetryDelayMs,
|
||||
CreatedBy: item.CreatedBy,
|
||||
CreatedAt: item.CreatedAt.Unix(),
|
||||
CreatedAt: s.safeUnix(item.CreatedAt),
|
||||
UpdatedBy: item.UpdatedBy,
|
||||
UpdatedAt: item.UpdatedAt.Unix(),
|
||||
UpdatedAt: s.safeUnix(item.UpdatedAt),
|
||||
})
|
||||
}
|
||||
|
||||
@@ -294,6 +294,14 @@ func (s *datasourcePlatformService) getAuthTypeName(authType string) string {
|
||||
return authType
|
||||
}
|
||||
|
||||
// safeUnix 安全地从 *time.Time 获取 Unix 时间戳，nil 返回 0
|
||||
func (s *datasourcePlatformService) safeUnix(t *time.Time) int64 {
|
||||
if t == nil {
|
||||
return 0
|
||||
}
|
||||
return t.Unix()
|
||||
}
|
||||
|
||||
// BatchUpdateStatus 批量更新平台状态
|
||||
func (s *datasourcePlatformService) BatchUpdateStatus(ctx context.Context, ids []int64, status string, updatedBy string) (err error) {
|
||||
if len(ids) == 0 {
|
||||
|
||||
@@ -7,6 +7,8 @@ import (
|
||||
dto "dataengine/model/dto/dict"
|
||||
entity "dataengine/model/entity/dict"
|
||||
"errors"
|
||||
|
||||
"github.com/gogf/gf/v2/os/gtime"
|
||||
)
|
||||
|
||||
type apiInterfaceService struct{}
|
||||
@@ -85,8 +87,8 @@ func (s *apiInterfaceService) List(ctx context.Context, req *dto.ListApiInterfac
|
||||
Method: item.Method,
|
||||
Status: item.Status,
|
||||
StatusName: s.getStatusName(item.Status),
|
||||
CreatedAt: item.CreatedAt.Unix(),
|
||||
UpdatedAt: item.UpdatedAt.Unix(),
|
||||
CreatedAt: safeUnix(item.CreatedAt),
|
||||
UpdatedAt: safeUnix(item.UpdatedAt),
|
||||
})
|
||||
}
|
||||
|
||||
@@ -181,3 +183,11 @@ func (s *apiInterfaceService) getStatusName(status consts.PlatformStatus) string
|
||||
}
|
||||
return string(status)
|
||||
}
|
||||
|
||||
// safeUnix 安全地从 *gtime.Time 获取 Unix 时间戳，nil 返回 0
|
||||
func safeUnix(t *gtime.Time) int64 {
|
||||
if t == nil {
|
||||
return 0
|
||||
}
|
||||
return t.Unix()
|
||||
}
|
||||
|
||||
@@ -0,0 +1,524 @@
|
||||
package public
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"regexp"
|
||||
"strings"
|
||||
|
||||
"dataengine/model/dto/public"
|
||||
"dataengine/model/entity/dict"
|
||||
|
||||
"gitea.redpowerfuture.com/red-future/common/db/gfdb"
|
||||
"github.com/gogf/gf/v2/util/gconv"
|
||||
)
|
||||
|
||||
var PublicQuery = new(publicQueryService)
|
||||
|
||||
// tableColumnsCache 表定义缓存
|
||||
var tableColumnsCache = make(map[string][]string)
|
||||
|
||||
type publicQueryService struct{}
|
||||
|
||||
// Query 执行公共查询
|
||||
func (s *publicQueryService) Query(ctx context.Context, req *public.QueryReq) (res *public.QueryRes, err error) {
|
||||
// 1. 验证表名白名单
|
||||
if err = s.validateTable(ctx, req.Table); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
// 2. 验证字段白名单
|
||||
allowedFields, err := s.getAllowedFields(ctx, req.Table)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
// 3. 构建 SELECT 部分
|
||||
selectFields := "*"
|
||||
if req.Fields != "" {
|
||||
selectFields, err = s.buildSelectFields(req.Fields, allowedFields)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
}
|
||||
|
||||
// 4. 构建 WHERE 条件
|
||||
whereClause, whereArgs, err := s.buildWhereClause(req.Where, allowedFields)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
// 5. 构建 GROUP BY
|
||||
groupByClause, err := s.buildGroupBy(req.GroupBy, allowedFields)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
// 6. 强制租户过滤
|
||||
tenantClause := s.buildTenantClause(allowedFields)
|
||||
|
||||
// 7. 组合完整 WHERE
|
||||
fullWhere := tenantClause
|
||||
if whereClause != "" {
|
||||
if fullWhere != "" {
|
||||
fullWhere += " AND " + whereClause
|
||||
} else {
|
||||
fullWhere = whereClause
|
||||
}
|
||||
}
|
||||
|
||||
// 8. 校验分页参数
|
||||
if req.Page < 1 {
|
||||
req.Page = 1
|
||||
}
|
||||
if req.PageSize < 1 {
|
||||
req.PageSize = 20
|
||||
}
|
||||
if req.PageSize > 100 {
|
||||
req.PageSize = 100
|
||||
}
|
||||
offset := (req.Page - 1) * req.PageSize
|
||||
|
||||
// 9. 构建 ORDER BY
|
||||
orderByClause, err := s.buildOrderBy(req.OrderBy, allowedFields)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
// 10. 统计总数
|
||||
countSql := fmt.Sprintf("SELECT COUNT(*) FROM %s", req.Table)
|
||||
if fullWhere != "" {
|
||||
countSql += " WHERE " + fullWhere
|
||||
}
|
||||
if groupByClause != "" {
|
||||
countSql = fmt.Sprintf("SELECT COUNT(*) FROM (SELECT 1 FROM %s WHERE %s GROUP BY %s) AS t",
|
||||
req.Table, fullWhere, groupByClause)
|
||||
}
|
||||
|
||||
result, err := gfdb.DB(ctx).GetAll(ctx, countSql, whereArgs...)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("统计总数失败: %v", err)
|
||||
}
|
||||
var total int64
|
||||
if result.Len() > 0 {
|
||||
total = result[0]["count"].Int64()
|
||||
}
|
||||
|
||||
// 11. 查询数据
|
||||
querySql := fmt.Sprintf("SELECT %s FROM %s", selectFields, req.Table)
|
||||
if fullWhere != "" {
|
||||
querySql += " WHERE " + fullWhere
|
||||
}
|
||||
if groupByClause != "" {
|
||||
querySql += " GROUP BY " + groupByClause
|
||||
}
|
||||
if orderByClause != "" {
|
||||
querySql += " ORDER BY " + orderByClause
|
||||
}
|
||||
querySql += fmt.Sprintf(" LIMIT %d OFFSET %d", req.PageSize, offset)
|
||||
|
||||
dataResult, err := gfdb.DB(ctx).GetAll(ctx, querySql, whereArgs...)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("查询数据失败: %v", err)
|
||||
}
|
||||
|
||||
var list []map[string]interface{}
|
||||
if dataResult.Len() > 0 {
|
||||
list = dataResult.List()
|
||||
}
|
||||
|
||||
return &public.QueryRes{
|
||||
List: list,
|
||||
Total: total,
|
||||
Page: req.Page,
|
||||
Size: req.PageSize,
|
||||
}, nil
|
||||
}
|
||||
|
||||
// GetTableList 获取可查询表列表
|
||||
func (s *publicQueryService) GetTableList(ctx context.Context) (*public.TableListRes, error) {
|
||||
var ifaces []dict.ApiInterface
|
||||
err := gfdb.DB(ctx).Model(ctx, "api_interface").
|
||||
Where("table_definition IS NOT NULL").
|
||||
Where("table_definition->>'table_name' != ''").
|
||||
Where("status", "active").
|
||||
Scan(&ifaces)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("查询表列表失败: %v", err)
|
||||
}
|
||||
|
||||
// 查询平台名称
|
||||
var platforms []dict.DatasourcePlatform
|
||||
_ = gfdb.DB(ctx).Model(ctx, "api_datasource_platform").Scan(&platforms)
|
||||
platformMap := make(map[int64]string)
|
||||
for _, p := range platforms {
|
||||
platformMap[p.ID] = p.PlatformName
|
||||
}
|
||||
|
||||
var list []public.TableInfo
|
||||
for _, iface := range ifaces {
|
||||
tableName := s.getStringFromMap(iface.TableDefinition, "table_name")
|
||||
if tableName == "" {
|
||||
continue
|
||||
}
|
||||
|
||||
columns := s.extractColumnsFromMap(iface.TableDefinition)
|
||||
list = append(list, public.TableInfo{
|
||||
TableName: tableName,
|
||||
PlatformName: platformMap[iface.PlatformId],
|
||||
InterfaceName: iface.Name,
|
||||
Columns: columns,
|
||||
})
|
||||
}
|
||||
|
||||
return &public.TableListRes{List: list}, nil
|
||||
}
|
||||
|
||||
// GetColumnList 获取表字段列表
|
||||
func (s *publicQueryService) GetColumnList(ctx context.Context, tableName string) (*public.ColumnListRes, error) {
|
||||
if err := s.validateTable(ctx, tableName); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
columns, err := s.getColumnDetails(ctx, tableName)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
return &public.ColumnListRes{
|
||||
TableName: tableName,
|
||||
Columns: columns,
|
||||
}, nil
|
||||
}
|
||||
|
||||
// validateTable 验证表名白名单
|
||||
func (s *publicQueryService) validateTable(ctx context.Context, tableName string) error {
|
||||
if tableName == "" {
|
||||
return fmt.Errorf("表名不能为空")
|
||||
}
|
||||
|
||||
// 表名格式校验
|
||||
if matched, _ := regexp.MatchString(`^[a-zA-Z_][a-zA-Z0-9_]*$`, tableName); !matched {
|
||||
return fmt.Errorf("表名格式非法，只允许字母、数字、下划线")
|
||||
}
|
||||
|
||||
// 禁止系统表
|
||||
systemTables := []string{"pg_catalog", "information_schema"}
|
||||
for _, t := range systemTables {
|
||||
if strings.HasPrefix(strings.ToLower(tableName), t) {
|
||||
return fmt.Errorf("禁止查询系统表")
|
||||
}
|
||||
}
|
||||
|
||||
// 检查白名单
|
||||
count, err := gfdb.DB(ctx).Model(ctx, "api_interface").
|
||||
Where("table_definition->>'table_name' = ?", tableName).
|
||||
Where("status", "active").
|
||||
Count()
|
||||
if err != nil {
|
||||
return fmt.Errorf("表名验证失败: %v", err)
|
||||
}
|
||||
if count == 0 {
|
||||
return fmt.Errorf("表 [%s] 不在可查询白名单中", tableName)
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// getAllowedFields 获取表允许的字段
|
||||
func (s *publicQueryService) getAllowedFields(ctx context.Context, tableName string) ([]string, error) {
|
||||
if cols, ok := tableColumnsCache[tableName]; ok {
|
||||
return cols, nil
|
||||
}
|
||||
|
||||
var iface dict.ApiInterface
|
||||
_, err := gfdb.DB(ctx).Model(ctx, "api_interface").
|
||||
Where("table_definition->>'table_name' = ?", tableName).
|
||||
Where("status", "active").
|
||||
One(&iface)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("获取表字段失败: %v", err)
|
||||
}
|
||||
|
||||
columns := s.extractColumnsFromMap(iface.TableDefinition)
|
||||
tableColumnsCache[tableName] = columns
|
||||
return columns, nil
|
||||
}
|
||||
|
||||
// extractColumnsFromMap 从 map 中提取字段
|
||||
func (s *publicQueryService) extractColumnsFromMap(tableDef map[string]interface{}) []string {
|
||||
var columns []string
|
||||
if cols, ok := tableDef["columns"].([]interface{}); ok {
|
||||
for _, c := range cols {
|
||||
if col, ok := c.(map[string]interface{}); ok {
|
||||
if name, ok := col["name"].(string); ok {
|
||||
columns = append(columns, name)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
return columns
|
||||
}
|
||||
|
||||
// buildSelectFields 构建 SELECT 字段
|
||||
func (s *publicQueryService) buildSelectFields(fields string, allowedFields []string) (string, error) {
|
||||
allowedMap := make(map[string]bool)
|
||||
for _, f := range allowedFields {
|
||||
allowedMap[strings.ToLower(f)] = true
|
||||
}
|
||||
allowedMap["id"] = true
|
||||
allowedMap["tenant_id"] = true
|
||||
allowedMap["created_at"] = true
|
||||
allowedMap["updated_at"] = true
|
||||
allowedMap["raw_data"] = true
|
||||
|
||||
var result []string
|
||||
for _, f := range strings.Split(fields, ",") {
|
||||
f = strings.TrimSpace(f)
|
||||
if f == "" {
|
||||
continue
|
||||
}
|
||||
if strings.Contains(f, " ") || strings.Contains(f, "(") {
|
||||
result = append(result, f)
|
||||
continue
|
||||
}
|
||||
fLower := strings.ToLower(f)
|
||||
if !allowedMap[fLower] {
|
||||
return "", fmt.Errorf("字段 [%s] 不在允许列表中", f)
|
||||
}
|
||||
result = append(result, f)
|
||||
}
|
||||
|
||||
if len(result) == 0 {
|
||||
return "*", nil
|
||||
}
|
||||
return strings.Join(result, ", "), nil
|
||||
}
|
||||
|
||||
// buildWhereClause 构建 WHERE 条件
|
||||
func (s *publicQueryService) buildWhereClause(where map[string]interface{}, allowedFields []string) (string, []interface{}, error) {
|
||||
if len(where) == 0 {
|
||||
return "", nil, nil
|
||||
}
|
||||
|
||||
allowedMap := make(map[string]bool)
|
||||
for _, f := range allowedFields {
|
||||
allowedMap[strings.ToLower(f)] = true
|
||||
}
|
||||
allowedMap["tenant_id"] = true
|
||||
|
||||
var conditions []string
|
||||
var args []interface{}
|
||||
|
||||
for field, value := range where {
|
||||
fieldLower := strings.ToLower(field)
|
||||
if !allowedMap[fieldLower] {
|
||||
return "", nil, fmt.Errorf("字段 [%s] 不在允许列表中", field)
|
||||
}
|
||||
|
||||
// 处理操作符后缀
|
||||
opSuffixes := []struct {
|
||||
suffix string
|
||||
format string
|
||||
like bool
|
||||
}{
|
||||
{"_eq", "%s = ?", false},
|
||||
{"_ne", "%s != ?", false},
|
||||
{"_gt", "%s > ?", false},
|
||||
{"_lt", "%s < ?", false},
|
||||
{"_ge", "%s >= ?", false},
|
||||
{"_le", "%s <= ?", false},
|
||||
{"_like", "%s LIKE ?", true},
|
||||
}
|
||||
|
||||
matched := false
|
||||
for _, op := range opSuffixes {
|
||||
if strings.HasSuffix(fieldLower, op.suffix) {
|
||||
cleanField := field[:len(field)-len(op.suffix)]
|
||||
conditions = append(conditions, fmt.Sprintf(op.format, cleanField))
|
||||
if op.like {
|
||||
args = append(args, "%"+gconv.String(value)+"%")
|
||||
} else {
|
||||
args = append(args, value)
|
||||
}
|
||||
matched = true
|
||||
break
|
||||
}
|
||||
}
|
||||
if matched {
|
||||
continue
|
||||
}
|
||||
|
||||
// 处理 _in
|
||||
if strings.HasSuffix(fieldLower, "_in") {
|
||||
cleanField := field[:len(field)-3]
|
||||
if arr, ok := value.([]interface{}); ok {
|
||||
placeholders := make([]string, len(arr))
|
||||
for i, v := range arr {
|
||||
placeholders[i] = "?"
|
||||
args = append(args, v)
|
||||
}
|
||||
conditions = append(conditions, fmt.Sprintf("%s IN (%s)", cleanField, strings.Join(placeholders, ",")))
|
||||
} else if str, ok := value.(string); ok {
|
||||
parts := strings.Split(str, ",")
|
||||
placeholders := make([]string, len(parts))
|
||||
for i, p := range parts {
|
||||
placeholders[i] = "?"
|
||||
args = append(args, strings.TrimSpace(p))
|
||||
}
|
||||
conditions = append(conditions, fmt.Sprintf("%s IN (%s)", cleanField, strings.Join(placeholders, ",")))
|
||||
}
|
||||
continue
|
||||
}
|
||||
|
||||
// 处理 _between
|
||||
if strings.HasSuffix(fieldLower, "_between") {
|
||||
cleanField := field[:len(field)-8]
|
||||
if arr, ok := value.([]interface{}); ok && len(arr) >= 2 {
|
||||
conditions = append(conditions, fmt.Sprintf("%s BETWEEN ? AND ?", cleanField))
|
||||
args = append(args, arr[0], arr[1])
|
||||
} else if arr, ok := value.([]string); ok && len(arr) >= 2 {
|
||||
conditions = append(conditions, fmt.Sprintf("%s BETWEEN ? AND ?", cleanField))
|
||||
args = append(args, arr[0], arr[1])
|
||||
}
|
||||
continue
|
||||
}
|
||||
|
||||
// 默认等于
|
||||
conditions = append(conditions, fmt.Sprintf("%s = ?", field))
|
||||
args = append(args, value)
|
||||
}
|
||||
|
||||
return strings.Join(conditions, " AND "), args, nil
|
||||
}
|
||||
|
||||
// buildGroupBy 构建 GROUP BY
|
||||
func (s *publicQueryService) buildGroupBy(groupBy string, allowedFields []string) (string, error) {
|
||||
if groupBy == "" {
|
||||
return "", nil
|
||||
}
|
||||
|
||||
allowedMap := make(map[string]bool)
|
||||
for _, f := range allowedFields {
|
||||
allowedMap[strings.ToLower(f)] = true
|
||||
}
|
||||
|
||||
var fields []string
|
||||
for _, f := range strings.Split(groupBy, ",") {
|
||||
f = strings.TrimSpace(f)
|
||||
if f == "" {
|
||||
continue
|
||||
}
|
||||
if !allowedMap[strings.ToLower(f)] {
|
||||
return "", fmt.Errorf("分组字段 [%s] 不在允许列表中", f)
|
||||
}
|
||||
fields = append(fields, f)
|
||||
}
|
||||
|
||||
return strings.Join(fields, ", "), nil
|
||||
}
|
||||
|
||||
// buildOrderBy 构建 ORDER BY
|
||||
func (s *publicQueryService) buildOrderBy(orderBy string, allowedFields []string) (string, error) {
|
||||
if orderBy == "" {
|
||||
return "", nil
|
||||
}
|
||||
|
||||
allowedMap := make(map[string]bool)
|
||||
for _, f := range allowedFields {
|
||||
allowedMap[strings.ToLower(f)] = true
|
||||
}
|
||||
allowedMap["id"] = true
|
||||
allowedMap["created_at"] = true
|
||||
allowedMap["updated_at"] = true
|
||||
|
||||
var clauses []string
|
||||
for _, part := range strings.Split(orderBy, ",") {
|
||||
part = strings.TrimSpace(part)
|
||||
if part == "" {
|
||||
continue
|
||||
}
|
||||
|
||||
parts := strings.Fields(part)
|
||||
if len(parts) == 0 {
|
||||
continue
|
||||
}
|
||||
|
||||
field := parts[0]
|
||||
dir := "ASC"
|
||||
if len(parts) > 1 {
|
||||
if strings.ToUpper(parts[1]) == "DESC" {
|
||||
dir = "DESC"
|
||||
}
|
||||
}
|
||||
|
||||
if !allowedMap[strings.ToLower(field)] {
|
||||
return "", fmt.Errorf("排序字段 [%s] 不在允许列表中", field)
|
||||
}
|
||||
field = regexp.MustCompile(`[^a-zA-Z0-9_]`).ReplaceAllString(field, "")
|
||||
clauses = append(clauses, field+" "+dir)
|
||||
}
|
||||
|
||||
return strings.Join(clauses, ", "), nil
|
||||
}
|
||||
|
||||
// buildTenantClause 构建租户过滤条件
|
||||
func (s *publicQueryService) buildTenantClause(allowedFields []string) string {
|
||||
for _, f := range allowedFields {
|
||||
if strings.ToLower(f) == "tenant_id" {
|
||||
return "tenant_id = 1"
|
||||
}
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
// getColumnDetails 获取表字段详情
|
||||
func (s *publicQueryService) getColumnDetails(ctx context.Context, tableName string) ([]public.Column, error) {
|
||||
var iface dict.ApiInterface
|
||||
_, err := gfdb.DB(ctx).Model(ctx, "api_interface").
|
||||
Where("table_definition->>'table_name' = ?", tableName).
|
||||
Where("status", "active").
|
||||
One(&iface)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("获取表字段详情失败: %v", err)
|
||||
}
|
||||
|
||||
var columns []public.Column
|
||||
if cols, ok := iface.TableDefinition["columns"].([]interface{}); ok {
|
||||
for _, c := range cols {
|
||||
if col, ok := c.(map[string]interface{}); ok {
|
||||
columns = append(columns, public.Column{
|
||||
Name: gconv.String(col["name"]),
|
||||
Type: gconv.String(col["type"]),
|
||||
Comment: gconv.String(col["comment"]),
|
||||
})
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
columns = append(columns, public.Column{Name: "id", Type: "BIGINT", Comment: "主键ID"})
|
||||
columns = append(columns, public.Column{Name: "tenant_id", Type: "BIGINT", Comment: "租户ID"})
|
||||
columns = append(columns, public.Column{Name: "created_at", Type: "TIMESTAMP", Comment: "创建时间"})
|
||||
columns = append(columns, public.Column{Name: "updated_at", Type: "TIMESTAMP", Comment: "更新时间"})
|
||||
columns = append(columns, public.Column{Name: "raw_data", Type: "JSONB", Comment: "原始数据"})
|
||||
|
||||
return columns, nil
|
||||
}
|
||||
|
||||
// getStringFromMap 从 map 中获取字符串值
|
||||
func (s *publicQueryService) getStringFromMap(data map[string]interface{}, key string) string {
|
||||
if v, ok := data[key].(string); ok {
|
||||
return v
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
// ClearTableCache 清除表缓存
|
||||
func (s *publicQueryService) ClearTableCache() {
|
||||
tableColumnsCache = make(map[string][]string)
|
||||
}
|
||||
|
||||
// InvalidateTableCache 失效指定表的缓存
|
||||
func (s *publicQueryService) InvalidateTableCache(tableName string) {
|
||||
delete(tableColumnsCache, tableName)
|
||||
}
|
||||
+123
-14
@@ -29,7 +29,7 @@ type ApiResult struct {
|
||||
type ApiClient struct {
|
||||
config *PlatformConfig
|
||||
client *http.Client
|
||||
rateLimiter <-chan time.Time // 限流 ticker
|
||||
rateLimiter *time.Ticker // 限流 ticker，可被 GC
|
||||
}
|
||||
|
||||
// NewApiClient 创建客户端
|
||||
@@ -38,14 +38,22 @@ func NewApiClient(config *PlatformConfig) *ApiClient {
|
||||
if config.RequestTimeoutMs > 0 {
|
||||
timeout = time.Duration(config.RequestTimeoutMs) * time.Millisecond
|
||||
}
|
||||
transport := &http.Transport{
|
||||
MaxIdleConns: 100,
|
||||
MaxIdleConnsPerHost: 20,
|
||||
IdleConnTimeout: 90 * time.Second,
|
||||
}
|
||||
ac := &ApiClient{
|
||||
config: config,
|
||||
client: &http.Client{Timeout: timeout},
|
||||
client: &http.Client{
|
||||
Timeout: timeout,
|
||||
Transport: transport,
|
||||
},
|
||||
}
|
||||
// 初始化限流
|
||||
if config.RateLimitPerMinute > 0 {
|
||||
interval := time.Minute / time.Duration(config.RateLimitPerMinute)
|
||||
ac.rateLimiter = time.Tick(interval)
|
||||
ac.rateLimiter = time.NewTicker(interval)
|
||||
logrus.Infof("限流已启用: %d 次/分钟, 间隔 %v", config.RateLimitPerMinute, interval)
|
||||
}
|
||||
return ac
|
||||
@@ -61,6 +69,13 @@ func (c *ApiClient) PostJSON(ctx context.Context, path string, body interface{})
|
||||
return c.doRequest(ctx, "POST", path, body, false)
|
||||
}
|
||||
|
||||
// Close 释放客户端资源（限流 ticker）
|
||||
func (c *ApiClient) Close() {
|
||||
if c.rateLimiter != nil {
|
||||
c.rateLimiter.Stop()
|
||||
}
|
||||
}
|
||||
|
||||
// Request 通用请求方法（支持 GET/POST，支持参数在 query 或 body）
|
||||
func (c *ApiClient) Request(ctx context.Context, method, path string, params map[string]interface{}, paramsInQuery bool) (*ApiResult, error) {
|
||||
if paramsInQuery {
|
||||
@@ -99,8 +114,9 @@ func (c *ApiClient) execute(ctx context.Context, method, path string, body inter
|
||||
// 限流等待
|
||||
if c.rateLimiter != nil {
|
||||
select {
|
||||
case <-c.rateLimiter:
|
||||
case <-c.rateLimiter.C:
|
||||
case <-ctx.Done():
|
||||
c.rateLimiter.Stop()
|
||||
return nil, ctx.Err()
|
||||
}
|
||||
}
|
||||
@@ -112,8 +128,13 @@ func (c *ApiClient) execute(ctx context.Context, method, path string, body inter
|
||||
fullURL = c.applyAuthURL(fullURL)
|
||||
|
||||
var reqBody io.Reader
|
||||
var reqBodyBytes []byte
|
||||
if body != nil && !paramsInQuery {
|
||||
b, _ := json.Marshal(body)
|
||||
b, err := json.Marshal(body)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("JSON序列化请求体失败: %w", err)
|
||||
}
|
||||
reqBodyBytes = b
|
||||
reqBody = bytes.NewBuffer(b)
|
||||
}
|
||||
|
||||
@@ -133,7 +154,7 @@ func (c *ApiClient) execute(ctx context.Context, method, path string, body inter
|
||||
return nil, fmt.Errorf("创建请求失败: %w", err)
|
||||
}
|
||||
|
||||
c.applyAuthHeader(req)
|
||||
c.applyAuthHeader(req, reqBodyBytes)
|
||||
req.Header.Set("User-Agent", "data-engine/1.0")
|
||||
if body != nil && !paramsInQuery {
|
||||
req.Header.Set("Content-Type", "application/json")
|
||||
@@ -145,7 +166,10 @@ func (c *ApiClient) execute(ctx context.Context, method, path string, body inter
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
|
||||
respBody, _ := io.ReadAll(resp.Body)
|
||||
respBody, err := io.ReadAll(resp.Body)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("读取响应体失败: %w", err)
|
||||
}
|
||||
result := &ApiResult{Body: respBody, DurationMs: time.Since(start).Milliseconds()}
|
||||
|
||||
if resp.StatusCode >= 400 {
|
||||
@@ -157,7 +181,11 @@ func (c *ApiClient) execute(ctx context.Context, method, path string, body inter
|
||||
// buildQueryURL 将 params 拼接到 URL 查询参数中
|
||||
// 支持数组/对象类型的值自动 JSON 序列化 + URL 编码
|
||||
func (c *ApiClient) buildQueryURL(rawURL string, params map[string]interface{}) string {
|
||||
parsed, _ := url.Parse(rawURL)
|
||||
parsed, err := url.Parse(rawURL)
|
||||
if err != nil || parsed == nil {
|
||||
logrus.Errorf("buildQueryURL: 解析 URL 失败: %v", err)
|
||||
return rawURL
|
||||
}
|
||||
q := parsed.Query()
|
||||
|
||||
for k, v := range params {
|
||||
@@ -224,7 +252,11 @@ func (c *ApiClient) applyAuthURL(rawURL string) string {
|
||||
return rawURL
|
||||
}
|
||||
|
||||
parsed, _ := url.Parse(rawURL)
|
||||
parsed, err := url.Parse(rawURL)
|
||||
if err != nil || parsed == nil {
|
||||
logrus.Errorf("applyAuthURL: 解析 URL 失败: %v", err)
|
||||
return rawURL
|
||||
}
|
||||
q := parsed.Query()
|
||||
if tokenInQuery && token != "" {
|
||||
q.Set(queryKey, token)
|
||||
@@ -236,10 +268,16 @@ func (c *ApiClient) applyAuthURL(rawURL string) string {
|
||||
return parsed.String()
|
||||
}
|
||||
|
||||
func (c *ApiClient) applyAuthHeader(req *http.Request) {
|
||||
func (c *ApiClient) applyAuthHeader(req *http.Request, bodyBytes []byte) {
|
||||
cfg := c.config.AuthConfig
|
||||
token := c.config.AccessToken
|
||||
|
||||
// APP_SIGNATURE 认证：app-id + signature 头部（如钉钉智能薪酬）
|
||||
if c.config.AuthType == "APP_SIGNATURE" {
|
||||
c.applyAppSignatureAuth(req, bodyBytes)
|
||||
return
|
||||
}
|
||||
|
||||
if cfg != nil {
|
||||
if tiq, _ := cfg["token_in_query"].(bool); tiq {
|
||||
return
|
||||
@@ -251,9 +289,9 @@ func (c *ApiClient) applyAuthHeader(req *http.Request) {
|
||||
|
||||
if cfg != nil {
|
||||
if h, ok := cfg["header_name"].(string); ok {
|
||||
f := cfg["header_format"].(string)
|
||||
if f == "" {
|
||||
f = "{token}"
|
||||
f := "{token}"
|
||||
if fv, ok2 := cfg["header_format"].(string); ok2 {
|
||||
f = fv
|
||||
}
|
||||
req.Header.Set(h, strings.ReplaceAll(f, "{token}", token))
|
||||
return
|
||||
@@ -268,6 +306,73 @@ func (c *ApiClient) applyAuthHeader(req *http.Request) {
|
||||
}
|
||||
}
|
||||
|
||||
// applyAppSignatureAuth 设置 app-id + signature 认证头部
|
||||
func (c *ApiClient) applyAppSignatureAuth(req *http.Request, bodyBytes []byte) {
|
||||
cfg := c.config.AuthConfig
|
||||
if cfg == nil {
|
||||
return
|
||||
}
|
||||
|
||||
// 1. 设置 app-id 头部
|
||||
appIdHeader := "app-id"
|
||||
if h, _ := cfg["app_id_header"].(string); h != "" {
|
||||
appIdHeader = h
|
||||
}
|
||||
appId := c.config.AppKey
|
||||
if appId == "" {
|
||||
if aid, _ := cfg["app_id"].(string); aid != "" {
|
||||
appId = aid
|
||||
}
|
||||
}
|
||||
if appId != "" {
|
||||
req.Header.Set(appIdHeader, appId)
|
||||
}
|
||||
|
||||
// 2. 计算签名并设置 signature 头部
|
||||
signHeader := "signature"
|
||||
if h, _ := cfg["sign_header"].(string); h != "" {
|
||||
signHeader = h
|
||||
}
|
||||
|
||||
secret := c.config.AppSecret
|
||||
|
||||
signAlgo := "md5_upper_body"
|
||||
if a, _ := cfg["sign_algorithm"].(string); a != "" {
|
||||
signAlgo = a
|
||||
}
|
||||
|
||||
sig := computeBodySignature(bodyBytes, secret, signAlgo)
|
||||
if sig != "" {
|
||||
req.Header.Set(signHeader, sig)
|
||||
}
|
||||
}
|
||||
|
||||
// computeBodySignature 计算基于请求体的签名
|
||||
// 支持的算法:
|
||||
// - md5_upper_body: MD5(body_string + secret) 大写（默认，钉钉智能薪酬）
|
||||
// - md5_body: MD5(body_string + secret) 小写
|
||||
func computeBodySignature(bodyBytes []byte, secret, algo string) string {
|
||||
if secret == "" {
|
||||
return ""
|
||||
}
|
||||
bodyStr := ""
|
||||
if len(bodyBytes) > 0 {
|
||||
bodyStr = string(bodyBytes)
|
||||
}
|
||||
switch algo {
|
||||
case "md5_body", "md5_upper_body":
|
||||
h := md5.Sum([]byte(bodyStr + secret))
|
||||
sig := hex.EncodeToString(h[:])
|
||||
if algo == "md5_upper_body" {
|
||||
sig = strings.ToUpper(sig)
|
||||
}
|
||||
return sig
|
||||
default:
|
||||
logrus.Warnf("未知签名算法: %s", algo)
|
||||
return ""
|
||||
}
|
||||
}
|
||||
|
||||
func generateNonce() string {
|
||||
nanoPart := time.Now().UnixNano() % 1000000000000
|
||||
r, _ := rand.Int(rand.Reader, big.NewInt(10000))
|
||||
@@ -293,7 +398,11 @@ func (c *ApiClient) applySignature(rawURL string, body interface{}, paramsInQuer
|
||||
return rawURL
|
||||
}
|
||||
|
||||
parsed, _ := url.Parse(rawURL)
|
||||
parsed, err := url.Parse(rawURL)
|
||||
if err != nil || parsed == nil {
|
||||
logrus.Errorf("applySignature: 解析 URL 失败: %v", err)
|
||||
return rawURL
|
||||
}
|
||||
q := parsed.Query()
|
||||
|
||||
// 收集所有参数并按 key 排序
|
||||
|
||||
@@ -8,7 +8,7 @@ import (
|
||||
dao "dataengine/dao/copydata"
|
||||
taskDto "dataengine/model/dto/copydata"
|
||||
|
||||
"gitea.com/red-future/common/beans"
|
||||
"gitea.redpowerfuture.com/red-future/common/beans"
|
||||
"github.com/gogf/gf/v2/frame/g"
|
||||
"github.com/sirupsen/logrus"
|
||||
)
|
||||
|
||||
@@ -4,7 +4,7 @@ import (
|
||||
"context"
|
||||
"time"
|
||||
|
||||
"gitea.com/red-future/common/db/gfdb"
|
||||
"gitea.redpowerfuture.com/red-future/common/db/gfdb"
|
||||
"github.com/sirupsen/logrus"
|
||||
)
|
||||
|
||||
@@ -19,12 +19,8 @@ func InsertRows(ctx context.Context, tableName string, conflictKeys []string, ro
|
||||
if rows[i] == nil {
|
||||
rows[i] = make(map[string]interface{})
|
||||
}
|
||||
if _, ok := rows[i]["created_at"]; !ok {
|
||||
rows[i]["created_at"] = now
|
||||
}
|
||||
if _, ok := rows[i]["updated_at"]; !ok {
|
||||
rows[i]["updated_at"] = now
|
||||
}
|
||||
// 始终覆盖 updated_at；不设置 created_at 让数据库维护首次值（upsert 时不会覆盖）
|
||||
rows[i]["updated_at"] = now
|
||||
}
|
||||
|
||||
batchSize := 100
|
||||
|
||||
+558
-115
@@ -4,6 +4,7 @@ import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"strconv"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
@@ -13,7 +14,7 @@ import (
|
||||
taskDto "dataengine/model/dto/copydata"
|
||||
entity "dataengine/model/entity/dict"
|
||||
|
||||
"gitea.com/red-future/common/db/gfdb"
|
||||
"gitea.redpowerfuture.com/red-future/common/db/gfdb"
|
||||
"github.com/sirupsen/logrus"
|
||||
)
|
||||
|
||||
@@ -38,8 +39,20 @@ type PrefetchConfig struct {
|
||||
ValueField string `json:"value_field"`
|
||||
}
|
||||
|
||||
// RecursiveConfig 递归遍历配置（如钉钉部门树）
|
||||
type RecursiveConfig struct {
|
||||
KeyField string `json:"key_field"`
|
||||
TargetParam string `json:"target_param"`
|
||||
}
|
||||
|
||||
// SyncByConfig 执行同步
|
||||
func SyncByConfig(ctx context.Context, platformCode, interfaceCode string, isFullSync bool) (*SyncResult, error) {
|
||||
// 创建超时 context 防止单次同步卡死
|
||||
timeoutMin := GetSyncTimeout(ctx)
|
||||
timeoutCtx, cancel := context.WithTimeout(ctx, time.Duration(timeoutMin)*time.Minute)
|
||||
defer cancel()
|
||||
ctx = timeoutCtx
|
||||
|
||||
// 内存锁：防止同一个接口被并发执行（两个调度周期重叠时跳过）
|
||||
lockKey := platformCode + "/" + interfaceCode
|
||||
if _, loaded := syncRunningMap.LoadOrStore(lockKey, true); loaded {
|
||||
@@ -93,11 +106,16 @@ func SyncByConfig(ctx context.Context, platformCode, interfaceCode string, isFul
|
||||
markSyncRunning(ctx, platformCode, interfaceCode, lastSyncTime)
|
||||
|
||||
api := NewApiClient(platform)
|
||||
defer api.Close()
|
||||
|
||||
prefetch := parsePrefetchConfig(iface.RequestConfig)
|
||||
if prefetch != nil {
|
||||
return syncWithPrefetch(ctx, api, platform, iface, ifaces, td, prefetch, isFullSync, lastSyncTime, start)
|
||||
}
|
||||
recursive := parseRecursiveConfig(iface.RequestConfig)
|
||||
if recursive != nil {
|
||||
return syncRecursive(ctx, api, platform, iface, td, recursive, start)
|
||||
}
|
||||
return syncSingleAPI(ctx, api, platform, iface, td, isFullSync, lastSyncTime, start)
|
||||
}
|
||||
|
||||
@@ -119,6 +137,8 @@ func syncSingleAPI(ctx context.Context, api *ApiClient, platform *PlatformConfig
|
||||
pageSize := GetSyncPageSize(ctx)
|
||||
if ps, ok := iface.RequestConfig["page_size"].(float64); ok {
|
||||
pageSize = int(ps)
|
||||
} else if ps, ok := iface.RequestConfig["pageSize"].(float64); ok {
|
||||
pageSize = int(ps)
|
||||
}
|
||||
|
||||
taskType := "incremental"
|
||||
@@ -129,14 +149,19 @@ func syncSingleAPI(ctx context.Context, api *ApiClient, platform *PlatformConfig
|
||||
inQuery := paramsInQuery(iface)
|
||||
method := string(iface.Method)
|
||||
|
||||
// 游标分页首次请求需要 cursor=""（通过 extraParams 覆盖 buildReqBody 的 page=1 赋值）
|
||||
// 游标分页首次请求需要处理初始游标值
|
||||
firstExtra := map[string]interface{}{}
|
||||
if isCursorPagination(iface) {
|
||||
cp := "cursor"
|
||||
if p, ok := iface.RequestConfig["page_param"].(string); ok && p != "" {
|
||||
cp = p
|
||||
}
|
||||
firstExtra[cp] = ""
|
||||
// 支持 initial_cursor 配置（如钉钉HRM首次传 0）
|
||||
if icv, ok := iface.RequestConfig["initial_cursor"]; ok {
|
||||
firstExtra[cp] = icv
|
||||
} else {
|
||||
firstExtra[cp] = ""
|
||||
}
|
||||
}
|
||||
body := buildReqBody(iface, 1, pageSize, lastSyncTime, firstExtra)
|
||||
resp, err := api.Request(ctx, method, iface.Url, body, inQuery)
|
||||
@@ -151,6 +176,8 @@ func syncSingleAPI(ctx context.Context, api *ApiClient, platform *PlatformConfig
|
||||
return nil, err
|
||||
}
|
||||
|
||||
injectRowFields(rows, body, iface.RequestConfig)
|
||||
|
||||
result := &SyncResult{TableName: td.TableName, TotalPages: totalPages}
|
||||
inserted, _ := savePage(ctx, td, rows)
|
||||
result.InsertedRows += inserted
|
||||
@@ -185,6 +212,7 @@ func syncSingleAPI(ctx context.Context, api *ApiClient, platform *PlatformConfig
|
||||
}
|
||||
nextCursor = nc
|
||||
|
||||
injectRowFields(rows, body, iface.RequestConfig)
|
||||
inserted, _ = savePage(ctx, td, rows)
|
||||
result.InsertedRows += inserted
|
||||
result.TotalRows += len(rows)
|
||||
@@ -194,22 +222,72 @@ func syncSingleAPI(ctx context.Context, api *ApiClient, platform *PlatformConfig
|
||||
result.TotalPages++
|
||||
time.Sleep(100 * time.Millisecond)
|
||||
}
|
||||
} else if iface.ResponseConfig != nil {
|
||||
// hasMore 分页（如钉钉 offset/size + hasMore）
|
||||
if hf, _ := iface.ResponseConfig["has_more_field"].(string); hf != "" {
|
||||
for page := 2; hasMoreCheck(resp.Body, hf); page++ {
|
||||
body := buildReqBody(iface, page, pageSize, lastSyncTime, nil)
|
||||
resp2, e2 := api.Request(ctx, method, iface.Url, body, inQuery)
|
||||
if e2 != nil {
|
||||
logrus.Errorf("第 %d 页请求失败: %v", page, e2)
|
||||
break
|
||||
}
|
||||
rows2, _, mt2, _, pe2 := parseRespExt(resp2.Body, iface.ResponseConfig)
|
||||
if pe2 != nil {
|
||||
logrus.Errorf("第 %d 页解析失败: %v", page, pe2)
|
||||
break
|
||||
}
|
||||
injectRowFields(rows2, body, iface.RequestConfig)
|
||||
inserted2, _ := savePage(ctx, td, rows2)
|
||||
result.InsertedRows += inserted2
|
||||
result.TotalRows += len(rows2)
|
||||
if mt2 > maxTime {
|
||||
maxTime = mt2
|
||||
}
|
||||
resp = resp2
|
||||
time.Sleep(100 * time.Millisecond)
|
||||
}
|
||||
} else {
|
||||
// 普通分页
|
||||
for page := 2; page <= totalPages; page++ {
|
||||
body := buildReqBody(iface, page, pageSize, lastSyncTime, nil)
|
||||
resp, err = api.Request(ctx, method, iface.Url, body, inQuery)
|
||||
if err != nil {
|
||||
logrus.Errorf("第 %d 页请求失败: %v", page, err)
|
||||
recordFailure(ctx, platform.PlatformCode, iface.Code, taskType, fmt.Sprintf("第 %d 页请求失败: %v", page, err))
|
||||
continue
|
||||
}
|
||||
rows, _, mt, _, pe := parseRespExt(resp.Body, iface.ResponseConfig)
|
||||
if pe != nil {
|
||||
logrus.Errorf("第 %d 页解析失败: %v", page, pe)
|
||||
recordFailure(ctx, platform.PlatformCode, iface.Code, taskType, fmt.Sprintf("第 %d 页解析失败: %v", page, pe))
|
||||
continue
|
||||
}
|
||||
injectRowFields(rows, body, iface.RequestConfig)
|
||||
inserted, _ = savePage(ctx, td, rows)
|
||||
result.InsertedRows += inserted
|
||||
result.TotalRows += len(rows)
|
||||
if mt > maxTime {
|
||||
maxTime = mt
|
||||
}
|
||||
time.Sleep(100 * time.Millisecond)
|
||||
}
|
||||
}
|
||||
} else {
|
||||
// 普通分页
|
||||
// 普通分页（无 response_config）
|
||||
for page := 2; page <= totalPages; page++ {
|
||||
body := buildReqBody(iface, page, pageSize, lastSyncTime, nil)
|
||||
resp, err := api.Request(ctx, method, iface.Url, body, inQuery)
|
||||
resp, err = api.Request(ctx, method, iface.Url, body, inQuery)
|
||||
if err != nil {
|
||||
logrus.Errorf("第 %d 页请求失败: %v", page, err)
|
||||
recordFailure(ctx, platform.PlatformCode, iface.Code, taskType, fmt.Sprintf("第 %d 页请求失败: %v", page, err))
|
||||
continue
|
||||
}
|
||||
rows, _, mt, _, pe := parseRespExt(resp.Body, iface.ResponseConfig)
|
||||
if pe != nil {
|
||||
logrus.Errorf("第 %d 页解析失败: %v", page, pe)
|
||||
recordFailure(ctx, platform.PlatformCode, iface.Code, taskType, fmt.Sprintf("第 %d 页解析失败: %v", page, pe))
|
||||
continue
|
||||
}
|
||||
injectRowFields(rows, body, iface.RequestConfig)
|
||||
inserted, _ = savePage(ctx, td, rows)
|
||||
result.InsertedRows += inserted
|
||||
result.TotalRows += len(rows)
|
||||
@@ -238,6 +316,33 @@ func isCursorPagination(iface *entity.ApiInterface) bool {
|
||||
return cp
|
||||
}
|
||||
|
||||
// hasMoreCheck 从响应体中提取 has_more_field 的值
|
||||
func hasMoreCheck(raw []byte, hasMorePath string) bool {
|
||||
var respMap map[string]interface{}
|
||||
if err := json.Unmarshal(raw, &respMap); err != nil {
|
||||
return false
|
||||
}
|
||||
parts := strings.Split(hasMorePath, ".")
|
||||
cc := respMap
|
||||
for i, p := range parts {
|
||||
if i == len(parts)-1 {
|
||||
if b, ok := cc[p].(bool); ok {
|
||||
return b
|
||||
}
|
||||
if s, ok := cc[p].(string); ok {
|
||||
return s == "true"
|
||||
}
|
||||
return false
|
||||
}
|
||||
if m, ok := cc[p].(map[string]interface{}); ok {
|
||||
cc = m
|
||||
} else {
|
||||
return false
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// collectPrefetchEntities 从 rows 中收集实体和行数据
|
||||
func collectPrefetchEntities(rows []map[string]interface{}, prefetch *PrefetchConfig, allEntities *[]interface{}, allRows *[]map[string]interface{}) {
|
||||
for _, item := range rows {
|
||||
@@ -266,6 +371,12 @@ func syncWithPrefetch(ctx context.Context, api *ApiClient, platform *PlatformCon
|
||||
// ====== 1. 预取阶段：分页拉取全部实体列表 ======
|
||||
prefetchIface := findInterfaceByURL(allIfaces, prefetch.URL)
|
||||
|
||||
// 判断预取来源是否有递归配置（如钉钉部门树）
|
||||
var prefetchRecursiveCfg *RecursiveConfig
|
||||
if prefetchIface != nil {
|
||||
prefetchRecursiveCfg = parseRecursiveConfig(prefetchIface.RequestConfig)
|
||||
}
|
||||
|
||||
// 判断预取来源是否游标分页，以及分页参数名
|
||||
prefetchIsCursor := false
|
||||
prefetchPageParam := "page"
|
||||
@@ -303,69 +414,127 @@ func syncWithPrefetch(ctx context.Context, api *ApiClient, platform *PlatformCon
|
||||
allEntities := make([]interface{}, 0)
|
||||
allRows := make([]map[string]interface{}, 0)
|
||||
|
||||
// 第一页（游标分页首次 cursor=""）
|
||||
firstExtra := make(map[string]interface{})
|
||||
if prefetchIsCursor {
|
||||
firstExtra[prefetchPageParam] = ""
|
||||
}
|
||||
prefetchReqIface := prefetchIface
|
||||
if prefetchReqIface == nil {
|
||||
prefetchReqIface = iface
|
||||
}
|
||||
body := buildReqBody(prefetchReqIface, 1, prefetchPageSize, lastSyncTime, firstExtra)
|
||||
resp, err := api.Request(ctx, prefetchMethod, prefetch.URL, body, prefetchInQuery)
|
||||
if err != nil {
|
||||
recordFailure(ctx, platform.PlatformCode, iface.Code, taskType, fmt.Sprintf("预取第一页请求失败: %v", err))
|
||||
return nil, fmt.Errorf("预取第一页失败: %w", err)
|
||||
}
|
||||
|
||||
rows, prefetchTotalPages, _, nextCursor, err := parseRespExt(resp.Body, prefetchRespCfg)
|
||||
if err != nil {
|
||||
recordFailure(ctx, platform.PlatformCode, iface.Code, taskType, fmt.Sprintf("解析预取响应失败: %v", err))
|
||||
return nil, fmt.Errorf("解析预取响应失败: %w", err)
|
||||
}
|
||||
collectPrefetchEntities(rows, prefetch, &allEntities, &allRows)
|
||||
if prefetchIface != nil && prefetchRecursiveCfg != nil {
|
||||
// ----- 递归遍历预取（如钉钉部门树）-----
|
||||
maxDepth := 20
|
||||
if md, ok := prefetchIface.RequestConfig["max_recursive_depth"].(float64); ok {
|
||||
maxDepth = int(md)
|
||||
}
|
||||
processedKeys := make(map[string]bool)
|
||||
type rItem struct {
|
||||
depth int
|
||||
keyVal interface{}
|
||||
}
|
||||
queue := []rItem{{depth: 0, keyVal: nil}}
|
||||
|
||||
// 分页循环
|
||||
if prefetchIsCursor {
|
||||
// 游标分页
|
||||
for nextCursor != "" && nextCursor != "nomore" {
|
||||
body := buildReqBody(prefetchReqIface, 1, prefetchPageSize, lastSyncTime, map[string]interface{}{
|
||||
prefetchPageParam: nextCursor,
|
||||
})
|
||||
resp, err := api.Request(ctx, prefetchMethod, prefetch.URL, body, prefetchInQuery)
|
||||
for len(queue) > 0 {
|
||||
item := queue[0]
|
||||
queue = queue[1:]
|
||||
if item.depth > maxDepth {
|
||||
continue
|
||||
}
|
||||
if item.keyVal != nil {
|
||||
keyStr := fmt.Sprintf("%v", item.keyVal)
|
||||
if processedKeys[keyStr] {
|
||||
continue
|
||||
}
|
||||
processedKeys[keyStr] = true
|
||||
}
|
||||
extra := make(map[string]interface{})
|
||||
if item.keyVal != nil {
|
||||
extra[prefetchRecursiveCfg.TargetParam] = item.keyVal
|
||||
}
|
||||
body := buildReqBody(prefetchReqIface, 1, prefetchPageSize, 0, extra)
|
||||
r2, err := api.Request(ctx, prefetchMethod, prefetch.URL, body, prefetchInQuery)
|
||||
if err != nil {
|
||||
logrus.Errorf("预取游标 %s 请求失败: %v", nextCursor, err)
|
||||
break
|
||||
logrus.Errorf("预取递归 [depth=%d] 请求失败: %v", item.depth, err)
|
||||
continue
|
||||
}
|
||||
rows, _, _, nc, pe := parseRespExt(resp.Body, prefetchRespCfg)
|
||||
itemRows, _, _, _, pe := parseRespExt(r2.Body, prefetchRespCfg)
|
||||
if pe != nil {
|
||||
logrus.Errorf("预取游标 %s 解析失败: %v", nextCursor, pe)
|
||||
break
|
||||
logrus.Errorf("预取递归 [depth=%d] 解析失败: %v", item.depth, pe)
|
||||
continue
|
||||
}
|
||||
if len(rows) == 0 {
|
||||
break
|
||||
for _, row := range itemRows {
|
||||
allRows = append(allRows, row)
|
||||
if prefetch.ValueField == "" {
|
||||
allEntities = append(allEntities, row)
|
||||
} else if v, ok := row[prefetch.ValueField]; ok {
|
||||
if f, ok := v.(float64); ok {
|
||||
allEntities = append(allEntities, int64(f))
|
||||
} else {
|
||||
allEntities = append(allEntities, v)
|
||||
}
|
||||
}
|
||||
if v, ok := row[prefetchRecursiveCfg.KeyField]; ok {
|
||||
queue = append(queue, rItem{depth: item.depth + 1, keyVal: v})
|
||||
}
|
||||
}
|
||||
nextCursor = nc
|
||||
collectPrefetchEntities(rows, prefetch, &allEntities, &allRows)
|
||||
time.Sleep(100 * time.Millisecond)
|
||||
}
|
||||
} else {
|
||||
// 普通分页
|
||||
for page := 2; page <= prefetchTotalPages; page++ {
|
||||
body := buildReqBody(prefetchReqIface, page, prefetchPageSize, lastSyncTime, nil)
|
||||
resp, err := api.Request(ctx, prefetchMethod, prefetch.URL, body, prefetchInQuery)
|
||||
if err != nil {
|
||||
logrus.Errorf("预取第 %d 页请求失败: %v", page, err)
|
||||
continue
|
||||
// ----- 常规分页预取 -----
|
||||
firstExtra := make(map[string]interface{})
|
||||
if prefetchIsCursor {
|
||||
firstExtra[prefetchPageParam] = ""
|
||||
}
|
||||
body := buildReqBody(prefetchReqIface, 1, prefetchPageSize, lastSyncTime, firstExtra)
|
||||
resp, err := api.Request(ctx, prefetchMethod, prefetch.URL, body, prefetchInQuery)
|
||||
if err != nil {
|
||||
recordFailure(ctx, platform.PlatformCode, iface.Code, taskType, fmt.Sprintf("预取第一页请求失败: %v", err))
|
||||
return nil, fmt.Errorf("预取第一页失败: %w", err)
|
||||
}
|
||||
|
||||
rows, prefetchTotalPages, _, nextCursor, err := parseRespExt(resp.Body, prefetchRespCfg)
|
||||
if err != nil {
|
||||
recordFailure(ctx, platform.PlatformCode, iface.Code, taskType, fmt.Sprintf("解析预取响应失败: %v", err))
|
||||
return nil, fmt.Errorf("解析预取响应失败: %w", err)
|
||||
}
|
||||
collectPrefetchEntities(rows, prefetch, &allEntities, &allRows)
|
||||
|
||||
if prefetchIsCursor {
|
||||
for nextCursor != "" && nextCursor != "nomore" {
|
||||
body := buildReqBody(prefetchReqIface, 1, prefetchPageSize, lastSyncTime, map[string]interface{}{
|
||||
prefetchPageParam: nextCursor,
|
||||
})
|
||||
resp, err := api.Request(ctx, prefetchMethod, prefetch.URL, body, prefetchInQuery)
|
||||
if err != nil {
|
||||
logrus.Errorf("预取游标 %s 请求失败: %v", nextCursor, err)
|
||||
break
|
||||
}
|
||||
rows, _, _, nc, pe := parseRespExt(resp.Body, prefetchRespCfg)
|
||||
if pe != nil {
|
||||
logrus.Errorf("预取游标 %s 解析失败: %v", nextCursor, pe)
|
||||
break
|
||||
}
|
||||
if len(rows) == 0 {
|
||||
break
|
||||
}
|
||||
nextCursor = nc
|
||||
collectPrefetchEntities(rows, prefetch, &allEntities, &allRows)
|
||||
time.Sleep(100 * time.Millisecond)
|
||||
}
|
||||
rows, _, _, _, pe := parseRespExt(resp.Body, prefetchRespCfg)
|
||||
if pe != nil {
|
||||
logrus.Errorf("预取第 %d 页解析失败: %v", page, pe)
|
||||
continue
|
||||
} else {
|
||||
for page := 2; page <= prefetchTotalPages; page++ {
|
||||
body := buildReqBody(prefetchReqIface, page, prefetchPageSize, lastSyncTime, nil)
|
||||
resp, err := api.Request(ctx, prefetchMethod, prefetch.URL, body, prefetchInQuery)
|
||||
if err != nil {
|
||||
logrus.Errorf("预取第 %d 页请求失败: %v", page, err)
|
||||
continue
|
||||
}
|
||||
rows, _, _, _, pe := parseRespExt(resp.Body, prefetchRespCfg)
|
||||
if pe != nil {
|
||||
logrus.Errorf("预取第 %d 页解析失败: %v", page, pe)
|
||||
continue
|
||||
}
|
||||
collectPrefetchEntities(rows, prefetch, &allEntities, &allRows)
|
||||
time.Sleep(100 * time.Millisecond)
|
||||
}
|
||||
collectPrefetchEntities(rows, prefetch, &allEntities, &allRows)
|
||||
time.Sleep(100 * time.Millisecond)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -375,7 +544,7 @@ func syncWithPrefetch(ctx context.Context, api *ApiClient, platform *PlatformCon
|
||||
}
|
||||
logrus.Infof("预取到 %d 个实体", len(allEntities))
|
||||
|
||||
// 2. 将预取的数据也存入库（如账户列表存入 tencent_account_relation）
|
||||
// 将预取的数据也存入库（如账户列表存入 tencent_account_relation）
|
||||
if prefetchIface != nil && prefetchIface.TableDefinition != nil {
|
||||
prefetchTd, err := ParseTableDefinition(prefetchIface.TableDefinition)
|
||||
if err == nil {
|
||||
@@ -386,11 +555,13 @@ func syncWithPrefetch(ctx context.Context, api *ApiClient, platform *PlatformCon
|
||||
}
|
||||
}
|
||||
|
||||
// 2. 并发处理每个实体的数据
|
||||
// 并发处理每个实体的数据
|
||||
result := &SyncResult{TableName: td.TableName}
|
||||
pageSize := GetSyncPageSize(ctx)
|
||||
if ps, ok := iface.RequestConfig["page_size"].(float64); ok {
|
||||
pageSize = int(ps)
|
||||
} else if ps, ok := iface.RequestConfig["pageSize"].(float64); ok {
|
||||
pageSize = int(ps)
|
||||
}
|
||||
|
||||
dataMethod := string(iface.Method)
|
||||
@@ -411,52 +582,118 @@ func syncWithPrefetch(ctx context.Context, api *ApiClient, platform *PlatformCon
|
||||
defer func() { <-sem }()
|
||||
|
||||
logrus.Infof(" 处理实体 [%d/%d]: %v", idx+1, len(allEntities), val)
|
||||
|
||||
page := 1
|
||||
totalPages := 1
|
||||
entityMaxTime := int64(0)
|
||||
|
||||
for page <= totalPages {
|
||||
body := buildReqBody(iface, page, pageSize, lastSyncTime, map[string]interface{}{
|
||||
if isCursorPagination(iface) {
|
||||
// ----- 游标分页（如钉钉 user_list）-----
|
||||
cp := "cursor"
|
||||
if p, ok := iface.RequestConfig["page_param"].(string); ok && p != "" {
|
||||
cp = p
|
||||
}
|
||||
firstExtra := map[string]interface{}{
|
||||
prefetch.TargetParam: val,
|
||||
})
|
||||
|
||||
}
|
||||
if icv, ok := iface.RequestConfig["initial_cursor"]; ok {
|
||||
firstExtra[cp] = icv
|
||||
} else {
|
||||
firstExtra[cp] = ""
|
||||
}
|
||||
body := buildReqBody(iface, 1, pageSize, lastSyncTime, firstExtra)
|
||||
resp, err := api.Request(ctx, dataMethod, iface.Url, body, inQuery)
|
||||
if err != nil {
|
||||
logrus.Errorf(" 实体 %v 第 %d 页失败: %v", val, page, err)
|
||||
page++
|
||||
time.Sleep(200 * time.Millisecond)
|
||||
continue
|
||||
logrus.Errorf(" 实体 %v 首次请求失败: %v", val, err)
|
||||
return
|
||||
}
|
||||
|
||||
rows, tp, mt, parseErr := parseResp(resp.Body, iface.ResponseConfig)
|
||||
if parseErr != nil {
|
||||
logrus.Errorf(" 解析响应失败: %v", parseErr)
|
||||
page++
|
||||
continue
|
||||
rows, _, mt, nc, pe := parseRespExt(resp.Body, iface.ResponseConfig)
|
||||
if pe != nil {
|
||||
logrus.Errorf(" 实体 %v 解析首页失败: %v", val, pe)
|
||||
return
|
||||
}
|
||||
|
||||
if page == 1 {
|
||||
totalPages = tp
|
||||
}
|
||||
|
||||
for i := range rows {
|
||||
rows[i][prefetch.TargetParam] = val
|
||||
}
|
||||
|
||||
injectRowFields(rows, body, iface.RequestConfig)
|
||||
inserted, _ := savePage(ctx, td, rows)
|
||||
|
||||
mu.Lock()
|
||||
result.InsertedRows += inserted
|
||||
result.TotalRows += len(rows)
|
||||
mu.Unlock()
|
||||
|
||||
if mt > entityMaxTime {
|
||||
entityMaxTime = mt
|
||||
}
|
||||
|
||||
page++
|
||||
time.Sleep(100 * time.Millisecond)
|
||||
nextCursor := nc
|
||||
for nextCursor != "" && nextCursor != "nomore" {
|
||||
body := buildReqBody(iface, 1, pageSize, lastSyncTime, map[string]interface{}{
|
||||
cp: nextCursor,
|
||||
prefetch.TargetParam: val,
|
||||
})
|
||||
resp, err := api.Request(ctx, dataMethod, iface.Url, body, inQuery)
|
||||
if err != nil {
|
||||
logrus.Errorf(" 实体 %v 游标 %s 失败: %v", val, nextCursor, err)
|
||||
break
|
||||
}
|
||||
rows, _, mt, nc, pe := parseRespExt(resp.Body, iface.ResponseConfig)
|
||||
if pe != nil {
|
||||
logrus.Errorf(" 实体 %v 游标 %s 解析失败: %v", val, nextCursor, pe)
|
||||
break
|
||||
}
|
||||
if len(rows) == 0 {
|
||||
break
|
||||
}
|
||||
nextCursor = nc
|
||||
for i := range rows {
|
||||
rows[i][prefetch.TargetParam] = val
|
||||
}
|
||||
injectRowFields(rows, body, iface.RequestConfig)
|
||||
inserted, _ := savePage(ctx, td, rows)
|
||||
mu.Lock()
|
||||
result.InsertedRows += inserted
|
||||
result.TotalRows += len(rows)
|
||||
mu.Unlock()
|
||||
if mt > entityMaxTime {
|
||||
entityMaxTime = mt
|
||||
}
|
||||
time.Sleep(100 * time.Millisecond)
|
||||
}
|
||||
} else {
|
||||
// ----- 普通分页 -----
|
||||
page := 1
|
||||
totalPages := 1
|
||||
for page <= totalPages {
|
||||
body := buildReqBody(iface, page, pageSize, lastSyncTime, map[string]interface{}{
|
||||
prefetch.TargetParam: val,
|
||||
})
|
||||
resp, err := api.Request(ctx, dataMethod, iface.Url, body, inQuery)
|
||||
if err != nil {
|
||||
logrus.Errorf(" 实体 %v 第 %d 页失败: %v", val, page, err)
|
||||
page++
|
||||
time.Sleep(200 * time.Millisecond)
|
||||
continue
|
||||
}
|
||||
rows, tp, mt, parseErr := parseResp(resp.Body, iface.ResponseConfig)
|
||||
if parseErr != nil {
|
||||
logrus.Errorf(" 解析响应失败: %v", parseErr)
|
||||
page++
|
||||
continue
|
||||
}
|
||||
if page == 1 {
|
||||
totalPages = tp
|
||||
}
|
||||
for i := range rows {
|
||||
rows[i][prefetch.TargetParam] = val
|
||||
}
|
||||
injectRowFields(rows, body, iface.RequestConfig)
|
||||
inserted, _ := savePage(ctx, td, rows)
|
||||
mu.Lock()
|
||||
result.InsertedRows += inserted
|
||||
result.TotalRows += len(rows)
|
||||
mu.Unlock()
|
||||
if mt > entityMaxTime {
|
||||
entityMaxTime = mt
|
||||
}
|
||||
page++
|
||||
time.Sleep(100 * time.Millisecond)
|
||||
}
|
||||
}
|
||||
|
||||
if entityMaxTime > 0 {
|
||||
@@ -481,6 +718,90 @@ func syncWithPrefetch(ctx context.Context, api *ApiClient, platform *PlatformCon
|
||||
return result, nil
|
||||
}
|
||||
|
||||
// syncRecursive 递归遍历同步（如钉钉部门树：先查根级 → 对每个子部门递归查下级）
|
||||
func syncRecursive(ctx context.Context, api *ApiClient, platform *PlatformConfig, iface *entity.ApiInterface, td *TableDefinition, recursive *RecursiveConfig, start time.Time) (*SyncResult, error) {
|
||||
maxDepth := 20
|
||||
if md, ok := iface.RequestConfig["max_recursive_depth"].(float64); ok {
|
||||
maxDepth = int(md)
|
||||
}
|
||||
|
||||
inQuery := paramsInQuery(iface)
|
||||
method := string(iface.Method)
|
||||
|
||||
allRows := make([]map[string]interface{}, 0)
|
||||
processedKeys := make(map[string]bool)
|
||||
|
||||
type queueItem struct {
|
||||
depth int
|
||||
keyVal interface{} // nil 表示根级
|
||||
}
|
||||
queue := []queueItem{{depth: 0, keyVal: nil}}
|
||||
|
||||
for len(queue) > 0 {
|
||||
item := queue[0]
|
||||
queue = queue[1:]
|
||||
|
||||
if item.depth > maxDepth {
|
||||
logrus.Warnf("递归已达最大深度 %d，终止该分支", maxDepth)
|
||||
continue
|
||||
}
|
||||
|
||||
// 防重复处理
|
||||
if item.keyVal != nil {
|
||||
keyStr := fmt.Sprintf("%v", item.keyVal)
|
||||
if processedKeys[keyStr] {
|
||||
continue
|
||||
}
|
||||
processedKeys[keyStr] = true
|
||||
}
|
||||
|
||||
extraParams := make(map[string]interface{})
|
||||
if item.keyVal != nil {
|
||||
extraParams[recursive.TargetParam] = item.keyVal
|
||||
}
|
||||
|
||||
body := buildReqBody(iface, 1, 100, 0, extraParams)
|
||||
resp, err := api.Request(ctx, method, iface.Url, body, inQuery)
|
||||
if err != nil {
|
||||
logrus.Errorf("递归 [depth=%d] 请求失败: %v", item.depth, err)
|
||||
recordFailure(ctx, platform.PlatformCode, iface.Code, "full", fmt.Sprintf("递归深度 %d 请求失败: %v", item.depth, err))
|
||||
continue
|
||||
}
|
||||
|
||||
rows, _, _, _, err := parseRespExt(resp.Body, iface.ResponseConfig)
|
||||
if err != nil {
|
||||
logrus.Errorf("递归 [depth=%d] 解析失败: %v", item.depth, err)
|
||||
continue
|
||||
}
|
||||
|
||||
for _, row := range rows {
|
||||
allRows = append(allRows, row)
|
||||
if v, ok := row[recursive.KeyField]; ok {
|
||||
queue = append(queue, queueItem{depth: item.depth + 1, keyVal: v})
|
||||
}
|
||||
}
|
||||
|
||||
time.Sleep(100 * time.Millisecond)
|
||||
}
|
||||
|
||||
if len(allRows) == 0 {
|
||||
logrus.Warn("递归结果为空，跳过入库")
|
||||
return &SyncResult{TableName: td.TableName, Duration: fmt.Sprintf("%.1fs", time.Since(start).Seconds())}, nil
|
||||
}
|
||||
|
||||
inserted, _ := savePage(ctx, td, allRows)
|
||||
updateSyncTime(ctx, platform.PlatformCode, iface.Code, time.Now().Unix())
|
||||
|
||||
result := &SyncResult{
|
||||
TableName: td.TableName,
|
||||
TotalRows: len(allRows),
|
||||
InsertedRows: inserted,
|
||||
Duration: fmt.Sprintf("%.1fs", time.Since(start).Seconds()),
|
||||
}
|
||||
logrus.Infof("递归同步完成 - 表:%s, %d条, 写入%d条, 耗时%s", td.TableName, result.TotalRows, result.InsertedRows, result.Duration)
|
||||
return result, nil
|
||||
}
|
||||
|
||||
// getTotalPages 从响应中提取总页数
|
||||
func getTotalPages(raw []byte) int {
|
||||
rows, tp, _, _, err := parseRespExt(raw, nil)
|
||||
@@ -498,6 +819,12 @@ func toFloat64(v interface{}) (float64, bool) {
|
||||
return float64(val), true
|
||||
case int64:
|
||||
return float64(val), true
|
||||
case string:
|
||||
// 支持字符串类型的成功值（如钉钉智能薪酬返回 code: "200"）
|
||||
if f, err := strconv.ParseFloat(val, 64); err == nil {
|
||||
return f, true
|
||||
}
|
||||
return 0, false
|
||||
default:
|
||||
return 0, false
|
||||
}
|
||||
@@ -524,7 +851,10 @@ func buildPrefetchParams(iface *entity.ApiInterface) map[string]interface{} {
|
||||
k == "page_size_param" || k == "time_field" || k == "parameters_location" ||
|
||||
k == "filtering" || k == "group_by" || k == "date_range" ||
|
||||
k == "body_wrapper_field" || k == "exclude_from_wrapper" ||
|
||||
k == "cursor_pagination" || k == "time_field_mode" {
|
||||
k == "cursor_pagination" || k == "time_field_mode" ||
|
||||
k == "recursive" || k == "max_recursive_depth" ||
|
||||
k == "initial_cursor" || k == "pagination_mode" ||
|
||||
k == "full_sync_start_time" || k == "row_inject" {
|
||||
continue
|
||||
}
|
||||
if k == pageParam || k == psParam {
|
||||
@@ -567,6 +897,33 @@ func parsePrefetchConfig(requestConfig map[string]interface{}) *PrefetchConfig {
|
||||
return pc
|
||||
}
|
||||
|
||||
// parseRecursiveConfig 解析递归遍历配置
|
||||
func parseRecursiveConfig(requestConfig map[string]interface{}) *RecursiveConfig {
|
||||
if requestConfig == nil {
|
||||
return nil
|
||||
}
|
||||
raw, ok := requestConfig["recursive"]
|
||||
if !ok || raw == nil {
|
||||
return nil
|
||||
}
|
||||
m, ok := raw.(map[string]interface{})
|
||||
if !ok {
|
||||
return nil
|
||||
}
|
||||
rc := &RecursiveConfig{}
|
||||
if kf, _ := m["key_field"].(string); kf != "" {
|
||||
rc.KeyField = kf
|
||||
} else {
|
||||
return nil
|
||||
}
|
||||
if tp, _ := m["target_param"].(string); tp != "" {
|
||||
rc.TargetParam = tp
|
||||
} else {
|
||||
return nil
|
||||
}
|
||||
return rc
|
||||
}
|
||||
|
||||
// extractValues 从 JSON 响应中提取值列表
|
||||
func extractValues(raw []byte, path, valueField string) ([]interface{}, error) {
|
||||
var resp map[string]interface{}
|
||||
@@ -612,7 +969,10 @@ func buildReqBody(iface *entity.ApiInterface, page, pageSize int, lastSyncTime i
|
||||
k == "page_param" || k == "page_size_param" || k == "parameters_location" ||
|
||||
k == "cursor_pagination" || k == "time_field_mode" ||
|
||||
k == "body_wrapper_field" || k == "exclude_from_wrapper" ||
|
||||
k == "top_level_params" {
|
||||
k == "top_level_params" || k == "recursive" ||
|
||||
k == "max_recursive_depth" || k == "initial_cursor" ||
|
||||
k == "pagination_mode" || k == "full_sync_start_time" ||
|
||||
k == "row_inject" {
|
||||
continue
|
||||
}
|
||||
body[k] = v
|
||||
@@ -628,39 +988,68 @@ func buildReqBody(iface *entity.ApiInterface, page, pageSize int, lastSyncTime i
|
||||
psParam = p
|
||||
}
|
||||
}
|
||||
body[pageParam] = page
|
||||
// 偏移量分页（如钉钉 offset）：offset = (page-1) * pageSize
|
||||
paginationMode := ""
|
||||
if iface.RequestConfig != nil {
|
||||
if pm, ok := iface.RequestConfig["pagination_mode"].(string); ok {
|
||||
paginationMode = pm
|
||||
}
|
||||
}
|
||||
if paginationMode == "offset" {
|
||||
body[pageParam] = (page - 1) * pageSize
|
||||
} else {
|
||||
body[pageParam] = page
|
||||
}
|
||||
body[psParam] = pageSize
|
||||
|
||||
// 时间过滤处理：支持两种模式
|
||||
// 1. "filtering" 模式（默认）：生成 filtering=[{"field":"...","operator":"GREATER_EQUALS","values":["..."]}]（腾讯）
|
||||
// 2. "range" 模式：生成 beginTime/endTime + queryType（快手）
|
||||
if tf, ok := iface.RequestConfig["time_field"].(string); ok && tf != "" {
|
||||
timeMode := "filtering"
|
||||
if tm, ok := iface.RequestConfig["time_field_mode"].(string); ok && tm != "" {
|
||||
timeMode = tm
|
||||
}
|
||||
if iface.RequestConfig != nil {
|
||||
if tf, ok := iface.RequestConfig["time_field"].(string); ok && tf != "" {
|
||||
timeMode := "filtering"
|
||||
if tm, ok := iface.RequestConfig["time_field_mode"].(string); ok && tm != "" {
|
||||
timeMode = tm
|
||||
}
|
||||
|
||||
if timeMode == "range" {
|
||||
// 快手模式：beginTime/endTime（毫秒时间戳）
|
||||
timeMs := lastSyncTime
|
||||
if timeMs <= 0 {
|
||||
// 全量：默认90天前
|
||||
timeMs = time.Now().Add(-90 * 24 * time.Hour).UnixMilli()
|
||||
}
|
||||
body["queryType"] = 2
|
||||
body["beginTime"] = timeMs
|
||||
body["endTime"] = time.Now().UnixMilli()
|
||||
} else if lastSyncTime > 0 {
|
||||
// 腾讯 filtering 模式（仅增量时）
|
||||
timeFilter := map[string]interface{}{
|
||||
"field": tf,
|
||||
"operator": "GREATER_EQUALS",
|
||||
"values": []interface{}{fmt.Sprintf("%d", lastSyncTime)},
|
||||
}
|
||||
if existing, ok := body["filtering"].([]interface{}); ok {
|
||||
body["filtering"] = append(existing, timeFilter)
|
||||
} else {
|
||||
body["filtering"] = []interface{}{timeFilter}
|
||||
if timeMode == "range" {
|
||||
// 快手模式：beginTime/endTime（毫秒时间戳）
|
||||
timeMs := lastSyncTime
|
||||
if timeMs <= 0 {
|
||||
// 全量：优先使用配置的 full_sync_start_time，否则默认90天前
|
||||
if fst, ok := iface.RequestConfig["full_sync_start_time"].(float64); ok && fst > 0 {
|
||||
timeMs = int64(fst)
|
||||
} else {
|
||||
timeMs = time.Now().Add(-90 * 24 * time.Hour).UnixMilli()
|
||||
}
|
||||
}
|
||||
body["queryType"] = 2
|
||||
body["beginTime"] = timeMs
|
||||
body["endTime"] = time.Now().UnixMilli()
|
||||
} else if lastSyncTime > 0 {
|
||||
// 腾讯 filtering 模式（仅增量时）
|
||||
timeFilter := map[string]interface{}{
|
||||
"field": tf,
|
||||
"operator": "GREATER_EQUALS",
|
||||
"values": []interface{}{fmt.Sprintf("%d", lastSyncTime)},
|
||||
}
|
||||
if existing, ok := body["filtering"].([]interface{}); ok {
|
||||
body["filtering"] = append(existing, timeFilter)
|
||||
} else {
|
||||
body["filtering"] = []interface{}{timeFilter}
|
||||
}
|
||||
} else if fst, ok := iface.RequestConfig["full_sync_start_time"].(float64); ok && fst > 0 {
|
||||
// 全量 filtering 模式：指定了 full_sync_start_time，从该时间戳开始拉取
|
||||
timeFilter := map[string]interface{}{
|
||||
"field": tf,
|
||||
"operator": "GREATER_EQUALS",
|
||||
"values": []interface{}{fmt.Sprintf("%d", int64(fst))},
|
||||
}
|
||||
if existing, ok := body["filtering"].([]interface{}); ok {
|
||||
body["filtering"] = append(existing, timeFilter)
|
||||
} else {
|
||||
body["filtering"] = []interface{}{timeFilter}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -687,8 +1076,12 @@ func buildReqBody(iface *entity.ApiInterface, page, pageSize int, lastSyncTime i
|
||||
delete(body, k)
|
||||
}
|
||||
}
|
||||
b, _ := json.Marshal(wrapperObj)
|
||||
body[wf] = string(b)
|
||||
b, err := json.Marshal(wrapperObj)
|
||||
if err != nil {
|
||||
logrus.Errorf("JSON序列化 wrapper 失败: %v", err)
|
||||
} else {
|
||||
body[wf] = string(b)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -703,6 +1096,7 @@ func parseRespExt(raw []byte, rc map[string]interface{}) ([]map[string]interface
|
||||
}
|
||||
successField, successVal := "code", float64(0)
|
||||
msgField, listPath, cursorPath := "message", "data", ""
|
||||
hasMorePath := ""
|
||||
singleRecord := false
|
||||
if rc != nil {
|
||||
if sf, _ := rc["success_field"].(string); sf != "" {
|
||||
@@ -725,6 +1119,9 @@ func parseRespExt(raw []byte, rc map[string]interface{}) ([]map[string]interface
|
||||
if sr, _ := rc["single_record"].(bool); sr {
|
||||
singleRecord = true
|
||||
}
|
||||
if hm, _ := rc["has_more_field"].(string); hm != "" {
|
||||
hasMorePath = hm
|
||||
}
|
||||
}
|
||||
if v, ok := respMap[successField]; ok {
|
||||
actual, _ := toFloat64(v)
|
||||
@@ -820,6 +1217,23 @@ func parseRespExt(raw []byte, rc map[string]interface{}) ([]map[string]interface
|
||||
if i == len(cp)-1 {
|
||||
if s, ok := cc[p].(string); ok {
|
||||
nextCursor = s
|
||||
} else if f, ok := cc[p].(float64); ok {
|
||||
// 数字游标（如钉钉 next_cursor=10）
|
||||
nextCursor = fmt.Sprintf("%.0f", f)
|
||||
}
|
||||
} else if m, ok := cc[p].(map[string]interface{}); ok {
|
||||
cc = m
|
||||
}
|
||||
}
|
||||
}
|
||||
// has_more 字段支持：false 时标记游标结束
|
||||
if hasMorePath != "" {
|
||||
parts := strings.Split(hasMorePath, ".")
|
||||
cc := respMap
|
||||
for i, p := range parts {
|
||||
if i == len(parts)-1 {
|
||||
if b, ok := cc[p].(bool); ok && !b {
|
||||
nextCursor = "nomore"
|
||||
}
|
||||
} else if m, ok := cc[p].(map[string]interface{}); ok {
|
||||
cc = m
|
||||
@@ -950,3 +1364,32 @@ func findInterfaceByURL(ifaces []entity.ApiInterface, url string) *entity.ApiInt
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// injectRowFields 将请求参数中 row_inject 指定的字段注入到响应行中
|
||||
// 用于需要将请求参数（如 statisticsMonth）持久化到表中，但响应不含该字段的场景
|
||||
func injectRowFields(rows []map[string]interface{}, body map[string]interface{}, requestConfig map[string]interface{}) {
|
||||
if requestConfig == nil || body == nil {
|
||||
return
|
||||
}
|
||||
rawInject, ok := requestConfig["row_inject"]
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
injectList, ok := rawInject.([]interface{})
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
for _, item := range injectList {
|
||||
fieldName, ok := item.(string)
|
||||
if !ok {
|
||||
continue
|
||||
}
|
||||
val, exists := body[fieldName]
|
||||
if !exists {
|
||||
continue
|
||||
}
|
||||
for i := range rows {
|
||||
rows[i][fieldName] = val
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -41,3 +41,12 @@ func GetRetryCount(ctx context.Context) int {
|
||||
}
|
||||
return r
|
||||
}
|
||||
|
||||
// GetSyncTimeout 获取单次同步超时时间（分钟，默认120），全量超大表可适当调大
|
||||
func GetSyncTimeout(ctx context.Context) int {
|
||||
t := g.Cfg().MustGet(ctx, "sync.sync_timeout_minutes", 120).Int()
|
||||
if t < 1 {
|
||||
return 120
|
||||
}
|
||||
return t
|
||||
}
|
||||
|
||||
@@ -63,6 +63,15 @@ func (m *PlatformManager) GetPlatform(ctx context.Context, platformCode string)
|
||||
cfg.AppSecret = as
|
||||
}
|
||||
}
|
||||
case "APP_SIGNATURE":
|
||||
if platform.AuthConfig != nil {
|
||||
if aid, _ := platform.AuthConfig["app_id"].(string); aid != "" {
|
||||
cfg.AppKey = aid
|
||||
}
|
||||
if as, _ := platform.AuthConfig["app_secret"].(string); as != "" {
|
||||
cfg.AppSecret = as
|
||||
}
|
||||
}
|
||||
default:
|
||||
logrus.Warnf("平台 %s 认证类型 %s 未处理", platformCode, platform.AuthType)
|
||||
}
|
||||
|
||||
@@ -7,7 +7,7 @@ import (
|
||||
dao "dataengine/dao/dict"
|
||||
dto "dataengine/model/dto/dict"
|
||||
|
||||
"gitea.com/red-future/common/beans"
|
||||
"gitea.redpowerfuture.com/red-future/common/beans"
|
||||
"github.com/gogf/gf/v2/frame/g"
|
||||
"github.com/sirupsen/logrus"
|
||||
)
|
||||
@@ -71,7 +71,7 @@ func runAutoSync(ctx context.Context) {
|
||||
// InitAndStartAutoSync 在 main 中调用：初始化配置后启动自动同步和补偿
|
||||
func InitAndStartAutoSync(ctx context.Context) {
|
||||
// 读取配置中的同步开关
|
||||
enabled := g.Cfg().MustGet(ctx, "sync.auto_sync_enabled", true).Bool()
|
||||
enabled := g.Cfg().MustGet(ctx, "sync.auto_sync_enabled", false).Bool()
|
||||
if enabled {
|
||||
go StartAutoSync(ctx)
|
||||
} else {
|
||||
|
||||
@@ -5,7 +5,7 @@ import (
|
||||
"fmt"
|
||||
"strings"
|
||||
|
||||
"gitea.com/red-future/common/db/gfdb"
|
||||
"gitea.redpowerfuture.com/red-future/common/db/gfdb"
|
||||
"github.com/sirupsen/logrus"
|
||||
)
|
||||
|
||||
|
||||
@@ -0,0 +1,117 @@
|
||||
-- ============================================================
|
||||
-- 报表系统表 - 新增 tenant_id 迁移脚本
|
||||
-- 为已存在的5张报表系统表添加租户隔离字段
|
||||
-- 执行前请确认当前数据库中表是否存在
|
||||
-- ============================================================
|
||||
|
||||
-- ----------------------------------------------------------
|
||||
-- 1. report_business_config (业务配置表)
|
||||
-- ----------------------------------------------------------
|
||||
|
||||
-- 添加 tenant_id 列
|
||||
ALTER TABLE IF EXISTS report_business_config
|
||||
ADD COLUMN IF NOT EXISTS tenant_id BIGINT NOT NULL DEFAULT 0;
|
||||
|
||||
-- 重建唯一索引（加入 tenant_id）
|
||||
DROP INDEX IF EXISTS idx_business_code;
|
||||
CREATE UNIQUE INDEX IF NOT EXISTS idx_business_code
|
||||
ON report_business_config (tenant_id, business_code);
|
||||
|
||||
COMMENT ON COLUMN report_business_config.tenant_id IS '租户ID';
|
||||
|
||||
|
||||
-- ----------------------------------------------------------
|
||||
-- 2. report_report_config (报表配置表)
|
||||
-- ----------------------------------------------------------
|
||||
|
||||
-- 添加 tenant_id 列
|
||||
ALTER TABLE IF EXISTS report_report_config
|
||||
ADD COLUMN IF NOT EXISTS tenant_id BIGINT NOT NULL DEFAULT 0;
|
||||
|
||||
-- 重建唯一约束（加入 tenant_id）
|
||||
ALTER TABLE IF EXISTS report_report_config
|
||||
DROP CONSTRAINT IF EXISTS uk_business_report_code;
|
||||
ALTER TABLE IF EXISTS report_report_config
|
||||
ADD CONSTRAINT uk_business_report_code UNIQUE (tenant_id, business_code, report_code);
|
||||
|
||||
-- 重建索引
|
||||
DROP INDEX IF EXISTS idx_report_business_code;
|
||||
CREATE INDEX IF NOT EXISTS idx_report_business_code
|
||||
ON report_report_config (tenant_id, business_code);
|
||||
|
||||
COMMENT ON COLUMN report_report_config.tenant_id IS '租户ID';
|
||||
|
||||
|
||||
-- ----------------------------------------------------------
|
||||
-- 3. report_field_config (字段配置表)
|
||||
-- ----------------------------------------------------------
|
||||
|
||||
-- 添加 tenant_id 列
|
||||
ALTER TABLE IF EXISTS report_field_config
|
||||
ADD COLUMN IF NOT EXISTS tenant_id BIGINT NOT NULL DEFAULT 0;
|
||||
|
||||
-- 重建唯一约束（加入 tenant_id）
|
||||
ALTER TABLE IF EXISTS report_field_config
|
||||
DROP CONSTRAINT IF EXISTS uk_business_report_field_code;
|
||||
ALTER TABLE IF EXISTS report_field_config
|
||||
ADD CONSTRAINT uk_business_report_field_code UNIQUE (tenant_id, business_code, report_code, field_code);
|
||||
|
||||
-- 重建索引
|
||||
DROP INDEX IF EXISTS idx_field_business_report;
|
||||
CREATE INDEX IF NOT EXISTS idx_field_business_report
|
||||
ON report_field_config (tenant_id, business_code, report_code);
|
||||
DROP INDEX IF EXISTS idx_field_data_type;
|
||||
CREATE INDEX IF NOT EXISTS idx_field_data_type
|
||||
ON report_field_config (tenant_id, data_type);
|
||||
DROP INDEX IF EXISTS idx_field_field_role;
|
||||
CREATE INDEX IF NOT EXISTS idx_field_field_role
|
||||
ON report_field_config (tenant_id, field_role);
|
||||
|
||||
COMMENT ON COLUMN report_field_config.tenant_id IS '租户ID';
|
||||
|
||||
|
||||
-- ----------------------------------------------------------
|
||||
-- 4. report_extract_config (抽取配置表)
|
||||
-- ----------------------------------------------------------
|
||||
|
||||
-- 添加 tenant_id 列
|
||||
ALTER TABLE IF EXISTS report_extract_config
|
||||
ADD COLUMN IF NOT EXISTS tenant_id BIGINT NOT NULL DEFAULT 0;
|
||||
|
||||
-- 重建唯一约束（加入 tenant_id）
|
||||
ALTER TABLE IF EXISTS report_extract_config
|
||||
DROP CONSTRAINT IF EXISTS uk_business_report_extract_code;
|
||||
ALTER TABLE IF EXISTS report_extract_config
|
||||
ADD CONSTRAINT uk_business_report_extract_code UNIQUE (tenant_id, business_code, report_code, extract_code);
|
||||
|
||||
-- 重建索引
|
||||
DROP INDEX IF EXISTS idx_extract_business_report;
|
||||
CREATE INDEX IF NOT EXISTS idx_extract_business_report
|
||||
ON report_extract_config (tenant_id, business_code, report_code);
|
||||
|
||||
COMMENT ON COLUMN report_extract_config.tenant_id IS '租户ID';
|
||||
|
||||
|
||||
-- ----------------------------------------------------------
|
||||
-- 5. report_extract_log (抽取记录表)
|
||||
-- ----------------------------------------------------------
|
||||
|
||||
-- 添加 tenant_id 列
|
||||
ALTER TABLE IF EXISTS report_extract_log
|
||||
ADD COLUMN IF NOT EXISTS tenant_id BIGINT NOT NULL DEFAULT 0;
|
||||
|
||||
-- 重建唯一约束（加入 tenant_id）
|
||||
ALTER TABLE IF EXISTS report_extract_log
|
||||
DROP CONSTRAINT IF EXISTS uk_extract_keys;
|
||||
ALTER TABLE IF EXISTS report_extract_log
|
||||
ADD CONSTRAINT uk_extract_keys UNIQUE (tenant_id, business_code, report_code, extract_code, stat_date);
|
||||
|
||||
-- 重建索引
|
||||
DROP INDEX IF EXISTS idx_extract_log_business_report;
|
||||
CREATE INDEX IF NOT EXISTS idx_extract_log_business_report
|
||||
ON report_extract_log (tenant_id, business_code, report_code);
|
||||
DROP INDEX IF EXISTS idx_extract_log_stat_date;
|
||||
CREATE INDEX IF NOT EXISTS idx_extract_log_stat_date
|
||||
ON report_extract_log (tenant_id, stat_date);
|
||||
|
||||
COMMENT ON COLUMN report_extract_log.tenant_id IS '租户ID';
|
||||
@@ -0,0 +1,301 @@
|
||||
-- ============================================================
|
||||
-- 通用报表公共包 - 系统表 DDL
|
||||
-- 业务隔离、多业务支持、表/字段 100% 配置化
|
||||
-- ============================================================
|
||||
|
||||
-- ----------------------------------------------------------
|
||||
-- 1. 业务配置表 (business_config)
|
||||
-- ----------------------------------------------------------
|
||||
CREATE TABLE IF NOT EXISTS report_business_config (
|
||||
id BIGSERIAL PRIMARY KEY,
|
||||
tenant_id BIGINT NOT NULL DEFAULT 0,
|
||||
business_code VARCHAR(64) NOT NULL,
|
||||
business_name VARCHAR(128) NOT NULL,
|
||||
description TEXT DEFAULT '',
|
||||
status VARCHAR(16) NOT NULL DEFAULT 'ACTIVE',
|
||||
config JSONB DEFAULT '{}',
|
||||
creator VARCHAR(64) DEFAULT '',
|
||||
created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
|
||||
updater VARCHAR(64) DEFAULT '',
|
||||
updated_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
|
||||
deleted_at TIMESTAMP WITH TIME ZONE
|
||||
);
|
||||
|
||||
COMMENT ON TABLE report_business_config IS '报表业务配置表';
|
||||
COMMENT ON COLUMN report_business_config.tenant_id IS '租户ID';
|
||||
COMMENT ON COLUMN report_business_config.business_code IS '业务编码(唯一标识)';
|
||||
COMMENT ON COLUMN report_business_config.business_name IS '业务名称';
|
||||
COMMENT ON COLUMN report_business_config.description IS '业务描述';
|
||||
COMMENT ON COLUMN report_business_config.status IS '状态: ACTIVE启用/INACTIVE停用';
|
||||
COMMENT ON COLUMN report_business_config.config IS '业务级配置(JSON)，如默认数据源等';
|
||||
|
||||
CREATE UNIQUE INDEX IF NOT EXISTS idx_business_code ON report_business_config (tenant_id, business_code);
|
||||
|
||||
-- ----------------------------------------------------------
|
||||
-- 2. 报表配置表 (report_config)
|
||||
-- ----------------------------------------------------------
|
||||
CREATE TABLE IF NOT EXISTS report_report_config (
|
||||
id BIGSERIAL PRIMARY KEY,
|
||||
tenant_id BIGINT NOT NULL DEFAULT 0,
|
||||
business_code VARCHAR(64) NOT NULL,
|
||||
report_code VARCHAR(64) NOT NULL,
|
||||
report_name VARCHAR(128) NOT NULL,
|
||||
description TEXT DEFAULT '',
|
||||
status VARCHAR(16) NOT NULL DEFAULT 'ACTIVE',
|
||||
stat_table_name VARCHAR(128) NOT NULL,
|
||||
stat_table_comment VARCHAR(256) DEFAULT '',
|
||||
date_field VARCHAR(64) DEFAULT 'stat_date',
|
||||
primary_keys JSONB DEFAULT '["id"]'::jsonb,
|
||||
conflict_keys JSONB DEFAULT '["stat_date"]'::jsonb,
|
||||
config JSONB DEFAULT '{}',
|
||||
creator VARCHAR(64) DEFAULT '',
|
||||
created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
|
||||
updater VARCHAR(64) DEFAULT '',
|
||||
updated_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
|
||||
deleted_at TIMESTAMP WITH TIME ZONE,
|
||||
|
||||
CONSTRAINT uk_business_report_code UNIQUE (tenant_id, business_code, report_code)
|
||||
);
|
||||
|
||||
COMMENT ON TABLE report_report_config IS '报表配置表';
|
||||
COMMENT ON COLUMN report_report_config.tenant_id IS '租户ID';
|
||||
COMMENT ON COLUMN report_report_config.business_code IS '所属业务编码';
|
||||
COMMENT ON COLUMN report_report_config.report_code IS '报表编码(唯一标识)';
|
||||
COMMENT ON COLUMN report_report_config.report_name IS '报表名称';
|
||||
COMMENT ON COLUMN report_report_config.stat_table_name IS '统计宽表表名';
|
||||
COMMENT ON COLUMN report_report_config.stat_table_comment IS '统计宽表注释';
|
||||
COMMENT ON COLUMN report_report_config.date_field IS '日期字段名';
|
||||
COMMENT ON COLUMN report_report_config.primary_keys IS '主键字段列表(JSON数组)';
|
||||
COMMENT ON COLUMN report_report_config.conflict_keys IS '冲突键(用于ON CONFLICT upsert)';
|
||||
COMMENT ON COLUMN report_report_config.config IS '报表级配置(JSON)';
|
||||
|
||||
CREATE INDEX IF NOT EXISTS idx_report_business_code ON report_report_config (tenant_id, business_code);
|
||||
|
||||
-- ----------------------------------------------------------
|
||||
-- 3. 报表字段配置表 (report_field_config)
|
||||
-- 维度、指标、筛选字段 100% 配置化
|
||||
-- ----------------------------------------------------------
|
||||
CREATE TABLE IF NOT EXISTS report_field_config (
|
||||
id BIGSERIAL PRIMARY KEY,
|
||||
tenant_id BIGINT NOT NULL DEFAULT 0,
|
||||
business_code VARCHAR(64) NOT NULL,
|
||||
report_code VARCHAR(64) NOT NULL,
|
||||
field_code VARCHAR(64) NOT NULL,
|
||||
field_name VARCHAR(128) NOT NULL,
|
||||
field_type VARCHAR(32) NOT NULL,
|
||||
data_type VARCHAR(32) NOT NULL DEFAULT 'STRING',
|
||||
field_role VARCHAR(32) NOT NULL,
|
||||
is_aggregatable BOOLEAN DEFAULT FALSE,
|
||||
is_filterable BOOLEAN DEFAULT TRUE,
|
||||
is_queryable BOOLEAN DEFAULT TRUE,
|
||||
is_sortable BOOLEAN DEFAULT TRUE,
|
||||
default_aggregate VARCHAR(32) DEFAULT '',
|
||||
valid_aggregates JSONB DEFAULT '[]'::jsonb,
|
||||
filter_operators JSONB DEFAULT '["=","!=",">","<",">=","<=","IN","LIKE","BETWEEN"]'::jsonb,
|
||||
expression VARCHAR(512) DEFAULT '',
|
||||
expression_type VARCHAR(32) DEFAULT '',
|
||||
format_pattern VARCHAR(64) DEFAULT '',
|
||||
unit VARCHAR(32) DEFAULT '',
|
||||
dict_code VARCHAR(64) DEFAULT '',
|
||||
sort_order INT DEFAULT 0,
|
||||
group_name VARCHAR(64) DEFAULT '',
|
||||
status VARCHAR(16) NOT NULL DEFAULT 'ACTIVE',
|
||||
creator VARCHAR(64) DEFAULT '',
|
||||
created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
|
||||
updater VARCHAR(64) DEFAULT '',
|
||||
updated_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
|
||||
deleted_at TIMESTAMP WITH TIME ZONE,
|
||||
|
||||
CONSTRAINT uk_business_report_field_code UNIQUE (tenant_id, business_code, report_code, field_code)
|
||||
);
|
||||
|
||||
COMMENT ON TABLE report_field_config IS '报表字段配置表(维度/指标/筛选)';
|
||||
COMMENT ON COLUMN report_field_config.tenant_id IS '租户ID';
|
||||
COMMENT ON COLUMN report_field_config.business_code IS '所属业务编码';
|
||||
COMMENT ON COLUMN report_field_config.report_code IS '所属报表编码';
|
||||
COMMENT ON COLUMN report_field_config.field_code IS '字段编码(唯一)';
|
||||
COMMENT ON COLUMN report_field_config.field_name IS '字段显示名称';
|
||||
COMMENT ON COLUMN report_field_config.field_type IS '字段类型: STRING/INT/FLOAT/DATE/DATETIME/JSONB';
|
||||
COMMENT ON COLUMN report_field_config.data_type IS '数据类型: DIMENSION维度/INDICATOR指标/FILTER筛选';
|
||||
COMMENT ON COLUMN report_field_config.field_role IS '字段角色: DIMENSION维度/INDICATOR指标/FILTER筛选/FILTER_ONLY仅筛选';
|
||||
COMMENT ON COLUMN report_field_config.is_aggregatable IS '是否可统计(sum/count/avg)';
|
||||
COMMENT ON COLUMN report_field_config.is_filterable IS '是否可筛选';
|
||||
COMMENT ON COLUMN report_field_config.is_queryable IS '是否可查询';
|
||||
COMMENT ON COLUMN report_field_config.is_sortable IS '是否可排序';
|
||||
COMMENT ON COLUMN report_field_config.default_aggregate IS '默认聚合方式: SUM/COUNT/AVG/MAX/MIN';
|
||||
COMMENT ON COLUMN report_field_config.valid_aggregates IS '支持的聚合方式(JSON数组)';
|
||||
COMMENT ON COLUMN report_field_config.filter_operators IS '支持的筛选操作符(JSON数组)';
|
||||
COMMENT ON COLUMN report_field_config.expression IS '计算表达式(如退款率=退款金额/销售额*100)';
|
||||
COMMENT ON COLUMN report_field_config.expression_type IS '表达式类型: CALCULATED衍生/ORIGINAL原始';
|
||||
COMMENT ON COLUMN report_field_config.format_pattern IS '格式化模式(如 #,##0.00%)';
|
||||
COMMENT ON COLUMN report_field_config.unit IS '单位(如元/件/%)';
|
||||
COMMENT ON COLUMN report_field_config.dict_code IS '字典编码(用于下拉选项)';
|
||||
COMMENT ON COLUMN report_field_config.sort_order IS '排序顺序';
|
||||
COMMENT ON COLUMN report_field_config.group_name IS '字段分组名称';
|
||||
|
||||
CREATE INDEX IF NOT EXISTS idx_field_business_report ON report_field_config (business_code, report_code);
|
||||
CREATE INDEX IF NOT EXISTS idx_field_data_type ON report_field_config (data_type);
|
||||
CREATE INDEX IF NOT EXISTS idx_field_field_role ON report_field_config (field_role);
|
||||
|
||||
-- ----------------------------------------------------------
|
||||
-- 4. 数据抽取配置表 (extract_config)
|
||||
-- 100% 配置化，不硬编码任何抽取逻辑
|
||||
-- ----------------------------------------------------------
|
||||
CREATE TABLE IF NOT EXISTS report_extract_config (
|
||||
id BIGSERIAL PRIMARY KEY,
|
||||
tenant_id BIGINT NOT NULL DEFAULT 0,
|
||||
business_code VARCHAR(64) NOT NULL,
|
||||
report_code VARCHAR(64) NOT NULL,
|
||||
extract_code VARCHAR(64) NOT NULL,
|
||||
extract_name VARCHAR(128) NOT NULL,
|
||||
source_table_name VARCHAR(128) NOT NULL,
|
||||
source_table_alias VARCHAR(64) DEFAULT '',
|
||||
target_table_name VARCHAR(128) NOT NULL,
|
||||
is_enabled BOOLEAN DEFAULT TRUE,
|
||||
extract_type VARCHAR(32) NOT NULL DEFAULT 'FULL',
|
||||
extract_mode VARCHAR(32) NOT NULL DEFAULT 'DIRECT',
|
||||
extract_key_field VARCHAR(64) DEFAULT '',
|
||||
extract_key_format VARCHAR(64) DEFAULT '',
|
||||
group_by_fields JSONB DEFAULT '[]'::jsonb,
|
||||
filter_expression TEXT DEFAULT '',
|
||||
join_configs JSONB DEFAULT '[]'::jsonb,
|
||||
field_mappings JSONB DEFAULT '[]'::jsonb,
|
||||
transform_rules JSONB DEFAULT '[]'::jsonb,
|
||||
batch_size INT DEFAULT 1000,
|
||||
status VARCHAR(16) NOT NULL DEFAULT 'ACTIVE',
|
||||
creator VARCHAR(64) DEFAULT '',
|
||||
created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
|
||||
updater VARCHAR(64) DEFAULT '',
|
||||
updated_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
|
||||
deleted_at TIMESTAMP WITH TIME ZONE,
|
||||
|
||||
CONSTRAINT uk_business_report_extract_code UNIQUE (tenant_id, business_code, report_code, extract_code)
|
||||
);
|
||||
|
||||
COMMENT ON TABLE report_extract_config IS '数据抽取配置表';
|
||||
COMMENT ON COLUMN report_extract_config.tenant_id IS '租户ID';
|
||||
COMMENT ON COLUMN report_extract_config.business_code IS '所属业务编码';
|
||||
COMMENT ON COLUMN report_extract_config.report_code IS '所属报表编码';
|
||||
COMMENT ON COLUMN report_extract_config.extract_code IS '抽取配置编码';
|
||||
COMMENT ON COLUMN report_extract_config.extract_name IS '抽取配置名称';
|
||||
COMMENT ON COLUMN report_extract_config.source_table_name IS '源表表名';
|
||||
COMMENT ON COLUMN report_extract_config.source_table_alias IS '源表别名';
|
||||
COMMENT ON COLUMN report_extract_config.target_table_name IS '目标表表名(统计宽表)';
|
||||
COMMENT ON COLUMN report_extract_config.extract_type IS '抽取类型: FULL全量/INCREMENTAL增量';
|
||||
COMMENT ON COLUMN report_extract_config.extract_mode IS '抽取模式: DIRECT逐行抽取/AGGREGATE按GROUP BY聚合';
|
||||
COMMENT ON COLUMN report_extract_config.extract_key_field IS '增量抽取关键字段(如updated_at)';
|
||||
COMMENT ON COLUMN report_extract_config.extract_key_format IS '关键字段格式(如yyyy-MM-dd HH:mm:ss)';
|
||||
COMMENT ON COLUMN report_extract_config.group_by_fields IS 'AGGREGATE模式下的GROUP BY字段(JSON数组)';
|
||||
COMMENT ON COLUMN report_extract_config.filter_expression IS '抽取条件表达式';
|
||||
COMMENT ON COLUMN report_extract_config.join_configs IS '关联配置(JSON数组)';
|
||||
COMMENT ON COLUMN report_extract_config.field_mappings IS '字段映射配置(JSON数组)';
|
||||
COMMENT ON COLUMN report_extract_config.transform_rules IS '转换规则(JSON数组)';
|
||||
COMMENT ON COLUMN report_extract_config.batch_size IS '批次大小';
|
||||
|
||||
CREATE INDEX IF NOT EXISTS idx_extract_business_report ON report_extract_config (business_code, report_code);
|
||||
|
||||
-- ----------------------------------------------------------
|
||||
-- 5. 抽取记录表 (extract_log) - 用于幂等和追踪
|
||||
-- ----------------------------------------------------------
|
||||
CREATE TABLE IF NOT EXISTS report_extract_log (
|
||||
id BIGSERIAL PRIMARY KEY,
|
||||
tenant_id BIGINT NOT NULL DEFAULT 0,
|
||||
business_code VARCHAR(64) NOT NULL,
|
||||
report_code VARCHAR(64) NOT NULL,
|
||||
extract_code VARCHAR(64) NOT NULL,
|
||||
stat_date VARCHAR(16) NOT NULL,
|
||||
extract_type VARCHAR(32) NOT NULL,
|
||||
status VARCHAR(16) NOT NULL DEFAULT 'RUNNING',
|
||||
total_count INT DEFAULT 0,
|
||||
success_count INT DEFAULT 0,
|
||||
fail_count INT DEFAULT 0,
|
||||
start_time TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
|
||||
end_time TIMESTAMP WITH TIME ZONE,
|
||||
error_message TEXT DEFAULT '',
|
||||
executor VARCHAR(64) DEFAULT '',
|
||||
created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
|
||||
updated_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
|
||||
|
||||
CONSTRAINT uk_extract_keys UNIQUE (tenant_id, business_code, report_code, extract_code, stat_date)
|
||||
);
|
||||
|
||||
COMMENT ON TABLE report_extract_log IS '抽取记录表(幂等追踪)';
|
||||
COMMENT ON COLUMN report_extract_log.tenant_id IS '租户ID';
|
||||
COMMENT ON COLUMN report_extract_log.business_code IS '业务编码';
|
||||
COMMENT ON COLUMN report_extract_log.report_code IS '报表编码';
|
||||
COMMENT ON COLUMN report_extract_log.extract_code IS '抽取配置编码';
|
||||
COMMENT ON COLUMN report_extract_log.stat_date IS '统计日期(yyyy-MM-dd)';
|
||||
COMMENT ON COLUMN report_extract_log.extract_type IS '抽取类型: FULL/INCREMENTAL';
|
||||
COMMENT ON COLUMN report_extract_log.status IS '状态: RUNNING运行中/SUCCESS成功/FAILED失败';
|
||||
COMMENT ON COLUMN report_extract_log.total_count IS '总记录数';
|
||||
COMMENT ON COLUMN report_extract_log.success_count IS '成功记录数';
|
||||
COMMENT ON COLUMN report_extract_log.fail_count IS '失败记录数';
|
||||
CREATE INDEX IF NOT EXISTS idx_extract_log_business_report ON report_extract_log (business_code, report_code);
|
||||
CREATE INDEX IF NOT EXISTS idx_extract_log_stat_date ON report_extract_log (stat_date);
|
||||
|
||||
-- ============================================================
|
||||
-- 字段映射配置结构说明 (field_mappings JSONB)
|
||||
-- ============================================================
|
||||
-- [
|
||||
-- {
|
||||
-- "source_field": "order_id",
|
||||
-- "target_field": "order_id",
|
||||
-- "field_type": "STRING",
|
||||
-- "default_value": "",
|
||||
-- "transform_rule": {
|
||||
-- "type": "DIRECT/MAPPING/CALCULATE",
|
||||
-- "expression": ""
|
||||
-- }
|
||||
-- },
|
||||
-- {
|
||||
-- "source_field": "sale_amount",
|
||||
-- "target_field": "sale_amount_yuan",
|
||||
-- "field_type": "FLOAT",
|
||||
-- "default_value": 0,
|
||||
-- "transform_rule": {
|
||||
-- "type": "CALCULATE",
|
||||
-- "expression": "source_value / 100"
|
||||
-- }
|
||||
-- }
|
||||
-- ]
|
||||
|
||||
-- ============================================================
|
||||
-- 转换规则配置结构说明 (transform_rules JSONB)
|
||||
-- ============================================================
|
||||
-- [
|
||||
-- {
|
||||
-- "rule_code": "status_mapping",
|
||||
-- "rule_type": "MAPPING",
|
||||
-- "source_field": "order_status",
|
||||
-- "target_field": "status_name",
|
||||
-- "mapping": {
|
||||
-- "PENDING": "待支付",
|
||||
-- "PAID": "已支付",
|
||||
-- "SHIPPED": "已发货",
|
||||
-- "COMPLETED": "已完成",
|
||||
-- "REFUNDED": "已退款"
|
||||
-- }
|
||||
-- },
|
||||
-- {
|
||||
-- "rule_code": "date_format",
|
||||
-- "rule_type": "FORMAT",
|
||||
-- "source_field": "create_time",
|
||||
-- "target_field": "stat_date",
|
||||
-- "format": "yyyy-MM-dd"
|
||||
-- }
|
||||
-- ]
|
||||
|
||||
-- ============================================================
|
||||
-- 关联配置结构说明 (join_configs JSONB)
|
||||
-- ============================================================
|
||||
-- [
|
||||
-- {
|
||||
-- "join_table": "product_info",
|
||||
-- "join_alias": "p",
|
||||
-- "join_type": "LEFT",
|
||||
-- "join_condition": "s.product_id = p.id",
|
||||
-- "field_mappings": [
|
||||
-- {"source_field": "p.category_name", "target_field": "category_name"}
|
||||
-- ]
|
||||
-- }
|
||||
-- ]
|
||||
@@ -0,0 +1,229 @@
|
||||
-- =============================================
|
||||
-- 钉钉开放平台初始化数据
|
||||
--
|
||||
-- 执行后需配置 token（OAUTH2 读取 token 列，非 api_key）：
|
||||
-- UPDATE api_datasource_platform SET token = '你的oapi access_token'
|
||||
-- WHERE platform_code = 'dingtalk';
|
||||
-- =============================================
|
||||
|
||||
-- 1. 创建钉钉平台
|
||||
INSERT INTO api_datasource_platform (
|
||||
tenant_id, creator, created_at, updater, updated_at,
|
||||
platform_code, platform_name, description, status,
|
||||
api_base_url, auth_type,
|
||||
auth_config,
|
||||
rate_limit_per_minute, rate_limit_per_hour,
|
||||
concurrency_limit, request_timeout_ms, max_retries, retry_delay_ms
|
||||
) VALUES (
|
||||
1, 'admin', NOW(), 'admin', NOW(),
|
||||
'dingtalk', '钉钉', '钉钉开放平台数据同步', 'ACTIVE',
|
||||
'https://oapi.dingtalk.com', 'OAUTH2',
|
||||
'{
|
||||
"token_in_query": true,
|
||||
"query_key": "access_token"
|
||||
}'::jsonb,
|
||||
60, 3600, 5, 30000, 3, 1000
|
||||
);
|
||||
|
||||
-- 2. 部门列表（递归遍历全量部门树）
|
||||
-- 先从根（不传 dept_id）获取一级部门，再对每个子部门递归调用获取下级
|
||||
-- 请求：POST，access_token 在 URL 查询参数中
|
||||
-- 响应：{"errcode":0,"errmsg":"ok","result":[{...}]}
|
||||
INSERT INTO api_interface (
|
||||
tenant_id, creator, created_at, updater, updated_at,
|
||||
platform_id, name, code, url, method, status, auth_type,
|
||||
request_config, response_config, table_definition
|
||||
) VALUES (
|
||||
1, 'admin', NOW(), 'admin', NOW(),
|
||||
(SELECT id FROM api_datasource_platform WHERE platform_code = 'dingtalk'),
|
||||
'部门列表', 'department_list',
|
||||
'/topapi/v2/department/listsub', 'POST', 'active', 'inherit',
|
||||
'{
|
||||
"parameters_location": "query",
|
||||
"page_param": "cursor",
|
||||
"page_size_param": "pageSize",
|
||||
"language": "zh_CN",
|
||||
"recursive": {
|
||||
"key_field": "dept_id",
|
||||
"target_param": "dept_id"
|
||||
},
|
||||
"max_recursive_depth": 20
|
||||
}'::jsonb,
|
||||
'{
|
||||
"success_field": "errcode",
|
||||
"success_value": 0,
|
||||
"message_field": "errmsg",
|
||||
"list_path": "result"
|
||||
}'::jsonb,
|
||||
'{
|
||||
"table_name": "dingtalk_department",
|
||||
"columns": [
|
||||
{"name": "dept_id", "type": "BIGINT", "comment": "部门ID"},
|
||||
{"name": "name", "type": "VARCHAR(300)", "comment": "部门名称"},
|
||||
{"name": "parent_id", "type": "BIGINT", "comment": "父部门ID"},
|
||||
{"name": "create_dept_group", "type": "BOOLEAN", "comment": "是否同步创建关联企业群"},
|
||||
{"name": "auto_add_user", "type": "BOOLEAN", "comment": "新人是否自动加入部门群"}
|
||||
],
|
||||
"conflict_keys": ["dept_id"]
|
||||
}'::jsonb
|
||||
);
|
||||
|
||||
-- 3. 部门用户列表（prefetch→department_list，游标分页）
|
||||
-- 先递归获取全量部门树，再对每个部门分页拉取用户
|
||||
-- dept_id 为必填，通过 prefetch 从部门列表注入
|
||||
-- 响应：result.list（用户数组），result.has_more（是否还有更多），result.next_cursor（下一游标）
|
||||
INSERT INTO api_interface (
|
||||
tenant_id, creator, created_at, updater, updated_at,
|
||||
platform_id, name, code, url, method, status, auth_type,
|
||||
request_config, response_config, table_definition
|
||||
) VALUES (
|
||||
1, 'admin', NOW(), 'admin', NOW(),
|
||||
(SELECT id FROM api_datasource_platform WHERE platform_code = 'dingtalk'),
|
||||
'部门用户列表', 'user_list',
|
||||
'/topapi/v2/user/list', 'POST', 'active', 'inherit',
|
||||
'{
|
||||
"parameters_location": "query",
|
||||
"page_param": "cursor",
|
||||
"page_size_param": "size",
|
||||
"cursor_pagination": true,
|
||||
"size": 100,
|
||||
"order_field": "modify_desc",
|
||||
"language": "zh_CN",
|
||||
"initial_cursor": 0,
|
||||
"prefetch": {
|
||||
"url": "/topapi/v2/department/listsub",
|
||||
"method": "POST",
|
||||
"response_path": "result",
|
||||
"target_param": "dept_id",
|
||||
"value_field": "dept_id"
|
||||
}
|
||||
}'::jsonb,
|
||||
'{
|
||||
"success_field": "errcode",
|
||||
"success_value": 0,
|
||||
"message_field": "errmsg",
|
||||
"list_path": "result.list",
|
||||
"cursor_field": "result.next_cursor",
|
||||
"has_more_field": "result.has_more"
|
||||
}'::jsonb,
|
||||
'{
|
||||
"table_name": "dingtalk_user",
|
||||
"columns": [
|
||||
{"name": "userid", "type": "VARCHAR(100)", "comment": "用户userId"},
|
||||
{"name": "unionid", "type": "VARCHAR(200)", "comment": "用户唯一标识"},
|
||||
{"name": "name", "type": "VARCHAR(200)", "comment": "用户姓名"},
|
||||
{"name": "avatar", "type": "TEXT", "comment": "头像地址"},
|
||||
{"name": "state_code", "type": "VARCHAR(20)", "comment": "国际电话区号"},
|
||||
{"name": "mobile", "type": "VARCHAR(50)", "comment": "手机号码"},
|
||||
{"name": "hide_mobile", "type": "BOOLEAN", "comment": "是否号码隐藏"},
|
||||
{"name": "telephone", "type": "VARCHAR(50)", "comment": "分机号"},
|
||||
{"name": "job_number", "type": "VARCHAR(100)", "comment": "员工工号"},
|
||||
{"name": "title", "type": "VARCHAR(200)", "comment": "职位"},
|
||||
{"name": "email", "type": "VARCHAR(200)", "comment": "员工邮箱"},
|
||||
{"name": "org_email", "type": "VARCHAR(200)", "comment": "企业邮箱"},
|
||||
{"name": "work_place", "type": "VARCHAR(300)", "comment": "办公地点"},
|
||||
{"name": "remark", "type": "TEXT", "comment": "备注"},
|
||||
{"name": "dept_id_list", "type": "JSONB", "comment": "所属部门id列表"},
|
||||
{"name": "dept_order", "type": "INT", "comment": "员工在部门中的排序"},
|
||||
{"name": "extension", "type": "TEXT", "comment": "扩展属性"},
|
||||
{"name": "hired_date", "type": "BIGINT", "comment": "入职时间"},
|
||||
{"name": "active", "type": "BOOLEAN", "comment": "是否激活钉钉"},
|
||||
{"name": "admin", "type": "BOOLEAN", "comment": "是否企业管理员"},
|
||||
{"name": "boss", "type": "BOOLEAN", "comment": "是否企业老板"},
|
||||
{"name": "leader", "type": "BOOLEAN", "comment": "是否部门主管"},
|
||||
{"name": "exclusive_account", "type": "BOOLEAN", "comment": "是否企业账号"}
|
||||
],
|
||||
"conflict_keys": ["userid"]
|
||||
}'::jsonb
|
||||
);
|
||||
|
||||
-- 4. 考勤数据查询（单个用户单日，prefetch→user_list）
|
||||
-- 需先同步 user_list 获取全量用户，再对每个用户调用此接口查询考勤数据
|
||||
-- userid 通过 prefetch 从 user_list 注入，work_date 固定为当天日期
|
||||
-- 响应：result 对象（包含 attendance_result_list/check_record_list/approve_list 等）
|
||||
-- 注意：当前代码支持两级 prefetch 链（即本接口 prefetch→user_list，
|
||||
-- user_list 自有 prefetch→department_list），但 user_list 的 dept_id 依赖
|
||||
-- 在作为 prefetch 来源时不会自动解析。若需全量同步所有用户的考勤数据，
|
||||
-- 建议先手动同步 user_list 和 department_list，再同步此接口。
|
||||
-- 使用前请将 work_date 更新为目标日期，如 '2026-06-03'
|
||||
INSERT INTO api_interface (
|
||||
tenant_id, creator, created_at, updater, updated_at,
|
||||
platform_id, name, code, url, method, status, auth_type,
|
||||
request_config, response_config, table_definition
|
||||
) VALUES (
|
||||
1, 'admin', NOW(), 'admin', NOW(),
|
||||
(SELECT id FROM api_datasource_platform WHERE platform_code = 'dingtalk'),
|
||||
'考勤数据查询', 'attendance_getupdatedata',
|
||||
'/topapi/attendance/getupdatedata', 'POST', 'active', 'inherit',
|
||||
'{
|
||||
"parameters_location": "query",
|
||||
"prefetch": {
|
||||
"url": "/topapi/v2/user/list",
|
||||
"method": "POST",
|
||||
"response_path": "result.list",
|
||||
"target_param": "userid",
|
||||
"value_field": "userid"
|
||||
},
|
||||
"work_date": "2026-06-03"
|
||||
}'::jsonb,
|
||||
'{
|
||||
"success_field": "errcode",
|
||||
"success_value": 0,
|
||||
"message_field": "errmsg",
|
||||
"list_path": "result",
|
||||
"single_record": true
|
||||
}'::jsonb,
|
||||
'{
|
||||
"table_name": "dingtalk_attendance",
|
||||
"columns": [
|
||||
{"name": "userid", "type": "VARCHAR(100)", "comment": "用户userId"},
|
||||
{"name": "work_date", "type": "VARCHAR(30)", "comment": "查询日期"},
|
||||
{"name": "corpId", "type": "VARCHAR(100)", "comment": "企业corpId"},
|
||||
{"name": "attendance_result_list", "type": "JSONB", "comment": "打卡结果列表"},
|
||||
{"name": "check_record_list", "type": "JSONB", "comment": "打卡详情列表"},
|
||||
{"name": "approve_list", "type": "JSONB", "comment": "审批单列表"},
|
||||
{"name": "rest_time_vo_list", "type": "JSONB", "comment": "班次内休息信息"},
|
||||
{"name": "rest_end_time", "type": "BIGINT", "comment": "休息结束时间"},
|
||||
{"name": "rest_begin_time", "type": "BIGINT", "comment": "休息开始时间"}
|
||||
],
|
||||
"conflict_keys": ["userid", "work_date"]
|
||||
}'::jsonb
|
||||
);
|
||||
|
||||
-- 5. 角色列表（偏移量分页，offset/size + hasMore）
|
||||
-- 响应：result.list（角色组数组），result.hasMore（是否还有更多）
|
||||
-- 每个角色组内包含 roles 数组（{id, name}）你
|
||||
INSERT INTO api_interface (
|
||||
tenant_id, creator, created_at, updater, updated_at,
|
||||
platform_id, name, code, url, method, status, auth_type,
|
||||
request_config, response_config, table_definition
|
||||
) VALUES (
|
||||
1, 'admin', NOW(), 'admin', NOW(),
|
||||
(SELECT id FROM api_datasource_platform WHERE platform_code = 'dingtalk'),
|
||||
'角色列表', 'role_list',
|
||||
'/topapi/role/list', 'POST', 'active', 'inherit',
|
||||
'{
|
||||
"parameters_location": "query",
|
||||
"page_param": "offset",
|
||||
"page_size_param": "size",
|
||||
"size": 200,
|
||||
"offset": 0,
|
||||
"pagination_mode": "offset"
|
||||
}'::jsonb,
|
||||
'{
|
||||
"success_field": "errcode",
|
||||
"success_value": 0,
|
||||
"message_field": "errmsg",
|
||||
"list_path": "result.list",
|
||||
"has_more_field": "result.has_more"
|
||||
}'::jsonb,
|
||||
'{
|
||||
"table_name": "dingtalk_role",
|
||||
"columns": [
|
||||
{"name": "groupId", "type": "BIGINT", "comment": "角色组ID"},
|
||||
{"name": "name", "type": "VARCHAR(200)", "comment": "角色组名称"},
|
||||
{"name": "roles", "type": "JSONB", "comment": "角色列表"}
|
||||
],
|
||||
"conflict_keys": ["groupId"]
|
||||
}'::jsonb
|
||||
);
|
||||
@@ -0,0 +1,69 @@
|
||||
-- =============================================
|
||||
-- 钉钉智能人事平台初始化数据
|
||||
-- 注意：此平台使用 x-acs-dingtalk-access-token Header 认证
|
||||
-- API 基础地址为 https://api.dingtalk.com（与 oapi 不同）
|
||||
--
|
||||
-- 执行后需配置 token（OAUTH2 读取 token 列，非 api_key）：
|
||||
-- UPDATE api_datasource_platform SET token = '你的hrm access_token'
|
||||
-- WHERE platform_code = 'dingtalk_hrm';
|
||||
-- =============================================
|
||||
|
||||
-- 1. 创建钉钉智能人事平台
|
||||
INSERT INTO api_datasource_platform (
|
||||
tenant_id, creator, created_at, updater, updated_at,
|
||||
platform_code, platform_name, description, status,
|
||||
api_base_url, auth_type,
|
||||
auth_config,
|
||||
rate_limit_per_minute, rate_limit_per_hour,
|
||||
concurrency_limit, request_timeout_ms, max_retries, retry_delay_ms
|
||||
) VALUES (
|
||||
1, 'admin', NOW(), 'admin', NOW(),
|
||||
'dingtalk_hrm', '钉钉智能人事', '钉钉智能人事数据同步（header 认证）', 'ACTIVE',
|
||||
'https://api.dingtalk.com', 'OAUTH2',
|
||||
'{
|
||||
"token_in_query": false,
|
||||
"header_name": "x-acs-dingtalk-access-token",
|
||||
"header_format": "{token}"
|
||||
}'::jsonb,
|
||||
60, 3600, 5, 30000, 3, 1000
|
||||
);
|
||||
|
||||
-- 2. 企业职位列表（游标分页，hasMore 判结束）
|
||||
-- 响应：list（职位数组），nextToken（游标），hasMore（是否还有更多）
|
||||
-- 注意：此接口成功时无 errcode 字段，parseRespExt 在找不到 success_field 时自动跳过检查
|
||||
INSERT INTO api_interface (
|
||||
tenant_id, creator, created_at, updater, updated_at,
|
||||
platform_id, name, code, url, method, status, auth_type,
|
||||
request_config, response_config, table_definition
|
||||
) VALUES (
|
||||
1, 'admin', NOW(), 'admin', NOW(),
|
||||
(SELECT id FROM api_datasource_platform WHERE platform_code = 'dingtalk_hrm'),
|
||||
'企业职位列表', 'position_list',
|
||||
'/v1.0/hrm/positions/query', 'POST', 'active', 'inherit',
|
||||
'{
|
||||
"cursor_pagination": true,
|
||||
"page_param": "nextToken",
|
||||
"page_size_param": "maxResults",
|
||||
"pageSize": 200,
|
||||
"initial_cursor": 0,
|
||||
"parameters_location": "query"
|
||||
}'::jsonb,
|
||||
'{
|
||||
"list_path": "list",
|
||||
"cursor_field": "nextToken",
|
||||
"has_more_field": "hasMore"
|
||||
}'::jsonb,
|
||||
'{
|
||||
"table_name": "dingtalk_position",
|
||||
"columns": [
|
||||
{"name": "positionId", "type": "VARCHAR(100)", "comment": "职位ID"},
|
||||
{"name": "positionName", "type": "VARCHAR(300)", "comment": "职位名称"},
|
||||
{"name": "positionCategoryId", "type": "VARCHAR(100)", "comment": "职位类别ID"},
|
||||
{"name": "jobId", "type": "VARCHAR(100)", "comment": "所属职务ID"},
|
||||
{"name": "positionDes", "type": "TEXT", "comment": "职位描述"},
|
||||
{"name": "rankIdList", "type": "JSONB", "comment": "职级ID列表"},
|
||||
{"name": "status", "type": "INT", "comment": "职位状态 0启用 1停用"}
|
||||
],
|
||||
"conflict_keys": ["positionId"]
|
||||
}'::jsonb
|
||||
);
|
||||
@@ -0,0 +1,175 @@
|
||||
-- =============================================
|
||||
-- 钉钉智能薪酬平台初始化数据
|
||||
--
|
||||
-- 认证方式：app-id + signature 头部
|
||||
-- 签名算法：MD5(request_body_string + app_secret).toUpperCase()
|
||||
--
|
||||
-- 执行后需在 auth_config 中配置 app_id 和 app_secret：
|
||||
-- UPDATE api_datasource_platform SET auth_config = jsonb_set(
|
||||
-- jsonb_set(auth_config, '{app_id}', '"你的app-id"'),
|
||||
-- '{app_secret}', '"你的app-secret"'
|
||||
-- ) WHERE platform_code = 'dingtalk_salary';
|
||||
-- =============================================
|
||||
|
||||
-- 1. 创建钉钉智能薪酬平台
|
||||
INSERT INTO api_datasource_platform (
|
||||
tenant_id, creator, created_at, updater, updated_at,
|
||||
platform_code, platform_name, description, status,
|
||||
api_base_url, auth_type,
|
||||
auth_config,
|
||||
rate_limit_per_minute, rate_limit_per_hour,
|
||||
concurrency_limit, request_timeout_ms, max_retries, retry_delay_ms
|
||||
) VALUES (
|
||||
1, 'admin', NOW(), 'admin', NOW(),
|
||||
'dingtalk_salary', '钉钉智能薪酬', '钉钉智能薪酬数据同步（app-id + signature 头部认证）', 'ACTIVE',
|
||||
'https://salary.eapps.dingtalkcloud.com', 'APP_SIGNATURE',
|
||||
'{
|
||||
"app_id": "",
|
||||
"app_secret": "",
|
||||
"sign_algorithm": "md5_upper_body",
|
||||
"app_id_header": "app-id",
|
||||
"sign_header": "signature"
|
||||
}'::jsonb,
|
||||
60, 3600, 3, 30000, 3, 1000
|
||||
);
|
||||
|
||||
-- 2. 部门列表（无分页，单次请求返回全部部门）
|
||||
-- 无请求入参，直接 POST 即可
|
||||
-- 响应：{"code":"200","success":true,"data":[{"name":"研发部","value":"93639012","ext":"-1"},...]}
|
||||
INSERT INTO api_interface (
|
||||
tenant_id, creator, created_at, updater, updated_at,
|
||||
platform_id, name, code, url, method, status, auth_type,
|
||||
request_config, response_config, table_definition
|
||||
) VALUES (
|
||||
1, 'admin', NOW(), 'admin', NOW(),
|
||||
(SELECT id FROM api_datasource_platform WHERE platform_code = 'dingtalk_salary'),
|
||||
'公司部门列表', 'salary_dept_list',
|
||||
'/oapi/salary/dept/list', 'POST', 'active', 'inherit',
|
||||
'{}'::jsonb,
|
||||
'{
|
||||
"success_field": "code",
|
||||
"success_value": 200,
|
||||
"message_field": "msg",
|
||||
"list_path": "data"
|
||||
}'::jsonb,
|
||||
'{
|
||||
"table_name": "dingtalk_salary_dept",
|
||||
"columns": [
|
||||
{"name": "name", "type": "VARCHAR(300)", "comment": "部门名称"},
|
||||
{"name": "value", "type": "VARCHAR(100)", "comment": "部门ID"},
|
||||
{"name": "ext", "type": "VARCHAR(100)", "comment": "父部门ID，-1为根部门"}
|
||||
],
|
||||
"conflict_keys": ["value"]
|
||||
}'::jsonb
|
||||
);
|
||||
|
||||
-- 3. 按部门导出当月薪资报表（prefetch→salary_dept_list）
|
||||
-- 先通过 dept/list 获取全量部门列表，再对每个部门调用此接口
|
||||
-- deptId 通过 prefetch 注入，statisticsMonth 通过 row_inject 注入到每行
|
||||
-- 使用前请将 statisticsMonth 更新为目标月份，如 "2026-06"
|
||||
INSERT INTO api_interface (
|
||||
tenant_id, creator, created_at, updater, updated_at,
|
||||
platform_id, name, code, url, method, status, auth_type,
|
||||
request_config, response_config, table_definition
|
||||
) VALUES (
|
||||
1, 'admin', NOW(), 'admin', NOW(),
|
||||
(SELECT id FROM api_datasource_platform WHERE platform_code = 'dingtalk_salary'),
|
||||
'按部门薪资报表', 'salary_statistics',
|
||||
'/oapi/salary/statistics/dept', 'POST', 'active', 'inherit',
|
||||
'{
|
||||
"statisticsMonth": "2026-06",
|
||||
"row_inject": ["statisticsMonth"],
|
||||
"prefetch": {
|
||||
"url": "/oapi/salary/dept/list",
|
||||
"method": "POST",
|
||||
"response_path": "data",
|
||||
"target_param": "deptId",
|
||||
"value_field": "value"
|
||||
}
|
||||
}'::jsonb,
|
||||
'{
|
||||
"success_field": "code",
|
||||
"success_value": 200,
|
||||
"message_field": "msg",
|
||||
"list_path": "data"
|
||||
}'::jsonb,
|
||||
'{
|
||||
"table_name": "dingtalk_salary_statistics",
|
||||
"columns": [
|
||||
{"name": "deptId", "type": "BIGINT", "comment": "部门ID"},
|
||||
{"name": "statisticsMonth", "type": "VARCHAR(7)", "comment": "薪资月(yyyy-MM)"},
|
||||
{"name": "name", "type": "VARCHAR(200)", "comment": "报表项名称"},
|
||||
{"name": "value", "type": "VARCHAR(100)", "comment": "报表项数据"}
|
||||
],
|
||||
"conflict_keys": ["deptId", "statisticsMonth", "name"]
|
||||
}'::jsonb
|
||||
);
|
||||
|
||||
-- 4. 人力成本报表（无分页，单次请求返回指定月份的人力成本汇总）
|
||||
-- calBizId 为空时默认为当前月份，格式为 yyyyMM + "M"，如 "202606M"
|
||||
-- 使用前可将 calBizId 更新为目标月份
|
||||
-- 响应：data.sumStatisticsData 包含各成本项（itemId/itemName/itemDesc/sValue/value）
|
||||
INSERT INTO api_interface (
|
||||
tenant_id, creator, created_at, updater, updated_at,
|
||||
platform_id, name, code, url, method, status, auth_type,
|
||||
request_config, response_config, table_definition
|
||||
) VALUES (
|
||||
1, 'admin', NOW(), 'admin', NOW(),
|
||||
(SELECT id FROM api_datasource_platform WHERE platform_code = 'dingtalk_salary'),
|
||||
'人力成本报表', 'salary_statistic_report',
|
||||
'/oapi/salary/statistic/report/data', 'POST', 'active', 'inherit',
|
||||
'{
|
||||
"calBizId": "202606M"
|
||||
}'::jsonb,
|
||||
'{
|
||||
"success_field": "code",
|
||||
"success_value": 200,
|
||||
"message_field": "msg",
|
||||
"list_path": "data",
|
||||
"single_record": true
|
||||
}'::jsonb,
|
||||
'{
|
||||
"table_name": "dingtalk_salary_statistic_report",
|
||||
"columns": [
|
||||
{"name": "calBizId", "type": "VARCHAR(20)", "comment": "薪资月份"},
|
||||
{"name": "sumStatisticsData", "type": "JSONB", "comment": "人力成本报表数据"}
|
||||
],
|
||||
"conflict_keys": ["calBizId"]
|
||||
}'::jsonb
|
||||
);
|
||||
|
||||
-- 5. 按薪资组获取人力成本报表（无分页，单次请求返回指定薪资组的人力成本汇总）
|
||||
-- salaryGroupName 为必填，使用前需更新为实际的薪资组名称
|
||||
-- calBizId 为空时默认为当前月份，格式为 yyyyMM + "M"
|
||||
-- row_inject 会将请求中的 salaryGroupName 注入到每行，便于区分不同薪资组的报表
|
||||
INSERT INTO api_interface (
|
||||
tenant_id, creator, created_at, updater, updated_at,
|
||||
platform_id, name, code, url, method, status, auth_type,
|
||||
request_config, response_config, table_definition
|
||||
) VALUES (
|
||||
1, 'admin', NOW(), 'admin', NOW(),
|
||||
(SELECT id FROM api_datasource_platform WHERE platform_code = 'dingtalk_salary'),
|
||||
'按薪资组人力成本报表', 'salary_statistic_report_group',
|
||||
'/oapi/salary/statistic/report/groupData', 'POST', 'active', 'inherit',
|
||||
'{
|
||||
"calBizId": "202606M",
|
||||
"salaryGroupName": "",
|
||||
"row_inject": ["salaryGroupName"]
|
||||
}'::jsonb,
|
||||
'{
|
||||
"success_field": "code",
|
||||
"success_value": 200,
|
||||
"message_field": "msg",
|
||||
"list_path": "data",
|
||||
"single_record": true
|
||||
}'::jsonb,
|
||||
'{
|
||||
"table_name": "dingtalk_salary_statistic_report_group",
|
||||
"columns": [
|
||||
{"name": "calBizId", "type": "VARCHAR(20)", "comment": "薪资月份"},
|
||||
{"name": "salaryGroupName", "type": "VARCHAR(200)", "comment": "薪资组名称"},
|
||||
{"name": "sumStatisticsData", "type": "JSONB", "comment": "人力成本报表数据"}0
|
||||
],
|
||||
"conflict_keys": ["calBizId", "salaryGroupName"]
|
||||
}'::jsonb
|
||||
);
|
||||
@@ -0,0 +1,435 @@
|
||||
-- ============================================================
|
||||
-- 快手电商 - 报表系统初始化数据
|
||||
-- 基于 seed_data_kuaishou.sql 中实际表结构生成
|
||||
-- 覆盖报表: 订单 / 商品 / 售后 / 分销订单 / 代发订单
|
||||
-- ============================================================
|
||||
|
||||
-- =============================================
|
||||
-- 1. 业务配置 (report_business_config)
|
||||
-- =============================================
|
||||
INSERT INTO report_business_config (tenant_id, business_code, business_name, description, status, config, creator, created_at, updater, updated_at) VALUES
|
||||
(1, 'kuaishou_ecommerce', '快手电商', '快手电商平台订单、商品、售后、分销数据分析', 'ACTIVE',
|
||||
'{"platform":"kuaishou","timezone":"Asia/Shanghai"}'::jsonb,
|
||||
'admin', NOW(), 'admin', NOW())
|
||||
ON CONFLICT (tenant_id, business_code) DO NOTHING;
|
||||
|
||||
|
||||
-- =============================================
|
||||
-- 2. 报表配置 (report_report_config)
|
||||
-- =============================================
|
||||
|
||||
-- 2.1 订单分析
|
||||
INSERT INTO report_report_config (tenant_id, business_code, report_code, report_name, description, status, stat_table_name, stat_table_comment, date_field, primary_keys, conflict_keys, config, creator, created_at, updater, updated_at) VALUES
|
||||
(1, 'kuaishou_ecommerce', 'order_analysis', '订单分析', '基于 kuaishou_order_list 的订单维度聚合分析，支持按日期/状态/地区/支付方式/渠道等维度', 'ACTIVE',
|
||||
'kuaishou_order_stat', '快手订单统计宽表（聚合自 kuaishou_order_list）',
|
||||
'stat_date',
|
||||
'["id"]'::jsonb,
|
||||
'["stat_date","oid"]'::jsonb,
|
||||
'{"defaultTimeRange":"last_30_days","defaultDimensions":["stat_date"],"defaultIndicators":[{"field":"order_count","aggregate":"COUNT"},{"field":"total_fee","aggregate":"SUM"}]}'::jsonb,
|
||||
'admin', NOW(), 'admin', NOW())
|
||||
ON CONFLICT (tenant_id, business_code, report_code) DO NOTHING;
|
||||
|
||||
-- 2.2 商品分析
|
||||
INSERT INTO report_report_config (tenant_id, business_code, report_code, report_name, description, status, stat_table_name, stat_table_comment, date_field, primary_keys, conflict_keys, config, creator, created_at, updater, updated_at) VALUES
|
||||
(1, 'kuaishou_ecommerce', 'item_analysis', '商品分析', '基于 kuaishou_item_list 的商品维度销量、销售额分析，支持按类目/类型/上下架状态等维度', 'ACTIVE',
|
||||
'kuaishou_item_stat', '快手商品统计宽表（聚合自 kuaishou_item_list）',
|
||||
'stat_date',
|
||||
'["id"]'::jsonb,
|
||||
'["stat_date","kwai_item_id","category_id"]'::jsonb,
|
||||
'{"defaultTimeRange":"last_30_days","defaultDimensions":["stat_date","category_name"],"defaultIndicators":[{"field":"sales_volume","aggregate":"SUM"},{"field":"sales_amount","aggregate":"SUM"}]}'::jsonb,
|
||||
'admin', NOW(), 'admin', NOW())
|
||||
ON CONFLICT (tenant_id, business_code, report_code) DO NOTHING;
|
||||
|
||||
-- 2.3 售后分析
|
||||
INSERT INTO report_report_config (tenant_id, business_code, report_code, report_name, description, status, stat_table_name, stat_table_comment, date_field, primary_keys, conflict_keys, config, creator, created_at, updater, updated_at) VALUES
|
||||
(1, 'kuaishou_ecommerce', 'refund_analysis', '售后分析', '基于 kuaishou_refund_list 的售后维度分析，退款金额/退款率等核心指标', 'ACTIVE',
|
||||
'kuaishou_refund_stat', '快手售后统计宽表（聚合自 kuaishou_refund_list）',
|
||||
'stat_date',
|
||||
'["id"]'::jsonb,
|
||||
'["stat_date","refund_id"]'::jsonb,
|
||||
'{"defaultTimeRange":"last_30_days","defaultDimensions":["stat_date","refund_type"],"defaultIndicators":[{"field":"refund_count","aggregate":"COUNT"},{"field":"refund_fee","aggregate":"SUM"}]}'::jsonb,
|
||||
'admin', NOW(), 'admin', NOW())
|
||||
ON CONFLICT (tenant_id, business_code, report_code) DO NOTHING;
|
||||
|
||||
-- 2.4 分销订单分析
|
||||
INSERT INTO report_report_config (tenant_id, business_code, report_code, report_name, description, status, stat_table_name, stat_table_comment, date_field, primary_keys, conflict_keys, config, creator, created_at, updater, updated_at) VALUES
|
||||
(1, 'kuaishou_ecommerce', 'cps_order_analysis', '分销订单分析', '基于 kuaishou_cps_order_list 的分销订单维度分析，含分销收入/佣金等指标', 'ACTIVE',
|
||||
'kuaishou_cps_order_stat', '快手分销订单统计宽表（聚合自 kuaishou_cps_order_list）',
|
||||
'stat_date',
|
||||
'["id"]'::jsonb,
|
||||
'["stat_date","oid"]'::jsonb,
|
||||
'{"defaultTimeRange":"last_30_days","defaultDimensions":["stat_date","status"],"defaultIndicators":[{"field":"order_count","aggregate":"COUNT"},{"field":"estimated_income","aggregate":"SUM"}]}'::jsonb,
|
||||
'admin', NOW(), 'admin', NOW())
|
||||
ON CONFLICT (tenant_id, business_code, report_code) DO NOTHING;
|
||||
|
||||
-- 2.5 代发订单分析
|
||||
INSERT INTO report_report_config (tenant_id, business_code, report_code, report_name, description, status, stat_table_name, stat_table_comment, date_field, primary_keys, conflict_keys, config, creator, created_at, updater, updated_at) VALUES
|
||||
(1, 'kuaishou_ecommerce', 'dropshipping_analysis', '代发订单分析', '基于 kuaishou_dropshipping_order_list 的代发订单维度分析，按厂家/物流/地区等维度', 'ACTIVE',
|
||||
'kuaishou_dropshipping_stat', '快手代发订单统计宽表（聚合自 kuaishou_dropshipping_order_list）',
|
||||
'stat_date',
|
||||
'["id"]'::jsonb,
|
||||
'["stat_date","dropshipping_order_code"]'::jsonb,
|
||||
'{"defaultTimeRange":"last_30_days","defaultDimensions":["stat_date","dropshipping_status"],"defaultIndicators":[{"field":"order_count","aggregate":"COUNT"}]}'::jsonb,
|
||||
'admin', NOW(), 'admin', NOW())
|
||||
ON CONFLICT (tenant_id, business_code, report_code) DO NOTHING;
|
||||
|
||||
|
||||
-- =============================================
|
||||
-- 3. 字段配置 (report_field_config)
|
||||
-- =============================================
|
||||
|
||||
-- ----------------------------------------------------------
|
||||
-- 3.1 订单分析 - 字段配置
|
||||
-- 源表: kuaishou_order_list
|
||||
-- 实际列: oid, status, createTime, updateTime, payTime, sendTime, recvTime,
|
||||
-- refundTime, totalFee, expressFee, discountFee, originalPrice,
|
||||
-- buyerOpenId, sellerOpenId, buyerNick, sellerNick, remark, itemTitle,
|
||||
-- num, price, activityType, cpsType, payType, payChannel, channel,
|
||||
-- commentStatus, priorityDelivery, carrierType, carrierId,
|
||||
-- province, city, district, provinceCode, cityCode, districtCode
|
||||
-- ----------------------------------------------------------
|
||||
|
||||
-- 维度字段
|
||||
INSERT INTO report_field_config (tenant_id, business_code, report_code, field_code, field_name, field_type, data_type, field_role, is_aggregatable, is_filterable, is_queryable, is_sortable, default_aggregate, valid_aggregates, filter_operators, expression, expression_type, format_pattern, unit, dict_code, sort_order, group_name, status, creator, created_at, updater, updated_at) VALUES
|
||||
(1, 'kuaishou_ecommerce', 'order_analysis', 'stat_date', '统计日期', 'DATE', 'STRING', 'DIMENSION', false, true, true, true, '', '[]'::jsonb, '["=",">=","<=","BETWEEN"]'::jsonb, '', '', 'yyyy-MM-dd', '', '', 1, '时间维度', 'ACTIVE', 'admin', NOW(), 'admin', NOW()),
|
||||
(1, 'kuaishou_ecommerce', 'order_analysis', 'oid', '订单ID', 'BIGINT', 'STRING', 'DIMENSION', false, true, true, true, '', '[]'::jsonb, '["=","IN"]'::jsonb, '', '', '', '', '', 2, '订单维度', 'ACTIVE', 'admin', NOW(), 'admin', NOW()),
|
||||
(1, 'kuaishou_ecommerce', 'order_analysis', 'status', '订单状态', 'INT', 'STRING', 'DIMENSION', false, true, true, true, '', '[]'::jsonb, '["=","IN"]'::jsonb, '', '', '', '', 'order_status_dict', 3, '订单维度', 'ACTIVE', 'admin', NOW(), 'admin', NOW()),
|
||||
(1, 'kuaishou_ecommerce', 'order_analysis', 'province', '省份', 'VARCHAR', 'STRING', 'DIMENSION', false, true, true, true, '', '[]'::jsonb, '["=","IN","LIKE"]'::jsonb, '', '', '', '', '', 4, '地域维度', 'ACTIVE', 'admin', NOW(), 'admin', NOW()),
|
||||
(1, 'kuaishou_ecommerce', 'order_analysis', 'city', '城市', 'VARCHAR', 'STRING', 'DIMENSION', false, true, true, true, '', '[]'::jsonb, '["=","IN","LIKE"]'::jsonb, '', '', '', '', '', 5, '地域维度', 'ACTIVE', 'admin', NOW(), 'admin', NOW()),
|
||||
(1, 'kuaishou_ecommerce', 'order_analysis', 'district', '区县', 'VARCHAR', 'STRING', 'DIMENSION', false, true, true, true, '', '[]'::jsonb, '["=","IN","LIKE"]'::jsonb, '', '', '', '', '', 6, '地域维度', 'ACTIVE', 'admin', NOW(), 'admin', NOW()),
|
||||
(1, 'kuaishou_ecommerce', 'order_analysis', 'pay_type', '支付类型', 'INT', 'STRING', 'DIMENSION', false, true, true, true, '', '[]'::jsonb, '["=","IN"]'::jsonb, '', '', '', '', 'pay_type_dict', 7, '支付维度', 'ACTIVE', 'admin', NOW(), 'admin', NOW()),
|
||||
(1, 'kuaishou_ecommerce', 'order_analysis', 'pay_channel', '支付渠道', 'VARCHAR', 'STRING', 'DIMENSION', false, true, true, true, '', '[]'::jsonb, '["=","IN"]'::jsonb, '', '', '', '', '', 8, '支付维度', 'ACTIVE', 'admin', NOW(), 'admin', NOW()),
|
||||
(1, 'kuaishou_ecommerce', 'order_analysis', 'channel', '分销渠道', 'VARCHAR', 'STRING', 'DIMENSION', false, true, true, true, '', '[]'::jsonb, '["=","IN"]'::jsonb, '', '', '', '', '', 9, '渠道维度', 'ACTIVE', 'admin', NOW(), 'admin', NOW()),
|
||||
(1, 'kuaishou_ecommerce', 'order_analysis', 'activity_type', '活动类型', 'INT', 'STRING', 'DIMENSION', false, true, true, true, '', '[]'::jsonb, '["=","IN"]'::jsonb, '', '', '', '', '', 10, '活动维度', 'ACTIVE', 'admin', NOW(), 'admin', NOW()),
|
||||
(1, 'kuaishou_ecommerce', 'order_analysis', 'buyer_nick', '买家昵称', 'VARCHAR', 'STRING', 'DIMENSION', false, true, true, true, '', '[]'::jsonb, '["=","LIKE"]'::jsonb, '', '', '', '', '', 11, '用户维度', 'ACTIVE', 'admin', NOW(), 'admin', NOW()),
|
||||
(1, 'kuaishou_ecommerce', 'order_analysis', 'item_title', '商品标题', 'VARCHAR', 'STRING', 'DIMENSION', false, true, true, true, '', '[]'::jsonb, '["=","LIKE"]'::jsonb, '', '', '', '', '', 12, '商品维度', 'ACTIVE', 'admin', NOW(), 'admin', NOW()),
|
||||
|
||||
-- 指标字段
|
||||
(1, 'kuaishou_ecommerce', 'order_analysis', 'order_count', '订单数', 'INT', 'STRING', 'INDICATOR', true, false, true, true, 'COUNT', '["COUNT","SUM"]'::jsonb, '[]'::jsonb, '1', 'CALCULATED', '#,##0', '单', '', 21, '订单指标', 'ACTIVE', 'admin', NOW(), 'admin', NOW()),
|
||||
(1, 'kuaishou_ecommerce', 'order_analysis', 'total_fee', '订单金额(分)', 'BIGINT', 'STRING', 'INDICATOR', true, true, true, true, 'SUM', '["SUM","AVG","MAX","MIN"]'::jsonb, '["=",">=","<=","BETWEEN"]'::jsonb, '', 'ORIGINAL', '', '分', '', 22, '金额指标', 'ACTIVE', 'admin', NOW(), 'admin', NOW()),
|
||||
(1, 'kuaishou_ecommerce', 'order_analysis', 'total_fee_yuan', '订单金额(元)', 'FLOAT', 'STRING', 'INDICATOR', true, true, true, true, 'SUM', '["SUM","AVG","MAX","MIN"]'::jsonb, '["=",">=","<=","BETWEEN"]'::jsonb, 'total_fee / 100.0', 'CALCULATED', '#,##0.00', '元', '', 23, '金额指标', 'ACTIVE', 'admin', NOW(), 'admin', NOW()),
|
||||
(1, 'kuaishou_ecommerce', 'order_analysis', 'express_fee', '运费(分)', 'BIGINT', 'STRING', 'INDICATOR', true, true, true, true, 'SUM', '["SUM","AVG"]'::jsonb, '["=",">=","<="]'::jsonb, '', 'ORIGINAL', '', '分', '', 24, '金额指标', 'ACTIVE', 'admin', NOW(), 'admin', NOW()),
|
||||
(1, 'kuaishou_ecommerce', 'order_analysis', 'express_fee_yuan', '运费(元)', 'FLOAT', 'STRING', 'INDICATOR', true, true, true, true, 'SUM', '["SUM","AVG"]'::jsonb, '["=",">=","<="]'::jsonb, 'express_fee / 100.0', 'CALCULATED', '#,##0.00', '元', '', 25, '金额指标', 'ACTIVE', 'admin', NOW(), 'admin', NOW()),
|
||||
(1, 'kuaishou_ecommerce', 'order_analysis', 'discount_fee', '优惠金额(分)', 'BIGINT', 'STRING', 'INDICATOR', true, true, true, true, 'SUM', '["SUM","AVG"]'::jsonb, '["=",">=","<="]'::jsonb, '', 'ORIGINAL', '', '分', '', 26, '金额指标', 'ACTIVE', 'admin', NOW(), 'admin', NOW()),
|
||||
(1, 'kuaishou_ecommerce', 'order_analysis', 'discount_fee_yuan', '优惠金额(元)', 'FLOAT', 'STRING', 'INDICATOR', true, true, true, true, 'SUM', '["SUM","AVG"]'::jsonb, '["=",">=","<="]'::jsonb, 'discount_fee / 100.0', 'CALCULATED', '#,##0.00', '元', '', 27, '金额指标', 'ACTIVE', 'admin', NOW(), 'admin', NOW()),
|
||||
(1, 'kuaishou_ecommerce', 'order_analysis', 'original_price', '原价(分)', 'BIGINT', 'STRING', 'INDICATOR', true, true, true, true, 'SUM', '["SUM","AVG"]'::jsonb, '["=",">=","<="]'::jsonb, '', 'ORIGINAL', '', '分', '', 28, '金额指标', 'ACTIVE', 'admin', NOW(), 'admin', NOW()),
|
||||
(1, 'kuaishou_ecommerce', 'order_analysis', 'item_num', '商品件数', 'INT', 'STRING', 'INDICATOR', true, true, true, true, 'SUM', '["SUM","AVG"]'::jsonb, '["=",">=","<="]'::jsonb, '', 'ORIGINAL', '#,##0', '件', '', 29, '商品指标', 'ACTIVE', 'admin', NOW(), 'admin', NOW()),
|
||||
(1, 'kuaishou_ecommerce', 'order_analysis', 'avg_order_amount', '客单价(元)', 'FLOAT', 'STRING', 'INDICATOR', true, false, true, true, 'AVG', '["AVG"]'::jsonb, '[]'::jsonb, 'total_fee / 100.0 / NULLIF(order_count, 0)', 'CALCULATED', '#,##0.00', '元', '', 30, '金额指标', 'ACTIVE', 'admin', NOW(), 'admin', NOW()),
|
||||
(1, 'kuaishou_ecommerce', 'order_analysis', 'discount_rate', '优惠率(%)', 'FLOAT', 'STRING', 'INDICATOR', true, false, true, true, 'AVG', '["AVG"]'::jsonb, '[]'::jsonb, 'CASE WHEN SUM(total_fee) > 0 THEN SUM(discount_fee) * 100.0 / (SUM(total_fee) + SUM(discount_fee)) ELSE 0 END', 'CALCULATED', '#,##0.00', '%', '', 31, '比率指标', 'ACTIVE', 'admin', NOW(), 'admin', NOW())
|
||||
ON CONFLICT (tenant_id, business_code, report_code, field_code) DO NOTHING;
|
||||
|
||||
|
||||
-- ----------------------------------------------------------
|
||||
-- 3.2 商品分析 - 字段配置
|
||||
-- 源表: kuaishou_item_list
|
||||
-- 实际列: kwaiItemId, relItemId, title, details, categoryId, categoryName,
|
||||
-- price, volume, status, auditStatus, auditReason, shelfStatus,
|
||||
-- itemType, createTime, updateTime, expressTemplateId, linkUrl
|
||||
-- ----------------------------------------------------------
|
||||
|
||||
-- 维度字段
|
||||
INSERT INTO report_field_config (tenant_id, business_code, report_code, field_code, field_name, field_type, data_type, field_role, is_aggregatable, is_filterable, is_queryable, is_sortable, default_aggregate, valid_aggregates, filter_operators, expression, expression_type, format_pattern, unit, dict_code, sort_order, group_name, status, creator, created_at, updater, updated_at) VALUES
|
||||
(1, 'kuaishou_ecommerce', 'item_analysis', 'stat_date', '统计日期', 'DATE', 'STRING', 'DIMENSION', false, true, true, true, '', '[]'::jsonb, '["=",">=","<=","BETWEEN"]'::jsonb, '', '', 'yyyy-MM-dd', '', '', 1, '时间维度', 'ACTIVE', 'admin', NOW(), 'admin', NOW()),
|
||||
(1, 'kuaishou_ecommerce', 'item_analysis', 'kwai_item_id', '商品ID', 'BIGINT', 'STRING', 'DIMENSION', false, true, true, true, '', '[]'::jsonb, '["=","IN"]'::jsonb, '', '', '', '', '', 2, '商品维度', 'ACTIVE', 'admin', NOW(), 'admin', NOW()),
|
||||
(1, 'kuaishou_ecommerce', 'item_analysis', 'title', '商品标题', 'VARCHAR', 'STRING', 'DIMENSION', false, true, true, true, '', '[]'::jsonb, '["=","LIKE","IN"]'::jsonb, '', '', '', '', '', 3, '商品维度', 'ACTIVE', 'admin', NOW(), 'admin', NOW()),
|
||||
(1, 'kuaishou_ecommerce', 'item_analysis', 'category_id', '类目ID', 'INT', 'STRING', 'DIMENSION', false, true, true, true, '', '[]'::jsonb, '["=","IN"]'::jsonb, '', '', '', '', '', 4, '类目维度', 'ACTIVE', 'admin', NOW(), 'admin', NOW()),
|
||||
(1, 'kuaishou_ecommerce', 'item_analysis', 'category_name', '类目名称', 'VARCHAR', 'STRING', 'DIMENSION', false, true, true, true, '', '[]'::jsonb, '["=","IN","LIKE"]'::jsonb, '', '', '', '', '', 5, '类目维度', 'ACTIVE', 'admin', NOW(), 'admin', NOW()),
|
||||
(1, 'kuaishou_ecommerce', 'item_analysis', 'item_type', '商品类型', 'INT', 'STRING', 'DIMENSION', false, true, true, true, '', '[]'::jsonb, '["=","IN"]'::jsonb, '', '', '', '', 'item_type_dict', 6, '商品维度', 'ACTIVE', 'admin', NOW(), 'admin', NOW()),
|
||||
(1, 'kuaishou_ecommerce', 'item_analysis', 'shelf_status', '上下架状态', 'INT', 'STRING', 'DIMENSION', false, true, true, true, '', '[]'::jsonb, '["=","IN"]'::jsonb, '', '', '', '', 'shelf_status_dict', 7, '商品维度', 'ACTIVE', 'admin', NOW(), 'admin', NOW()),
|
||||
(1, 'kuaishou_ecommerce', 'item_analysis', 'audit_status', '审核状态', 'INT', 'STRING', 'DIMENSION', false, true, true, true, '', '[]'::jsonb, '["=","IN"]'::jsonb, '', '', '', '', '', 8, '商品维度', 'ACTIVE', 'admin', NOW(), 'admin', NOW()),
|
||||
|
||||
-- 指标字段
|
||||
(1, 'kuaishou_ecommerce', 'item_analysis', 'sales_volume', '销量', 'INT', 'STRING', 'INDICATOR', true, true, true, true, 'SUM', '["SUM","AVG","MAX","MIN"]'::jsonb, '["=",">=","<=","BETWEEN"]'::jsonb, '', 'ORIGINAL', '#,##0', '件', '', 21, '销售指标', 'ACTIVE', 'admin', NOW(), 'admin', NOW()),
|
||||
(1, 'kuaishou_ecommerce', 'item_analysis', 'price', '商品价格(分)', 'BIGINT', 'STRING', 'INDICATOR', true, true, true, true, 'AVG', '["AVG","MAX","MIN"]'::jsonb, '["=",">=","<="]'::jsonb, '', 'ORIGINAL', '', '分', '', 22, '价格指标', 'ACTIVE', 'admin', NOW(), 'admin', NOW()),
|
||||
(1, 'kuaishou_ecommerce', 'item_analysis', 'price_yuan', '商品价格(元)', 'FLOAT', 'STRING', 'INDICATOR', true, true, true, true, 'AVG', '["AVG","MAX","MIN"]'::jsonb, '["=",">=","<="]'::jsonb, 'price / 100.0', 'CALCULATED', '#,##0.00', '元', '', 23, '价格指标', 'ACTIVE', 'admin', NOW(), 'admin', NOW()),
|
||||
(1, 'kuaishou_ecommerce', 'item_analysis', 'sales_amount', '销售额(分)', 'BIGINT', 'STRING', 'INDICATOR', true, true, true, true, 'SUM', '["SUM","AVG","MAX","MIN"]'::jsonb, '["=",">=","<="]'::jsonb, '', 'ORIGINAL', '', '分', '', 24, '销售指标', 'ACTIVE', 'admin', NOW(), 'admin', NOW()),
|
||||
(1, 'kuaishou_ecommerce', 'item_analysis', 'sales_amount_yuan', '销售额(元)', 'FLOAT', 'STRING', 'INDICATOR', true, true, true, true, 'SUM', '["SUM","AVG","MAX","MIN"]'::jsonb, '["=",">=","<="]'::jsonb, 'sales_amount / 100.0', 'CALCULATED', '#,##0.00', '元', '', 25, '销售指标', 'ACTIVE', 'admin', NOW(), 'admin', NOW()),
|
||||
(1, 'kuaishou_ecommerce', 'item_analysis', 'item_count', '商品数', 'INT', 'STRING', 'INDICATOR', true, false, true, true, 'COUNT', '["COUNT"]'::jsonb, '[]'::jsonb, 'COUNT(DISTINCT kwai_item_id)', 'CALCULATED', '#,##0', '个', '', 26, '商品指标', 'ACTIVE', 'admin', NOW(), 'admin', NOW()),
|
||||
(1, 'kuaishou_ecommerce', 'item_analysis', 'avg_price_per_unit', '件均价(元)', 'FLOAT', 'STRING', 'INDICATOR', true, false, true, true, 'AVG', '["AVG"]'::jsonb, '[]'::jsonb, 'SUM(sales_amount) / 100.0 / NULLIF(SUM(sales_volume), 0)', 'CALCULATED', '#,##0.00', '元', '', 27, '价格指标', 'ACTIVE', 'admin', NOW(), 'admin', NOW())
|
||||
ON CONFLICT (tenant_id, business_code, report_code, field_code) DO NOTHING;
|
||||
|
||||
|
||||
-- ----------------------------------------------------------
|
||||
-- 3.3 售后分析 - 字段配置
|
||||
-- 源表: kuaishou_refund_list
|
||||
-- 实际列: refundId, oid, itemId, skuId, relSkuId, skuNick, handlingWay,
|
||||
-- negotiateStatus, refundFee, refundReason, refundReasonDesc,
|
||||
-- refundDesc, refundType, status, receiptStatus, buyerId, sellerId,
|
||||
-- logisticsId, relItemId, submitTime, createTime, updateTime,
|
||||
-- negotiateUpdateTime, endTime, expireTime
|
||||
-- ----------------------------------------------------------
|
||||
|
||||
-- 维度字段
|
||||
INSERT INTO report_field_config (tenant_id, business_code, report_code, field_code, field_name, field_type, data_type, field_role, is_aggregatable, is_filterable, is_queryable, is_sortable, default_aggregate, valid_aggregates, filter_operators, expression, expression_type, format_pattern, unit, dict_code, sort_order, group_name, status, creator, created_at, updater, updated_at) VALUES
|
||||
(1, 'kuaishou_ecommerce', 'refund_analysis', 'stat_date', '统计日期', 'DATE', 'STRING', 'DIMENSION', false, true, true, true, '', '[]'::jsonb, '["=",">=","<=","BETWEEN"]'::jsonb, '', '', 'yyyy-MM-dd', '', '', 1, '时间维度', 'ACTIVE', 'admin', NOW(), 'admin', NOW()),
|
||||
(1, 'kuaishou_ecommerce', 'refund_analysis', 'refund_id', '售后单ID', 'BIGINT', 'STRING', 'DIMENSION', false, true, true, true, '', '[]'::jsonb, '["=","IN"]'::jsonb, '', '', '', '', '', 2, '售后维度', 'ACTIVE', 'admin', NOW(), 'admin', NOW()),
|
||||
(1, 'kuaishou_ecommerce', 'refund_analysis', 'refund_type', '售后类型', 'INT', 'STRING', 'DIMENSION', false, true, true, true, '', '[]'::jsonb, '["=","IN"]'::jsonb, '', '', '', '', 'refund_type_dict', 3, '售后维度', 'ACTIVE', 'admin', NOW(), 'admin', NOW()),
|
||||
(1, 'kuaishou_ecommerce', 'refund_analysis', 'refund_reason_desc', '售后原因', 'VARCHAR', 'STRING', 'DIMENSION', false, true, true, true, '', '[]'::jsonb, '["=","IN","LIKE"]'::jsonb, '', '', '', '', '', 4, '售后维度', 'ACTIVE', 'admin', NOW(), 'admin', NOW()),
|
||||
(1, 'kuaishou_ecommerce', 'refund_analysis', 'status', '售后状态', 'INT', 'STRING', 'DIMENSION', false, true, true, true, '', '[]'::jsonb, '["=","IN"]'::jsonb, '', '', '', '', 'refund_status_dict', 5, '售后维度', 'ACTIVE', 'admin', NOW(), 'admin', NOW()),
|
||||
(1, 'kuaishou_ecommerce', 'refund_analysis', 'handling_way', '处理方式', 'INT', 'STRING', 'DIMENSION', false, true, true, true, '', '[]'::jsonb, '["=","IN"]'::jsonb, '', '', '', '', '', 6, '售后维度', 'ACTIVE', 'admin', NOW(), 'admin', NOW()),
|
||||
(1, 'kuaishou_ecommerce', 'refund_analysis', 'negotiate_status', '协商状态', 'INT', 'STRING', 'DIMENSION', false, true, true, true, '', '[]'::jsonb, '["=","IN"]'::jsonb, '', '', '', '', '', 7, '售后维度', 'ACTIVE', 'admin', NOW(), 'admin', NOW()),
|
||||
|
||||
-- 指标字段
|
||||
(1, 'kuaishou_ecommerce', 'refund_analysis', 'refund_count', '售后单数', 'INT', 'STRING', 'INDICATOR', true, false, true, true, 'COUNT', '["COUNT","SUM"]'::jsonb, '[]'::jsonb, '1', 'CALCULATED', '#,##0', '单', '', 21, '售后指标', 'ACTIVE', 'admin', NOW(), 'admin', NOW()),
|
||||
(1, 'kuaishou_ecommerce', 'refund_analysis', 'refund_fee', '退款金额(分)', 'BIGINT', 'STRING', 'INDICATOR', true, true, true, true, 'SUM', '["SUM","AVG","MAX","MIN"]'::jsonb, '["=",">=","<="]'::jsonb, '', 'ORIGINAL', '', '分', '', 22, '金额指标', 'ACTIVE', 'admin', NOW(), 'admin', NOW()),
|
||||
(1, 'kuaishou_ecommerce', 'refund_analysis', 'refund_fee_yuan', '退款金额(元)', 'FLOAT', 'STRING', 'INDICATOR', true, true, true, true, 'SUM', '["SUM","AVG","MAX","MIN"]'::jsonb, '["=",">=","<="]'::jsonb, 'refund_fee / 100.0', 'CALCULATED', '#,##0.00', '元', '', 23, '金额指标', 'ACTIVE', 'admin', NOW(), 'admin', NOW()),
|
||||
(1, 'kuaishou_ecommerce', 'refund_analysis', 'avg_refund_amount', '笔均退款(元)', 'FLOAT', 'STRING', 'INDICATOR', true, false, true, true, 'AVG', '["AVG"]'::jsonb, '[]'::jsonb, 'SUM(refund_fee) / 100.0 / NULLIF(SUM(refund_count), 0)', 'CALCULATED', '#,##0.00', '元', '', 24, '金额指标', 'ACTIVE', 'admin', NOW(), 'admin', NOW()),
|
||||
(1, 'kuaishou_ecommerce', 'refund_analysis', 'negotiate_count', '协商中数量', 'INT', 'STRING', 'INDICATOR', true, false, true, true, 'SUM', '["SUM"]'::jsonb, '[]'::jsonb, 'CASE WHEN negotiate_status = 1 THEN 1 ELSE 0 END', 'CALCULATED', '#,##0', '单', '', 25, '售后指标', 'ACTIVE', 'admin', NOW(), 'admin', NOW())
|
||||
ON CONFLICT (tenant_id, business_code, report_code, field_code) DO NOTHING;
|
||||
|
||||
|
||||
-- ----------------------------------------------------------
|
||||
-- 3.4 分销订单分析 - 字段配置
|
||||
-- 源表: kuaishou_cps_order_list
|
||||
-- 实际列: oid, distributorId, distributorName, sellerId, status,
|
||||
-- settlementTime, settlementSuccessTime, refundTime, payTime,
|
||||
-- expressFee, totalFee, commissionRate, estimatedIncome,
|
||||
-- createTime, updateTime, platformDpRate, activityUserId,
|
||||
-- activityUserNickname, investmentPromotionRate,
|
||||
-- investmentPromotionAmount, buyerOpenId, settlementBizType,
|
||||
-- promoterServiceInCome, promoterExcitationInCome,
|
||||
-- investmentServiceInCome, investmentExcitationInCome,
|
||||
-- orderChannel, activityId, itemId
|
||||
-- ----------------------------------------------------------
|
||||
|
||||
-- 维度字段
|
||||
INSERT INTO report_field_config (tenant_id, business_code, report_code, field_code, field_name, field_type, data_type, field_role, is_aggregatable, is_filterable, is_queryable, is_sortable, default_aggregate, valid_aggregates, filter_operators, expression, expression_type, format_pattern, unit, dict_code, sort_order, group_name, status, creator, created_at, updater, updated_at) VALUES
|
||||
(1, 'kuaishou_ecommerce', 'cps_order_analysis', 'stat_date', '统计日期', 'DATE', 'STRING', 'DIMENSION', false, true, true, true, '', '[]'::jsonb, '["=",">=","<=","BETWEEN"]'::jsonb, '', '', 'yyyy-MM-dd', '', '', 1, '时间维度', 'ACTIVE', 'admin', NOW(), 'admin', NOW()),
|
||||
(1, 'kuaishou_ecommerce', 'cps_order_analysis', 'oid', '订单ID', 'BIGINT', 'STRING', 'DIMENSION', false, true, true, true, '', '[]'::jsonb, '["=","IN"]'::jsonb, '', '', '', '', '', 2, '订单维度', 'ACTIVE', 'admin', NOW(), 'admin', NOW()),
|
||||
(1, 'kuaishou_ecommerce', 'cps_order_analysis', 'status', '分销单状态', 'INT', 'STRING', 'DIMENSION', false, true, true, true, '', '[]'::jsonb, '["=","IN"]'::jsonb, '', '', '', '', 'cps_status_dict', 3, '订单维度', 'ACTIVE', 'admin', NOW(), 'admin', NOW()),
|
||||
(1, 'kuaishou_ecommerce', 'cps_order_analysis', 'distributor_name', '分销者', 'VARCHAR', 'STRING', 'DIMENSION', false, true, true, true, '', '[]'::jsonb, '["=","LIKE"]'::jsonb, '', '', '', '', '', 4, '分销维度', 'ACTIVE', 'admin', NOW(), 'admin', NOW()),
|
||||
(1, 'kuaishou_ecommerce', 'cps_order_analysis', 'order_channel', '出单渠道', 'VARCHAR', 'STRING', 'DIMENSION', false, true, true, true, '', '[]'::jsonb, '["=","IN"]'::jsonb, '', '', '', '', '', 5, '渠道维度', 'ACTIVE', 'admin', NOW(), 'admin', NOW()),
|
||||
(1, 'kuaishou_ecommerce', 'cps_order_analysis', 'settlement_biz_type', '业务类型', 'INT', 'STRING', 'DIMENSION', false, true, true, true, '', '[]'::jsonb, '["=","IN"]'::jsonb, '', '', '', '', '', 6, '业务维度', 'ACTIVE', 'admin', NOW(), 'admin', NOW()),
|
||||
|
||||
-- 指标字段
|
||||
(1, 'kuaishou_ecommerce', 'cps_order_analysis', 'order_count', '订单数', 'INT', 'STRING', 'INDICATOR', true, false, true, true, 'COUNT', '["COUNT","SUM"]'::jsonb, '[]'::jsonb, '1', 'CALCULATED', '#,##0', '单', '', 21, '订单指标', 'ACTIVE', 'admin', NOW(), 'admin', NOW()),
|
||||
(1, 'kuaishou_ecommerce', 'cps_order_analysis', 'total_fee', '实付金额(分)', 'BIGINT', 'STRING', 'INDICATOR', true, true, true, true, 'SUM', '["SUM","AVG","MAX","MIN"]'::jsonb, '["=",">=","<="]'::jsonb, '', 'ORIGINAL', '', '分', '', 22, '金额指标', 'ACTIVE', 'admin', NOW(), 'admin', NOW()),
|
||||
(1, 'kuaishou_ecommerce', 'cps_order_analysis', 'total_fee_yuan', '实付金额(元)', 'FLOAT', 'STRING', 'INDICATOR', true, true, true, true, 'SUM', '["SUM","AVG","MAX","MIN"]'::jsonb, '["=",">=","<="]'::jsonb, 'total_fee / 100.0', 'CALCULATED', '#,##0.00', '元', '', 23, '金额指标', 'ACTIVE', 'admin', NOW(), 'admin', NOW()),
|
||||
(1, 'kuaishou_ecommerce', 'cps_order_analysis', 'express_fee', '运费(分)', 'BIGINT', 'STRING', 'INDICATOR', true, true, true, true, 'SUM', '["SUM","AVG"]'::jsonb, '["=",">=","<="]'::jsonb, '', 'ORIGINAL', '', '分', '', 24, '金额指标', 'ACTIVE', 'admin', NOW(), 'admin', NOW()),
|
||||
(1, 'kuaishou_ecommerce', 'cps_order_analysis', 'estimated_income', '分销预估收入(分)', 'BIGINT', 'STRING', 'INDICATOR', true, true, true, true, 'SUM', '["SUM","AVG","MAX"]'::jsonb, '["=",">=","<="]'::jsonb, '', 'ORIGINAL', '', '分', '', 25, '收入指标', 'ACTIVE', 'admin', NOW(), 'admin', NOW()),
|
||||
(1, 'kuaishou_ecommerce', 'cps_order_analysis', 'estimated_income_yuan', '分销预估收入(元)', 'FLOAT', 'STRING', 'INDICATOR', true, true, true, true, 'SUM', '["SUM","AVG","MAX"]'::jsonb, '["=",">=","<="]'::jsonb, 'estimated_income / 100.0', 'CALCULATED', '#,##0.00', '元', '', 26, '收入指标', 'ACTIVE', 'admin', NOW(), 'admin', NOW()),
|
||||
(1, 'kuaishou_ecommerce', 'cps_order_analysis', 'commission_rate', '分销佣金率(%)', 'BIGINT', 'STRING', 'INDICATOR', true, true, true, true, 'AVG', '["AVG","MAX","MIN"]'::jsonb, '["=",">=","<="]'::jsonb, 'commission_rate / 10.0', 'CALCULATED', '#,##0.0', '%', '', 27, '佣金指标', 'ACTIVE', 'admin', NOW(), 'admin', NOW())
|
||||
ON CONFLICT (tenant_id, business_code, report_code, field_code) DO NOTHING;
|
||||
|
||||
|
||||
-- ----------------------------------------------------------
|
||||
-- 3.5 代发订单分析 - 字段配置
|
||||
-- 源表: kuaishou_dropshipping_order_list
|
||||
-- 实际列: oid, dropshippingOrderCode, payTime, buyerOpenId, buyerNick,
|
||||
-- sellerOpenId, sellerNick, orderStatus, orderStatusDesc,
|
||||
-- refundStatus, refundStatusDesc, orderType, orderTypeDesc,
|
||||
-- deliveryTime, createTime, updateTime, waybillCode,
|
||||
-- expressCompanyCode, expressCompanyName, factoryCode, factoryName,
|
||||
-- allocateTime, dropshippingStatus, dropshippingStatusDesc,
|
||||
-- cancelAllocateTime, cancelAllocateReason, name, mobile,
|
||||
-- provinceName, cityName, districtName, detailAddress
|
||||
-- ----------------------------------------------------------
|
||||
|
||||
-- 维度字段
|
||||
INSERT INTO report_field_config (tenant_id, business_code, report_code, field_code, field_name, field_type, data_type, field_role, is_aggregatable, is_filterable, is_queryable, is_sortable, default_aggregate, valid_aggregates, filter_operators, expression, expression_type, format_pattern, unit, dict_code, sort_order, group_name, status, creator, created_at, updater, updated_at) VALUES
|
||||
(1, 'kuaishou_ecommerce', 'dropshipping_analysis', 'stat_date', '统计日期', 'DATE', 'STRING', 'DIMENSION', false, true, true, true, '', '[]'::jsonb, '["=",">=","<=","BETWEEN"]'::jsonb, '', '', 'yyyy-MM-dd', '', '', 1, '时间维度', 'ACTIVE', 'admin', NOW(), 'admin', NOW()),
|
||||
(1, 'kuaishou_ecommerce', 'dropshipping_analysis', 'dropshipping_order_code', '代发订单编码', 'VARCHAR', 'STRING', 'DIMENSION', false, true, true, true, '', '[]'::jsonb, '["=","IN"]'::jsonb, '', '', '', '', '', 2, '订单维度', 'ACTIVE', 'admin', NOW(), 'admin', NOW()),
|
||||
(1, 'kuaishou_ecommerce', 'dropshipping_analysis', 'order_status', '订单状态', 'INT', 'STRING', 'DIMENSION', false, true, true, true, '', '[]'::jsonb, '["=","IN"]'::jsonb, '', '', '', '', 'dropshipping_status_dict', 3, '订单维度', 'ACTIVE', 'admin', NOW(), 'admin', NOW()),
|
||||
(1, 'kuaishou_ecommerce', 'dropshipping_analysis', 'dropshipping_status', '代发状态', 'INT', 'STRING', 'DIMENSION', false, true, true, true, '', '[]'::jsonb, '["=","IN"]'::jsonb, '', '', '', '', '', 4, '代发维度', 'ACTIVE', 'admin', NOW(), 'admin', NOW()),
|
||||
(1, 'kuaishou_ecommerce', 'dropshipping_analysis', 'factory_name', '代发厂家', 'VARCHAR', 'STRING', 'DIMENSION', false, true, true, true, '', '[]'::jsonb, '["=","LIKE"]'::jsonb, '', '', '', '', '', 5, '厂家维度', 'ACTIVE', 'admin', NOW(), 'admin', NOW()),
|
||||
(1, 'kuaishou_ecommerce', 'dropshipping_analysis', 'express_company_name', '物流公司', 'VARCHAR', 'STRING', 'DIMENSION', false, true, true, true, '', '[]'::jsonb, '["=","LIKE"]'::jsonb, '', '', '', '', '', 6, '物流维度', 'ACTIVE', 'admin', NOW(), 'admin', NOW()),
|
||||
(1, 'kuaishou_ecommerce', 'dropshipping_analysis', 'province_name', '省份', 'VARCHAR', 'STRING', 'DIMENSION', false, true, true, true, '', '[]'::jsonb, '["=","IN","LIKE"]'::jsonb, '', '', '', '', '', 7, '地域维度', 'ACTIVE', 'admin', NOW(), 'admin', NOW()),
|
||||
(1, 'kuaishou_ecommerce', 'dropshipping_analysis', 'city_name', '城市', 'VARCHAR', 'STRING', 'DIMENSION', false, true, true, true, '', '[]'::jsonb, '["=","IN","LIKE"]'::jsonb, '', '', '', '', '', 8, '地域维度', 'ACTIVE', 'admin', NOW(), 'admin', NOW()),
|
||||
(1, 'kuaishou_ecommerce', 'dropshipping_analysis', 'order_type', '订单类型', 'INT', 'STRING', 'DIMENSION', false, true, true, true, '', '[]'::jsonb, '["=","IN"]'::jsonb, '', '', '', '', '', 9, '订单维度', 'ACTIVE', 'admin', NOW(), 'admin', NOW()),
|
||||
(1, 'kuaishou_ecommerce', 'dropshipping_analysis', 'refund_status', '售后状态', 'INT', 'STRING', 'DIMENSION', false, true, true, true, '', '[]'::jsonb, '["=","IN"]'::jsonb, '', '', '', '', '', 10, '售后维度', 'ACTIVE', 'admin', NOW(), 'admin', NOW()),
|
||||
|
||||
-- 指标字段
|
||||
(1, 'kuaishou_ecommerce', 'dropshipping_analysis', 'order_count', '订单数', 'INT', 'STRING', 'INDICATOR', true, false, true, true, 'COUNT', '["COUNT","SUM"]'::jsonb, '[]'::jsonb, '1', 'CALCULATED', '#,##0', '单', '', 21, '订单指标', 'ACTIVE', 'admin', NOW(), 'admin', NOW()),
|
||||
(1, 'kuaishou_ecommerce', 'dropshipping_analysis', 'factory_count', '厂家数', 'INT', 'STRING', 'INDICATOR', true, false, true, true, 'COUNT', '["COUNT"]'::jsonb, '[]'::jsonb, 'COUNT(DISTINCT factory_code)', 'CALCULATED', '#,##0', '个', '', 22, '厂家指标', 'ACTIVE', 'admin', NOW(), 'admin', NOW()),
|
||||
(1, 'kuaishou_ecommerce', 'dropshipping_analysis', 'pending_allocate_count', '待分配数', 'INT', 'STRING', 'INDICATOR', true, false, true, true, 'SUM', '["SUM"]'::jsonb, '[]'::jsonb, 'CASE WHEN dropshipping_status = 1 THEN 1 ELSE 0 END', 'CALCULATED', '#,##0', '单', '', 23, '订单指标', 'ACTIVE', 'admin', NOW(), 'admin', NOW())
|
||||
ON CONFLICT (tenant_id, business_code, report_code, field_code) DO NOTHING;
|
||||
|
||||
|
||||
-- =============================================
|
||||
-- 4. 抽取配置 (report_extract_config)
|
||||
-- =============================================
|
||||
|
||||
-- ----------------------------------------------------------
|
||||
-- 4.1 订单分析 - AGGREGATE 聚合抽取
|
||||
-- 源: kuaishou_order_list → 目标: kuaishou_order_stat
|
||||
-- 按 createTime 转 stat_date，GROUP BY 维度字段，聚合指标
|
||||
-- ----------------------------------------------------------
|
||||
INSERT INTO report_extract_config (tenant_id, business_code, report_code, extract_code, extract_name, source_table_name, source_table_alias, target_table_name, is_enabled, extract_type, extract_mode, extract_key_field, extract_key_format, group_by_fields, filter_expression, join_configs, field_mappings, transform_rules, batch_size, status, creator, created_at, updater, updated_at) VALUES
|
||||
(1, 'kuaishou_ecommerce', 'order_analysis', 'order_daily_stat', '订单每日统计抽取',
|
||||
'kuaishou_order_list', 'o',
|
||||
'kuaishou_order_stat', true, 'INCREMENTAL', 'AGGREGATE',
|
||||
'updateTime', 'yyyy-MM-dd',
|
||||
'["stat_date","oid","status","province","city","district","pay_type","pay_channel","channel","activity_type","buyer_nick","item_title"]'::jsonb,
|
||||
'',
|
||||
'[]'::jsonb,
|
||||
'[
|
||||
{"sourceField":"stat_date","targetField":"stat_date","fieldType":"DATE"},
|
||||
{"sourceField":"oid","targetField":"oid","fieldType":"BIGINT","defaultValue":0},
|
||||
{"sourceField":"status","targetField":"status","fieldType":"INT","defaultValue":0},
|
||||
{"sourceField":"province","targetField":"province","fieldType":"VARCHAR"},
|
||||
{"sourceField":"city","targetField":"city","fieldType":"VARCHAR"},
|
||||
{"sourceField":"district","targetField":"district","fieldType":"VARCHAR"},
|
||||
{"sourceField":"payType","targetField":"pay_type","fieldType":"INT","defaultValue":0},
|
||||
{"sourceField":"payChannel","targetField":"pay_channel","fieldType":"VARCHAR"},
|
||||
{"sourceField":"channel","targetField":"channel","fieldType":"VARCHAR"},
|
||||
{"sourceField":"activityType","targetField":"activity_type","fieldType":"INT","defaultValue":0},
|
||||
{"sourceField":"buyerNick","targetField":"buyer_nick","fieldType":"VARCHAR"},
|
||||
{"sourceField":"itemTitle","targetField":"item_title","fieldType":"VARCHAR"},
|
||||
{"sourceField":"oid","targetField":"order_count","fieldType":"INT","aggregateFunction":"COUNT","defaultValue":0},
|
||||
{"sourceField":"totalFee","targetField":"total_fee","fieldType":"BIGINT","aggregateFunction":"SUM","defaultValue":0},
|
||||
{"sourceField":"expressFee","targetField":"express_fee","fieldType":"BIGINT","aggregateFunction":"SUM","defaultValue":0},
|
||||
{"sourceField":"discountFee","targetField":"discount_fee","fieldType":"BIGINT","aggregateFunction":"SUM","defaultValue":0},
|
||||
{"sourceField":"originalPrice","targetField":"original_price","fieldType":"BIGINT","aggregateFunction":"SUM","defaultValue":0},
|
||||
{"sourceField":"num","targetField":"item_num","fieldType":"INT","aggregateFunction":"SUM","defaultValue":0}
|
||||
]'::jsonb,
|
||||
'[
|
||||
{"ruleCode":"ts_to_date","ruleType":"FORMAT","expression":"","format":"yyyy-MM-dd","sourceField":"createTime","targetField":"stat_date"}
|
||||
]'::jsonb,
|
||||
500, 'ACTIVE', 'admin', NOW(), 'admin', NOW())
|
||||
ON CONFLICT (tenant_id, business_code, report_code, extract_code) DO NOTHING;
|
||||
|
||||
|
||||
-- ----------------------------------------------------------
|
||||
-- 4.2 商品分析 - AGGREGATE 聚合抽取
|
||||
-- 源: kuaishou_item_list → 目标: kuaishou_item_stat
|
||||
-- ----------------------------------------------------------
|
||||
INSERT INTO report_extract_config (tenant_id, business_code, report_code, extract_code, extract_name, source_table_name, source_table_alias, target_table_name, is_enabled, extract_type, extract_mode, extract_key_field, extract_key_format, group_by_fields, filter_expression, join_configs, field_mappings, transform_rules, batch_size, status, creator, created_at, updater, updated_at) VALUES
|
||||
(1, 'kuaishou_ecommerce', 'item_analysis', 'item_daily_stat', '商品每日统计抽取',
|
||||
'kuaishou_item_list', 'i',
|
||||
'kuaishou_item_stat', true, 'INCREMENTAL', 'AGGREGATE',
|
||||
'updateTime', 'yyyy-MM-dd',
|
||||
'["stat_date","kwai_item_id","title","category_id","category_name","item_type","shelf_status","audit_status"]'::jsonb,
|
||||
'',
|
||||
'[]'::jsonb,
|
||||
'[
|
||||
{"sourceField":"stat_date","targetField":"stat_date","fieldType":"DATE"},
|
||||
{"sourceField":"kwaiItemId","targetField":"kwai_item_id","fieldType":"BIGINT","defaultValue":0},
|
||||
{"sourceField":"title","targetField":"title","fieldType":"VARCHAR"},
|
||||
{"sourceField":"categoryId","targetField":"category_id","fieldType":"INT","defaultValue":0},
|
||||
{"sourceField":"categoryName","targetField":"category_name","fieldType":"VARCHAR"},
|
||||
{"sourceField":"itemType","targetField":"item_type","fieldType":"INT","defaultValue":0},
|
||||
{"sourceField":"shelfStatus","targetField":"shelf_status","fieldType":"INT","defaultValue":0},
|
||||
{"sourceField":"auditStatus","targetField":"audit_status","fieldType":"INT","defaultValue":0},
|
||||
{"sourceField":"volume","targetField":"sales_volume","fieldType":"INT","aggregateFunction":"SUM","defaultValue":0},
|
||||
{"sourceField":"price","targetField":"price","fieldType":"BIGINT","aggregateFunction":"AVG","defaultValue":0},
|
||||
{"sourceField":"kwaiItemId","targetField":"sales_amount","fieldType":"BIGINT","aggregateFunction":"SUM","defaultValue":0,
|
||||
"transformRule":{"ruleCode":"volume_times_price","ruleType":"CALCULATE","expression":"volume * price"}}
|
||||
]'::jsonb,
|
||||
'[
|
||||
{"ruleCode":"ts_to_date","ruleType":"FORMAT","expression":"","format":"yyyy-MM-dd","sourceField":"createTime","targetField":"stat_date"}
|
||||
]'::jsonb,
|
||||
500, 'ACTIVE', 'admin', NOW(), 'admin', NOW())
|
||||
ON CONFLICT (tenant_id, business_code, report_code, extract_code) DO NOTHING;
|
||||
|
||||
|
||||
-- ----------------------------------------------------------
|
||||
-- 4.3 售后分析 - AGGREGATE 聚合抽取
|
||||
-- 源: kuaishou_refund_list → 目标: kuaishou_refund_stat
|
||||
-- ----------------------------------------------------------
|
||||
INSERT INTO report_extract_config (tenant_id, business_code, report_code, extract_code, extract_name, source_table_name, source_table_alias, target_table_name, is_enabled, extract_type, extract_mode, extract_key_field, extract_key_format, group_by_fields, filter_expression, join_configs, field_mappings, transform_rules, batch_size, status, creator, created_at, updater, updated_at) VALUES
|
||||
(1, 'kuaishou_ecommerce', 'refund_analysis', 'refund_daily_stat', '售后每日统计抽取',
|
||||
'kuaishou_refund_list', 'r',
|
||||
'kuaishou_refund_stat', true, 'INCREMENTAL', 'AGGREGATE',
|
||||
'updateTime', 'yyyy-MM-dd',
|
||||
'["stat_date","refund_id","refund_type","refund_reason_desc","status","handling_way","negotiate_status"]'::jsonb,
|
||||
'',
|
||||
'[]'::jsonb,
|
||||
'[
|
||||
{"sourceField":"stat_date","targetField":"stat_date","fieldType":"DATE"},
|
||||
{"sourceField":"refundId","targetField":"refund_id","fieldType":"BIGINT","defaultValue":0},
|
||||
{"sourceField":"refundType","targetField":"refund_type","fieldType":"INT","defaultValue":0},
|
||||
{"sourceField":"refundReasonDesc","targetField":"refund_reason_desc","fieldType":"VARCHAR"},
|
||||
{"sourceField":"status","targetField":"status","fieldType":"INT","defaultValue":0},
|
||||
{"sourceField":"handlingWay","targetField":"handling_way","fieldType":"INT","defaultValue":0},
|
||||
{"sourceField":"negotiateStatus","targetField":"negotiate_status","fieldType":"INT","defaultValue":0},
|
||||
{"sourceField":"refundId","targetField":"refund_count","fieldType":"INT","aggregateFunction":"COUNT","defaultValue":0},
|
||||
{"sourceField":"refundFee","targetField":"refund_fee","fieldType":"BIGINT","aggregateFunction":"SUM","defaultValue":0}
|
||||
]'::jsonb,
|
||||
'[
|
||||
{"ruleCode":"ts_to_date","ruleType":"FORMAT","expression":"","format":"yyyy-MM-dd","sourceField":"createTime","targetField":"stat_date"}
|
||||
]'::jsonb,
|
||||
500, 'ACTIVE', 'admin', NOW(), 'admin', NOW())
|
||||
ON CONFLICT (tenant_id, business_code, report_code, extract_code) DO NOTHING;
|
||||
|
||||
|
||||
-- ----------------------------------------------------------
|
||||
-- 4.4 分销订单分析 - AGGREGATE 聚合抽取
|
||||
-- 源: kuaishou_cps_order_list → 目标: kuaishou_cps_order_stat
|
||||
-- ----------------------------------------------------------
|
||||
INSERT INTO report_extract_config (tenant_id, business_code, report_code, extract_code, extract_name, source_table_name, source_table_alias, target_table_name, is_enabled, extract_type, extract_mode, extract_key_field, extract_key_format, group_by_fields, filter_expression, join_configs, field_mappings, transform_rules, batch_size, status, creator, created_at, updater, updated_at) VALUES
|
||||
(1, 'kuaishou_ecommerce', 'cps_order_analysis', 'cps_order_daily_stat', '分销订单每日统计抽取',
|
||||
'kuaishou_cps_order_list', 'c',
|
||||
'kuaishou_cps_order_stat', true, 'INCREMENTAL', 'AGGREGATE',
|
||||
'updateTime', 'yyyy-MM-dd',
|
||||
'["stat_date","oid","status","distributor_name","order_channel","settlement_biz_type"]'::jsonb,
|
||||
'',
|
||||
'[]'::jsonb,
|
||||
'[
|
||||
{"sourceField":"stat_date","targetField":"stat_date","fieldType":"DATE"},
|
||||
{"sourceField":"oid","targetField":"oid","fieldType":"BIGINT","defaultValue":0},
|
||||
{"sourceField":"status","targetField":"status","fieldType":"INT","defaultValue":0},
|
||||
{"sourceField":"distributorName","targetField":"distributor_name","fieldType":"VARCHAR"},
|
||||
{"sourceField":"orderChannel","targetField":"order_channel","fieldType":"VARCHAR"},
|
||||
{"sourceField":"settlementBizType","targetField":"settlement_biz_type","fieldType":"INT","defaultValue":0},
|
||||
{"sourceField":"oid","targetField":"order_count","fieldType":"INT","aggregateFunction":"COUNT","defaultValue":0},
|
||||
{"sourceField":"totalFee","targetField":"total_fee","fieldType":"BIGINT","aggregateFunction":"SUM","defaultValue":0},
|
||||
{"sourceField":"expressFee","targetField":"express_fee","fieldType":"BIGINT","aggregateFunction":"SUM","defaultValue":0},
|
||||
{"sourceField":"estimatedIncome","targetField":"estimated_income","fieldType":"BIGINT","aggregateFunction":"SUM","defaultValue":0},
|
||||
{"sourceField":"commissionRate","targetField":"commission_rate","fieldType":"BIGINT","aggregateFunction":"AVG","defaultValue":0}
|
||||
]'::jsonb,
|
||||
'[
|
||||
{"ruleCode":"ts_to_date","ruleType":"FORMAT","expression":"","format":"yyyy-MM-dd","sourceField":"createTime","targetField":"stat_date"}
|
||||
]'::jsonb,
|
||||
500, 'ACTIVE', 'admin', NOW(), 'admin', NOW())
|
||||
ON CONFLICT (tenant_id, business_code, report_code, extract_code) DO NOTHING;
|
||||
|
||||
|
||||
-- ----------------------------------------------------------
|
||||
-- 4.5 代发订单分析 - AGGREGATE 聚合抽取
|
||||
-- 源: kuaishou_dropshipping_order_list → 目标: kuaishou_dropshipping_stat
|
||||
-- ----------------------------------------------------------
|
||||
INSERT INTO report_extract_config (tenant_id, business_code, report_code, extract_code, extract_name, source_table_name, source_table_alias, target_table_name, is_enabled, extract_type, extract_mode, extract_key_field, extract_key_format, group_by_fields, filter_expression, join_configs, field_mappings, transform_rules, batch_size, status, creator, created_at, updater, updated_at) VALUES
|
||||
(1, 'kuaishou_ecommerce', 'dropshipping_analysis', 'dropshipping_daily_stat', '代发订单每日统计抽取',
|
||||
'kuaishou_dropshipping_order_list', 'd',
|
||||
'kuaishou_dropshipping_stat', true, 'INCREMENTAL', 'AGGREGATE',
|
||||
'updateTime', 'yyyy-MM-dd',
|
||||
'["stat_date","dropshipping_order_code","order_status","dropshipping_status","factory_name","express_company_name","province_name","city_name","order_type","refund_status"]'::jsonb,
|
||||
'',
|
||||
'[]'::jsonb,
|
||||
'[
|
||||
{"sourceField":"stat_date","targetField":"stat_date","fieldType":"DATE"},
|
||||
{"sourceField":"dropshippingOrderCode","targetField":"dropshipping_order_code","fieldType":"VARCHAR"},
|
||||
{"sourceField":"orderStatus","targetField":"order_status","fieldType":"INT","defaultValue":0},
|
||||
{"sourceField":"dropshippingStatus","targetField":"dropshipping_status","fieldType":"INT","defaultValue":0},
|
||||
{"sourceField":"factoryName","targetField":"factory_name","fieldType":"VARCHAR"},
|
||||
{"sourceField":"factoryCode","targetField":"factory_code","fieldType":"VARCHAR"},
|
||||
{"sourceField":"expressCompanyName","targetField":"express_company_name","fieldType":"VARCHAR"},
|
||||
{"sourceField":"provinceName","targetField":"province_name","fieldType":"VARCHAR"},
|
||||
{"sourceField":"cityName","targetField":"city_name","fieldType":"VARCHAR"},
|
||||
{"sourceField":"orderType","targetField":"order_type","fieldType":"INT","defaultValue":0},
|
||||
{"sourceField":"refundStatus","targetField":"refund_status","fieldType":"INT","defaultValue":0},
|
||||
{"sourceField":"dropshippingOrderCode","targetField":"order_count","fieldType":"INT","aggregateFunction":"COUNT","defaultValue":0}
|
||||
]'::jsonb,
|
||||
'[
|
||||
{"ruleCode":"ts_to_date","ruleType":"FORMAT","expression":"","format":"yyyy-MM-dd","sourceField":"createTime","targetField":"stat_date"}
|
||||
]'::jsonb,
|
||||
500, 'ACTIVE', 'admin', NOW(), 'admin', NOW())
|
||||
ON CONFLICT (tenant_id, business_code, report_code, extract_code) DO NOTHING;
|
||||
|
||||
|
||||
-- =============================================
|
||||
-- 5. 使用说明
|
||||
-- =============================================
|
||||
-- 初始化后，按以下步骤使用报表系统：
|
||||
-- 1. 确保已执行 sql/seed_data_kuaishou.sql 同步源表数据
|
||||
-- 2. 在管理后台「业务管理」确认快手电商业务已加载
|
||||
-- 3. 在「报表配置」点击「自动建表」为每个报表创建统计宽表
|
||||
-- 4. 在「抽取配置」点击「执行抽取」指定 stat_date，从源表聚合数据到宽表
|
||||
-- 5. 在「数据查询」选择维度/指标/筛选条件进行查询分析
|
||||
--
|
||||
-- 关键设计原则：
|
||||
-- - sourceField 严格对齐 kuaishou_xxx_list 表的实际列名（camelCase）
|
||||
-- - targetField / field_code 使用 snake_case 保持项目内部命名一致
|
||||
-- - 金额字段源表存分(BIGINT)，报表提供衍生元字段(CALCULATED expression / 100.0)
|
||||
-- - 抽取增量字段统一用 updateTime（毫秒时间戳 BIGINT）
|
||||
-- - AGGREGATE 模式按 GROUP BY 维度聚合，非聚合字段取第一条
|
||||
Reference in New Issue
Block a user