数据5分钟同步一次修改
This commit is contained in:
@@ -54,6 +54,28 @@ npm run incremental
|
||||
- 抓 orders / orderDetails / bills / messages
|
||||
- 以数据库 watermark + overlap 为增量窗口
|
||||
|
||||
### Hot 模式
|
||||
|
||||
执行:
|
||||
|
||||
```bash
|
||||
npm run hot
|
||||
```
|
||||
|
||||
行为:
|
||||
|
||||
- 每次只抓**当天订单**
|
||||
- 从订单第一页开始扫描
|
||||
- 订单列表按“连续稳定行 / 连续稳定页 / 最大页数”提前停止
|
||||
- 订单详情只抓:新增订单、列表有变化订单、缺失详情订单、非终态且到达兜底刷新时间的订单
|
||||
- 消息按数据库最新时间回退分钟 overlap 后抓取,并在旧页提前停止
|
||||
|
||||
适用场景:
|
||||
|
||||
- 白天高频追当天订单
|
||||
- 订单量较大,不希望每 5 分钟重复扫完整个当天分页
|
||||
- 需要兼顾详情完整性和抓取效率
|
||||
|
||||
## 登录
|
||||
|
||||
```bash
|
||||
@@ -117,6 +139,26 @@ npm run orders -- --resume
|
||||
npm run messages
|
||||
```
|
||||
|
||||
## 高频同步
|
||||
|
||||
手动执行一次高频同步:
|
||||
|
||||
```bash
|
||||
npm run hot
|
||||
```
|
||||
|
||||
如果 PowerShell 禁止 `npm.ps1`,可以直接执行:
|
||||
|
||||
```bash
|
||||
node src/index.js hot
|
||||
```
|
||||
|
||||
说明:
|
||||
|
||||
- `hot` 只覆盖当天订单、订单详情、消息
|
||||
- 不抓 customer / customerDetails / bills
|
||||
- 适合作为工作时间内的高频轮询任务
|
||||
|
||||
## 定时任务
|
||||
|
||||
```bash
|
||||
@@ -131,6 +173,26 @@ ALIYUN_APS_SCHEDULE_MODE=incremental
|
||||
|
||||
执行日增量。
|
||||
|
||||
如果要执行 5 分钟高频同步,可以设置:
|
||||
|
||||
```env
|
||||
ALIYUN_APS_SCHEDULE_MODE=hot
|
||||
ALIYUN_APS_HOT_CRON=*/5 * * * *
|
||||
```
|
||||
|
||||
然后执行:
|
||||
|
||||
```bash
|
||||
npm run schedule
|
||||
```
|
||||
|
||||
说明:
|
||||
|
||||
- `incremental`:按现有增量策略抓 orders / orderDetails / bills / messages
|
||||
- `full`:按全量策略执行
|
||||
- `hot`:每轮只抓当天 orders / orderDetails / messages
|
||||
- hot 模式内置任务锁;如果上一轮还没结束,会跳过下一轮,避免重叠执行
|
||||
|
||||
## 增量窗口
|
||||
|
||||
### orders
|
||||
@@ -157,6 +219,41 @@ ALIYUN_APS_BILL_INCREMENTAL_OVERLAP_DAYS=7
|
||||
ALIYUN_APS_MESSAGE_INCREMENTAL_OVERLAP_DAYS=7
|
||||
```
|
||||
|
||||
## 高频模式配置
|
||||
|
||||
推荐配置:
|
||||
|
||||
```env
|
||||
ALIYUN_APS_SCHEDULE_MODE=hot
|
||||
ALIYUN_APS_HOT_CRON=*/5 * * * *
|
||||
ALIYUN_APS_HOT_MESSAGE_OVERLAP_MINUTES=15
|
||||
ALIYUN_APS_HOT_ORDER_STABLE_THRESHOLD=100
|
||||
ALIYUN_APS_HOT_ORDER_STABLE_PAGE_THRESHOLD=2
|
||||
ALIYUN_APS_HOT_ORDER_MAX_PAGES=20
|
||||
ALIYUN_APS_HOT_MESSAGE_MAX_PAGES=10
|
||||
ALIYUN_APS_HOT_ORDER_DETAIL_REFRESH_MINUTES=30
|
||||
ALIYUN_APS_HOT_FINAL_STATUSES=已完成,已关闭,已取消,已退款完成
|
||||
```
|
||||
|
||||
含义:
|
||||
|
||||
- `ALIYUN_APS_HOT_CRON`:高频任务 cron,默认每 5 分钟一次
|
||||
- `ALIYUN_APS_HOT_MESSAGE_OVERLAP_MINUTES`:消息高频模式的回扫分钟数
|
||||
- `ALIYUN_APS_HOT_ORDER_STABLE_THRESHOLD`:订单扫描中连续多少条稳定记录后停止
|
||||
- `ALIYUN_APS_HOT_ORDER_STABLE_PAGE_THRESHOLD`:订单扫描中连续多少页无新增/变更后停止
|
||||
- `ALIYUN_APS_HOT_ORDER_MAX_PAGES`:订单每轮最多扫描页数,防止高峰期跑太久
|
||||
- `ALIYUN_APS_HOT_MESSAGE_MAX_PAGES`:消息每轮最多扫描页数
|
||||
- `ALIYUN_APS_HOT_ORDER_DETAIL_REFRESH_MINUTES`:非终态订单详情兜底刷新间隔
|
||||
- `ALIYUN_APS_HOT_FINAL_STATUSES`:视为终态的订单状态,终态订单在无变化时会尽量跳过详情抓取
|
||||
|
||||
默认策略:
|
||||
|
||||
- 订单按最新到最旧扫描
|
||||
- 新订单或列表字段变化的订单会进入详情抓取
|
||||
- 已抓过且无变化的终态订单会直接跳过详情
|
||||
- 非终态订单会按兜底刷新时间周期性重抓详情
|
||||
- 消息使用 watermark + overlap,避免 5 分钟轮询时漏边界消息
|
||||
|
||||
## 数据库配置
|
||||
|
||||
`.env` 需要配置:
|
||||
|
||||
Reference in New Issue
Block a user