283 lines
4.5 KiB
Markdown
283 lines
4.5 KiB
Markdown
# aliyun-aps-sync
|
||
|
||
Node 版阿里云 APS 同步工具。
|
||
|
||
当前主流程已经统一为：
|
||
|
||
- Playwright 抓取 APS 页面
|
||
- 本地保留 `current/history/delta/checkpoints/errors` 数据
|
||
- 同步过程中直接写入 MySQL
|
||
- 定时任务默认执行日增量
|
||
|
||
## 同步范围
|
||
|
||
- customers
|
||
- customerDetails
|
||
- orders
|
||
- orderDetails
|
||
- bills
|
||
- messages
|
||
|
||
## 模式说明
|
||
|
||
### Full 模式
|
||
|
||
执行：
|
||
|
||
```bash
|
||
npm run sync
|
||
```
|
||
|
||
如果要让 full sync 从已有 checkpoint 继续（当前主要覆盖 orders + bills）：
|
||
|
||
```bash
|
||
npm run sync -- --resume
|
||
```
|
||
|
||
行为：
|
||
|
||
- 抓全量 customer + customerDetails
|
||
- 抓 orders / orderDetails / bills / messages
|
||
- 同步过程中直接写数据库
|
||
|
||
### Incremental 模式
|
||
|
||
执行：
|
||
|
||
```bash
|
||
npm run incremental
|
||
```
|
||
|
||
行为：
|
||
|
||
- 不抓 customer
|
||
- 抓 orders / orderDetails / bills / messages
|
||
- 以数据库 watermark + overlap 为增量窗口
|
||
|
||
## 登录
|
||
|
||
```bash
|
||
npm run login
|
||
```
|
||
|
||
会自动验证：
|
||
|
||
- 我的客户
|
||
- 账单查询
|
||
|
||
并保存登录态到：
|
||
|
||
- `.browser/`
|
||
- `.browser/storage-state.json`
|
||
|
||
## 账单
|
||
|
||
### 单独抓账单
|
||
|
||
```bash
|
||
npm run bills
|
||
```
|
||
|
||
### 从最新 checkpoint 继续抓账单
|
||
|
||
```bash
|
||
npm run bills -- --resume
|
||
```
|
||
|
||
## 订单
|
||
|
||
只同步订单：
|
||
|
||
```bash
|
||
npm run orders
|
||
```
|
||
|
||
订单增量：
|
||
|
||
```bash
|
||
npm run orders -- --incremental
|
||
```
|
||
|
||
订单从 checkpoint 继续：
|
||
|
||
```bash
|
||
npm run orders -- --resume
|
||
```
|
||
|
||
## 消息
|
||
|
||
单独抓消息：
|
||
|
||
```bash
|
||
npm run messages
|
||
```
|
||
|
||
## 定时任务
|
||
|
||
```bash
|
||
npm run schedule
|
||
```
|
||
|
||
默认按 `.env` 中：
|
||
|
||
```env
|
||
ALIYUN_APS_SCHEDULE_MODE=incremental
|
||
```
|
||
|
||
执行日增量。
|
||
|
||
## 增量窗口
|
||
|
||
### orders
|
||
|
||
由数据库中 `aps_order.order_time` 最大值决定，回退：
|
||
|
||
```env
|
||
ALIYUN_APS_ORDER_INCREMENTAL_OVERLAP_DAYS=2
|
||
```
|
||
|
||
### bills
|
||
|
||
由数据库中 `aps_bill.consumption_time` 最大值决定，回退：
|
||
|
||
```env
|
||
ALIYUN_APS_BILL_INCREMENTAL_OVERLAP_DAYS=7
|
||
```
|
||
|
||
### messages
|
||
|
||
由数据库中 `aliyun_aps_messages.gmt_modified/gmt_created` 最大值决定，回退：
|
||
|
||
```env
|
||
ALIYUN_APS_MESSAGE_INCREMENTAL_OVERLAP_DAYS=7
|
||
```
|
||
|
||
## 数据库配置
|
||
|
||
`.env` 需要配置：
|
||
|
||
```env
|
||
ALIYUN_APS_DB_HOST=
|
||
ALIYUN_APS_DB_PORT=3306
|
||
ALIYUN_APS_DB_USER=
|
||
ALIYUN_APS_DB_PASSWORD=
|
||
ALIYUN_APS_DB_NAME=
|
||
ALIYUN_APS_DB_CHARSET=utf8mb4
|
||
ALIYUN_APS_DB_CONNECTION_LIMIT=5
|
||
```
|
||
|
||
## 浏览器配置
|
||
|
||
默认不再强制使用 Google Chrome。
|
||
|
||
可选配置：
|
||
|
||
```env
|
||
ALIYUN_APS_BROWSER_MODE=launch
|
||
ALIYUN_APS_BROWSER_CHANNEL=
|
||
ALIYUN_APS_BROWSER_EXECUTABLE_PATH=
|
||
ALIYUN_APS_CDP_URL=http://127.0.0.1:9222
|
||
```
|
||
|
||
说明：
|
||
|
||
- `ALIYUN_APS_BROWSER_MODE=launch`：由 Playwright 自己启动浏览器。
|
||
- `ALIYUN_APS_BROWSER_MODE=cdp`：附着到你手动打开的 Chrome/Edge。
|
||
- 两项都留空：使用 Playwright 自带 Chromium。
|
||
- `ALIYUN_APS_BROWSER_CHANNEL=chrome`：使用本机 Chrome。
|
||
- `ALIYUN_APS_BROWSER_CHANNEL=msedge`：使用本机 Edge。
|
||
- `ALIYUN_APS_BROWSER_EXECUTABLE_PATH=...`：指定本地浏览器可执行文件路径。
|
||
|
||
### 手动打开 Chrome 后再让脚本附着
|
||
|
||
如果阿里云风控要求你手动过滑块，可以改成：
|
||
|
||
```env
|
||
ALIYUN_APS_BROWSER_MODE=cdp
|
||
ALIYUN_APS_CDP_URL=http://127.0.0.1:9222
|
||
```
|
||
|
||
然后你手动启动浏览器：
|
||
|
||
```powershell
|
||
chrome.exe --remote-debugging-port=9222 --user-data-dir="C:\temp\aps-manual-profile"
|
||
```
|
||
|
||
在浏览器里手动登录并过验证码后，再执行：
|
||
|
||
```bash
|
||
npm run sync
|
||
```
|
||
|
||
或：
|
||
|
||
```bash
|
||
npm run bills -- --resume
|
||
```
|
||
|
||
附着模式下脚本不会自动关闭你手动打开的浏览器。
|
||
|
||
## 邮件告警
|
||
|
||
任意运行异常会尝试：
|
||
|
||
- 保存错误上下文 JSON
|
||
- 截图当前页面
|
||
- 发送告警邮件
|
||
|
||
`.env` 配置：
|
||
|
||
```env
|
||
ALIYUN_APS_SMTP_HOST=
|
||
ALIYUN_APS_SMTP_PORT=465
|
||
ALIYUN_APS_SMTP_SECURE=true
|
||
ALIYUN_APS_SMTP_USER=
|
||
ALIYUN_APS_SMTP_PASS=
|
||
ALIYUN_APS_NOTIFY_EMAIL=
|
||
```
|
||
|
||
错误文件目录：
|
||
|
||
```text
|
||
data/errors/<dataset>/
|
||
```
|
||
|
||
## 本地数据目录
|
||
|
||
```text
|
||
data/current/
|
||
data/history/
|
||
data/delta/
|
||
data/checkpoints/
|
||
data/runs/
|
||
data/errors/
|
||
```
|
||
|
||
## customer 状态规则
|
||
|
||
- full 抓到 customer 时，默认写为 `active=1`
|
||
- messages 中如果明确识别到“释放”，则标记：
|
||
- `active=0`
|
||
- `status='released'`
|
||
- messages 中如果明确识别到“关联/报备成功/新增客户/绑定客户”，则恢复：
|
||
- `active=1`
|
||
- `status='active'`
|
||
|
||
## 安装
|
||
|
||
```bash
|
||
npm install
|
||
```
|
||
|
||
## 运行时热键
|
||
|
||
- `F7` 暂停
|
||
- `F8` 继续
|
||
- `F9` 终止
|
||
|
||
## 说明
|
||
|
||
- Python 入库脚本已不再是主流程依赖。
|
||
- bills 仍保留 checkpoint/resume 能力。
|
||
- messages 当前先按列表分页抓取，如后续页面需要详情抓取，再补 detail flow。
|