electron 使用 web worker 上传批量的文件(7W+)报 Maximum call stack size exceeded 错误

2020-04-29 10:56:37 +08:00
 TomeWong

web worker 从主进程获取到上传文件的路径,在 onmessage 处理上传,通过 fs.promises.stat 获取文件的信息,通过 postMessage 回传至 worker 主进程中,在 worker 主进程中,将这些数据更新至 vuex 中,出现了下面的问题,小数量级的文件,不会出现这个问题

  (node:33176) UnhandledPromiseRejectionWarning: RangeError: Maximum call stack size exceeded
      at Array.mutator (D:\pc-win\SKYD-win\node_modules\vue\dist\vue.runtime.common.dev.js:870:27)
      at Store.addMultiTaskUL (webpack:///./src/renderer/store/modules/task.js?:164:72)
      at wrappedMutationHandler (D:\pc-win\SKYD-win\node_modules\vuex\dist\vuex.common.js:727:13)
      at commitIterator (D:\pc-win\SKYD-win\node_modules\vuex\dist\vuex.common.js:393:7)
      at Array.forEach (<anonymous>)
      at D:\pc-win\SKYD-win\node_modules\vuex\dist\vuex.common.js:392:11
      at Store._withCommit (D:\pc-win\SKYD-win\node_modules\vuex\dist\vuex.common.js:523:3)
      at Store.commit (D:\pc-win\SKYD-win\node_modules\vuex\dist\vuex.common.js:391:8)
      at Store.boundCommit [as commit] (D:\pc-win\SKYD-win\node_modules\vuex\dist\vuex.common.js:336:19)
      at local.commit (D:\pc-win\SKYD-win\node_modules\vuex\dist\vuex.common.js:681:13)
      at eval (webpack:///./src/renderer/store/modules/task.js?:476:7)
      at new Promise (<anonymous>)
      at new F (webpack:///./node_modules/core-js/library/modules/_export.js?:36:28)
      at Store.addMultiTaskUL (webpack:///./src/renderer/store/modules/task.js?:440:12)
      at Array.wrappedActionHandler (D:\pc-win\SKYD-win\node_modules\vuex\dist\vuex.common.js:734:23)
      at Store.dispatch (D:\pc-win\SKYD-win\node_modules\vuex\dist\vuex.common.js:439:15)
  (node:33176) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 10)
1696 次点击
所在节点    程序员
10 条回复
TomeWong
2020-04-29 11:10:42 +08:00
```
self.onmessage = async event => {
console.log(event)
const { diskType, files, dirpath, sid, usn, currentFileId, currentGroupId, currentCreatedByUsn, baseUrl, uploadLimit, uploadSpeed } = event.data
const queueData = [],
taskDataArr = []
let completeStatus
for (let elem of files) {
const filePath = elem,
// stats = await fs.promises.stat(filePath).catch(error => console.log(error)),
// size = stats.size,
chunkSize = 5 * 1024 * 1024, //每片分块的大小 5M
// pieces = Math.ceil(size / chunkSize),
uploadType = pieces == 1 ? 1 : 3,
num = filePath.lastIndexOf('\\') + 1,
name = filePath.substring(num),
uuid = uuidv4(),
basename = dirpath ? path.basename(dirpath) : '',
realPath = dirpath ? filePath.substring(filePath.indexOf(basename), filePath.lastIndexOf("\\")).replace(/\\/g, "/") : ''
const payload = {
appFileId: "",
diskType: diskType,
uploadType: uploadType,
creatorUsn: currentCreatedByUsn || usn,
parentid: currentFileId == -2 ? -1 : currentFileId || -1,
groupId: diskType == 1 ? '' : currentGroupId,
fileSize: 0,
fileName: name,
fileMd5: '',
fileRealPath: realPath,
comeFrom: 30,
model: 0,
discussContent: ""
}
const queueItemData = { sid, uuid, payload, name, size, currentFileId, uploadType, pieces, chunkSize, filePath, currentCreatedByUsn, baseUrl, uploadLimit, uploadSpeed }
// taskDataArr.push({ filePath, name, uuid, size, pieces, uploadType, diskType, parentid: currentFileId == -2 ? -1 : currentFileId || -1, fileRealPath: realPath })
taskDataArr.push({ filePath, name, uuid, size, pieces, uploadType, diskType, parentid: currentFileId == -2 ? -1 : currentFileId || -1, fileRealPath: realPath })
queueData.push(queueItemData)

}
completeStatus = {
status: 'updateStore'
}
self.postMessage({taskDataArr, completeStatus})
}
```
TomeWong
2020-04-29 11:23:57 +08:00
是否转换字符串来传输,而不是数组对象
wednesdayco
2020-04-29 13:59:28 +08:00
分……片……
TomeWong
2020-04-29 14:27:13 +08:00
@wednesdayco 现在都是小文件,可以正常上传,但将 web worker 中处理的信息回传至 worker 主进程,更新至 vuex 有问题,vuex 是不是存在存储大小限制的问题
Vegetable
2020-04-29 14:30:01 +08:00
显然这是递归层数过多
你是一个一个上传, 一个文件成功之后, 回调里传下一个吗?
Vegetable
2020-04-29 14:32:31 +08:00
let deep = 1

function c() {
deep++
c()
}

try {
c()
} catch (error) {
console.log(error)
//RangeError: Maximum call stack size exceeded
console.log(deep)
//15712
}
TomeWong
2020-04-29 14:39:58 +08:00
@Vegetable 是一个一个上传的,并发量为 3,上传是正常的,在上传前会先根据每条文件的路径获取一些信息,然后再将这些信息通过 postMessage 回传至 worker 主进程,在 worker 主进程将这些处理的信息更新至 vuex 中,这个更新过程出现了问题
VDimos
2020-04-29 14:40:39 +08:00
你这递归层级太多了吧,做下尾递归优化,不要保留递归栈。
MrYELiex
2020-04-29 14:41:00 +08:00
错误信息很明显 调用栈过多
TomeWong
2020-04-29 15:11:08 +08:00
@VDimos 在 vuex 中使用 map 来处理,可能会有问题

这是一个专为移动设备优化的页面(即为了让你能够在 Google 搜索结果里秒开这个页面),如果你希望参与 V2EX 社区的讨论,你可以继续到 V2EX 上打开本讨论主题的完整版本。

https://www.v2ex.com/t/667204

V2EX 是创意工作者们的社区,是一个分享自己正在做的有趣事物、交流想法,可以遇见新朋友甚至新机会的地方。

V2EX is a community of developers, designers and creative people.

© 2021 V2EX