小内存如何对两个大型列表求差集？

[ioven]

求 a 与 b 的差集，内存限制 1G

# 500w
a = [...] 

# 5000w
b = [...]

# 记录为 10 - 100 不等字符串

不知是不是关键词不对，搜索到的方案都是 set、numpy，或 set(b) 遍历 a，内存实在扛不住啊

求指点，速度慢些也可以接受

[clino]

@ioven 如果 sqlite 操作都放在一个事务里面,估计时间优化得比较短

[ioven]

@clino 多谢提醒，默认是智能事务，试试手工开启事务看速度有没有提升

[shamashii]

生成 110s，比较 120s,实验时感觉坑点竟然在于生成随机字符串效率,求改进
```
import timeit
def main():
import h5py, cyrandom
allchr = "".join((chr(i) for i in range(33,127)))
pspool = [[cyrandom.choice(allchr) for _ in range(cyrandom.randint(10, 100))] for x in range(100000)]

chunkl = []
for _ in range(5000000):
b1 = cyrandom.choice(pspool)
cyrandom.shuffle(b1)
chunkl.append(''.join(b1).encode('utf-8'))

f = h5py.File('h5.h5','w')
for k in range(50000000//5000000):
l = [str(k).encode('utf-8')]
# cyrandom.shuffle(chunkl)
print(k)
f.create_dataset(str(k), data=chunkl+l,)
del chunkl
f.close()

def query():
import h5py
f = h5py.File('h5.h5','a')
wbw = set(f['0'].value)
count = []
for k in f.keys():
print(k)
for x in f[k].value:
if x not in wbw:
count.append(x)
print(count)
f.close()

print(timeit.timeit(main, number=1))
print(timeit.timeit(query, number=1))
```

[shamashii]

sigh``
https://pastebin.com/uDna4GK9