architecture

2021-05-15T02:47:21.083900Z

Diffing with the previous results? But where do you keep those stored? Anyways, probably since I don't know all the details, just thought that could be something to consider, since S3 can be used to list all already downloaded documents, it can also solve this dedupe issue if you didn't have another strategy for it