We have a nightly process running on one of our datasets.
This process primarly aggregates data and creates a in memory graph
representation of the data. This data is not critical when looking at
changes and eventual consistenacy wouldn't be any problem. The Primary
requirement is that it should be fast.
It iterates through a collection with has about 10 mio entries. the current
given structure required to have a callback called for each entry.