Big Data Aggregation Process optimization

From: "'Boas Enkler' via mongodb-user" <mongodb-user@xxxxxxxxxxxxxxxx>
To: mongodb-user <mongodb-user@xxxxxxxxxxxxxxxx>
Date: Wed, 11 May 2016 00:27:31 -0700 (PDT)
Why ads?
We have a nightly process running on one of our datasets. 

This process primarly aggregates data and creates a in memory graph 
representation of the data. This data is not critical when looking at 
changes and eventual consistenacy wouldn't be any problem. The Primary 
requirement is that it should be fast.

It iterates through a collection with has about 10 mio entries. the current 
given structure required to have a callback called for each entry.

The code curretnly looks like this:

IAsyncCursor<StoredConnectionMemorySet> cursor = await collection.FindAsync(filter.Gte(i => i.DepartureDate, minDate.Date));
              while (await cursor.MoveNextAsync())
              {
                  IEnumerable<StoredConnectionMemorySet> batch = cursor.Current;
 
                  foreach (StoredConnectionMemorySet connectionSet in batch)
                  {
                      if (connectionSet.Modified.AddDays(MaxAge) > timestamp)
                      {
                          handler(connectionSet, connectionSet.Connections.Where(x => x.IsValidConnection()));
                      }
                  }
              }


Now my question are there any performance optimiazations / setting i should 
consider?


-- 
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.

For other MongoDB technical support options, see: https://docs.mongodb.org/manual/support/
--- 
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user+unsubscribe@xxxxxxxxxxxxxxxx.
To post to this group, send email to mongodb-user@xxxxxxxxxxxxxxxx.
Visit this group at https://groups.google.com/group/mongodb-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/02d814b1-1298-4130-893a-044ecfc63920%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Why ads?