I decided to upgrade to 3.2 and as an intermediate step, I have to upgrade
3.0 first. So to start with that, I added the aforementioned hidden member
the existing replica set. I started sending prod like read query traffic to
node to check if it will be able to withstand that much load. I did this
In general, you cannot get reliable load testing results if you are using
Secondary reads. A hidden Secondary is only different compared to a
non-hidden Secondary in their ability to be elected Primary.
A Secondary’s goal is to replicate the Primary’s oplog as quickly as it
can, so that it can take over the Primary’s job at a moment’s notice. For
this reason, a Secondary perform writes in a very different manner compared
to the Primary. This is described in the page How does concurrency affect
that you have seen, where it says “*Secondaries do not allow reads while
applying the write operations*“. Therefore, your load testing is creating a
resource contention on this Secondary, and you are seeing the effect of
this resource contention in your observations.
If you need to perform a load test, I would suggest creating a separate
replica set and routing the operations to both production set and the
testing set, which will give you a more accurate comparison.