I am running a replica set in production without sharding. All nodes are
running latest stable mongo 2.6 except one hidden node which has
mongo 3.0 with wiredTiger enabled.
*The data*I have around 4TB worth of data on each node (MMAPv1), with close
databases. The hidden node has 2TB of data because of wiredTiger.
*The plan*I decided to upgrade to 3.2 and as an intermediate step, I have
to upgrade to
3.0 first. So to start with that, I added the aforementioned hidden member
the existing replica set. I started sending prod like read query traffic to
node to check if it will be able to withstand that much load. I did this
The plan was to roll out 3.0 on all secondaries if latencies and rps are
to prod like pattern.
*The observation*As expected, wiredTiger ended up creating huge number of
files, the number is
close to 600k, with number of open files at any given time close to 150k.
node could handle upto 4k queries per second, but I saw a drop in traffic,
4k RPS to 0 (zero) after every 4 to 5 minutes. This behaviour was observed
over a week, consistently. At the same time CPU/Memory/Disk are doing just
In the process of understanding what exactly is happening, I ran
on that node where I found following.
Here you can see two operations are "in progress". First one is a query,
is "waiting for lock" for whooping 255 seconds, whereas another is a "repl
writer worker thread", which is again running for 255 seconds and holding
Global, Database and Collection level lock with "w" (Intent Exclusive) mode.
This behaviour is reflected in the graph below.
"In replication, MongoDB does not apply writes serially to secondaries.
Secondaries collect oplog entries in batches and then apply those batches in
parallel. Secondaries do not allow reads while applying the write
and apply write operations in the order that they appear in the oplog."
That explains why lock is present, but I am yet to understand why is it
more than 4 minutes to finish.
I would appreciate if you guide me in the right direction to understand what
exactly is happening here.
I am attaching some more wiredTiger stats charts. Let me know if more
information is required.
You received this message because you are subscribed to the Google Groups "mongodb-user"