Intermittent Stack Trace Causing MongoDB to Crash

From: John A <zylo47@xxxxxxxxx>
To: mongodb-user <mongodb-user@xxxxxxxxxxxxxxxx>
Date: Tue, 3 May 2016 07:31:13 -0700 (PDT)
Why ads?
We have a production 4 node replica set (1 hidden, non-voting data bearing 
member) running MongoDB v2.6.12.  We see an intermittent issue happening 
for the past 1-2 months where 1 of the 3 voting members just crashes.  When 
the crash happens there is a stack trace in the MongoDB log file:

2016-05-03T01:04:55.211-0400 [conn28306]  authenticate db: local { 
authenticate: 1, nonce: "xxx", user: "__system", key: "xxx" }
2016-05-03T01:04:55.273-0400 [journal] mongod.exe    ...\src\mongo\util\
stacktrace.cpp(169)                                      mongo::
printStackTrace+0x43
2016-05-03T01:04:55.273-0400 [journal] mongod.exe    ...\src\mongo\util\
signal_handlers.cpp(107)                                 mongo::`anonymous 
namespace'::abruptQuit+0xf2
2016-05-03T01:04:55.273-0400 [journal] mongod.exe   
 f:\dd\vctools\crt_bld\self_64_amd64\crt\src\winsig.c(593)                 
  raise+0x1ed
2016-05-03T01:04:55.273-0400 [journal] mongod.exe   
 f:\dd\vctools\crt_bld\self_64_amd64\crt\src\abort.c(81)                   
  abort+0x18
2016-05-03T01:04:55.273-0400 [journal] mongod.exe   
 ...\src\mongo\db\instance.cpp(342)                                         
 mongo::mongoAbort+0x6e
2016-05-03T01:04:55.273-0400 [journal] mongod.exe   
 ...\src\mongo\db\dur.cpp(647)                                             
  `mongo::dur::groupCommitWithLimitedLocks'::`1'::catch$0+0x97
2016-05-03T01:04:55.273-0400 [journal] mongod.exe    f:\dd\vctools\crt_bld\
SELF_64_amd64\crt\prebuild\eh\amd64\handlers.asm(44)  _CallSettingFrame+0x20
2016-05-03T01:04:55.273-0400 [journal] mongod.exe    f:\dd\vctools\crt_bld\
self_64_amd64\crt\prebuild\eh\frame.cpp(1337)         __CxxCallCatchBlock+
0xeb
2016-05-03T01:04:55.273-0400 [journal] ntdll.dll                           
                                                      RtlCaptureContext+
0x3c3
2016-05-03T01:04:55.273-0400 [journal] mongod.exe    ...\src\mongo\db\dur.
cpp(643)                                               mongo::dur::
groupCommitWithLimitedLocks+0x27
2016-05-03T01:04:55.273-0400 [journal] mongod.exe    ...\src\mongo\db\dur.
cpp(783)                                               mongo::dur::
durThreadGroupCommit+0x6d
2016-05-03T01:04:55.273-0400 [journal] mongod.exe    ...\src\mongo\db\dur.
cpp(855)                                               mongo::dur::durThread
+0x27a
2016-05-03T01:04:55.273-0400 [journal] mongod.exe    ...\src\third_party\
boost\libs\thread\src\win32\thread.cpp(185)             boost::`anonymous 
namespace'::thread_start_function+0x21
2016-05-03T01:04:55.273-0400 [journal] mongod.exe   
 f:\dd\vctools\crt_bld\self_64_amd64\crt\src\threadex.c(314)               
  _callthreadstartex+0x17
2016-05-03T01:04:55.273-0400 [journal] mongod.exe   
 f:\dd\vctools\crt_bld\self_64_amd64\crt\src\threadex.c(292)               
  _threadstartex+0x7f
2016-05-03T01:04:55.273-0400 [journal] KERNEL32.DLL                         
                                                     BaseThreadInitThunk+0xd
2016-05-03T01:04:55.273-0400 [journal] SEVERE: Got signal: 22 (SIGABRT).
Backtrace:

This is affecting our production replica set only.  I do not know how to 
proceed with troubleshooting this issue.  Can someone please advise me on 
what I can do to find the source of the problem?

-- 
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.

For other MongoDB technical support options, see: https://docs.mongodb.org/manual/support/
--- 
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user+unsubscribe@xxxxxxxxxxxxxxxx.
To post to this group, send email to mongodb-user@xxxxxxxxxxxxxxxx.
Visit this group at https://groups.google.com/group/mongodb-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/6435f3c8-ae87-438b-96ae-d24292d9a720%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Why ads?