locked
Microsoft Advanced Threat Analytics MongoDB failure RRS feed

  • Question

  • Hi! After some failure i tried to restore my ATA server from snapshot,but MOngoDB service wont start ar all, in services - 1067 error appears, and in cmd it cannot be repaired - it fails with this:

    C:\Program Files\Microsoft Advanced Threat Analytics\Center\MongoDB\bin>mongod -
    -dbpath "C:\Program Files\Microsoft Advanced Threat Analytics\Center\MongoDB\bin
    \data" --repair
    2017-11-22T17:59:46.325+0300 I CONTROL  [initandlisten] MongoDB starting : pid=3
    860 port=27017 dbpath=C:\Program Files\Microsoft Advanced Threat Analytics\Cente
    r\MongoDB\bin\data 64-bit host=MSK02MATA01
    2017-11-22T17:59:46.326+0300 I CONTROL  [initandlisten] targetMinOS: Windows 7/W
    indows Server 2008 R2
    2017-11-22T17:59:46.326+0300 I CONTROL  [initandlisten] db version v3.4.2
    2017-11-22T17:59:46.326+0300 I CONTROL  [initandlisten] git version: 3f76e40c105
    fc223b3e5aac3e20dcd026b83b38b
    2017-11-22T17:59:46.326+0300 I CONTROL  [initandlisten] allocator: tcmalloc
    2017-11-22T17:59:46.326+0300 I CONTROL  [initandlisten] modules: none
    2017-11-22T17:59:46.326+0300 I CONTROL  [initandlisten] build environment:
    2017-11-22T17:59:46.326+0300 I CONTROL  [initandlisten]     distmod: 2008plus
    2017-11-22T17:59:46.327+0300 I CONTROL  [initandlisten]     distarch: x86_64
    2017-11-22T17:59:46.327+0300 I CONTROL  [initandlisten]     target_arch: x86_64
    2017-11-22T17:59:46.327+0300 I CONTROL  [initandlisten] options: { repair: true,
     storage: { dbPath: "C:\Program Files\Microsoft Advanced Threat Analytics\Center
    \MongoDB\bin\data" } }
    2017-11-22T17:59:46.327+0300 I -        [initandlisten] Detected data files in C
    :\Program Files\Microsoft Advanced Threat Analytics\Center\MongoDB\bin\data crea
    ted by the 'wiredTiger' storage engine, so setting the active storage engine to
    'wiredTiger'.
    2017-11-22T17:59:46.328+0300 I STORAGE  [initandlisten] Detected WT journal file
    s.  Running recovery from last checkpoint.
    2017-11-22T17:59:46.329+0300 I STORAGE  [initandlisten] journal to nojournal tra
    nsition config: create,cache_size=24063M,session_max=20000,eviction=(threads_max
    =4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=jour
    nal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60
    ,log_size=2GB),statistics_log=(wait=0),
    2017-11-22T17:59:46.344+0300 E STORAGE  [initandlisten] WiredTiger error (0) [15
    11362786:343919][3860:140721647718528], file:WiredTiger.wt, connection: WiredTig
    er.turtle: encountered an illegal file format or internal value
    2017-11-22T17:59:46.345+0300 E STORAGE  [initandlisten] WiredTiger error (-31804
    ) [1511362786:344920][3860:140721647718528], file:WiredTiger.wt, connection: the
     process must exit and restart: WT_PANIC: WiredTiger library panic
    2017-11-22T17:59:46.345+0300 I -        [initandlisten] Fatal Assertion 28558 at
     src\mongo\db\storage\wiredtiger\wiredtiger_util.cpp 361
    2017-11-22T17:59:46.346+0300 I -        [initandlisten]

    ***aborting after fassert() failure


    2017-11-22T17:59:46.449+0300 I CONTROL  [initandlisten] mongod.exe    ...\src\mo
    ngo\util\stacktrace_windows.cpp(239)                     mongo::printStackTrace+
    0x43
    2017-11-22T17:59:46.450+0300 I CONTROL  [initandlisten] mongod.exe    ...\src\mo
    ngo\util\signal_handlers_synchronous.cpp(180)            mongo::`anonymous names
    pace'::printSignalAndBacktrace+0x73
    2017-11-22T17:59:46.450+0300 I CONTROL  [initandlisten] mongod.exe    ...\src\mo
    ngo\util\signal_handlers_synchronous.cpp(236)            mongo::`anonymous names
    pace'::abruptQuit+0x83
    2017-11-22T17:59:46.451+0300 I CONTROL  [initandlisten] mongod.exe    d:\th\mink
    ernel\crts\ucrt\src\appcrt\misc\signal.cpp(516)          raise+0x25e
    2017-11-22T17:59:46.451+0300 I CONTROL  [initandlisten] mongod.exe    d:\th\mink
    ernel\crts\ucrt\src\appcrt\startup\abort.cpp(71)         abort+0x18
    2017-11-22T17:59:46.452+0300 I CONTROL  [initandlisten] mongod.exe    ...\src\mo
    ngo\util\assert_util.cpp(172)                            mongo::fassertFailedWit
    hLocation+0x189
    2017-11-22T17:59:46.452+0300 I CONTROL  [initandlisten] mongod.exe    ...\src\mo
    ngo\db\storage\wiredtiger\wiredtiger_util.cpp(361)       mongo::`anonymous names
    pace'::mdb_handle_error+0x186
    2017-11-22T17:59:46.453+0300 I CONTROL  [initandlisten] mongod.exe    ...\src\th
    ird_party\wiredtiger\src\support\err.c(291)              __wt_eventv+0x3b8
    2017-11-22T17:59:46.453+0300 I CONTROL  [initandlisten] mongod.exe    ...\src\th
    ird_party\wiredtiger\src\support\err.c(318)              __wt_err+0x32
    2017-11-22T17:59:46.454+0300 I CONTROL  [initandlisten] mongod.exe    ...\src\th
    ird_party\wiredtiger\src\support\err.c(531)              __wt_illegal_value+0x5e

    2017-11-22T17:59:46.455+0300 I CONTROL  [initandlisten] mongod.exe    ...\src\th
    ird_party\wiredtiger\src\meta\meta_turtle.c(273)         __wt_turtle_read+0x1f1
    2017-11-22T17:59:46.455+0300 I CONTROL  [initandlisten] mongod.exe    ...\src\th
    ird_party\wiredtiger\src\meta\meta_table.c(272)          __wt_metadata_search+0x
    289
    2017-11-22T17:59:46.456+0300 I CONTROL  [initandlisten] mongod.exe    ...\src\th
    ird_party\wiredtiger\src\conn\conn_dhandle.c(261)        __conn_btree_config_set
    +0x22
    2017-11-22T17:59:46.456+0300 I CONTROL  [initandlisten] mongod.exe    ...\src\th
    ird_party\wiredtiger\src\conn\conn_dhandle.c(328)        __wt_conn_btree_open+0x
    59
    2017-11-22T17:59:46.457+0300 I CONTROL  [initandlisten] mongod.exe    ...\src\th
    ird_party\wiredtiger\src\session\session_dhandle.c(523)  __wt_session_get_btree+
    0xa1
    2017-11-22T17:59:46.458+0300 I CONTROL  [initandlisten] mongod.exe    ...\src\th
    ird_party\wiredtiger\src\session\session_dhandle.c(514)  __wt_session_get_btree+
    0x226
    2017-11-22T17:59:46.458+0300 I CONTROL  [initandlisten] mongod.exe    ...\src\th
    ird_party\wiredtiger\src\session\session_dhandle.c(336)  __wt_session_get_btree_
    ckpt+0xc3
    2017-11-22T17:59:46.459+0300 I CONTROL  [initandlisten] mongod.exe    ...\src\th
    ird_party\wiredtiger\src\cursor\cur_file.c(566)          __wt_curfile_open+0x1d9

    2017-11-22T17:59:46.459+0300 I CONTROL  [initandlisten] mongod.exe    ...\src\th
    ird_party\wiredtiger\src\session\session_api.c(391)      __session_open_cursor_i
    nt+0x2ef
    2017-11-22T17:59:46.460+0300 I CONTROL  [initandlisten] mongod.exe    ...\src\th
    ird_party\wiredtiger\src\session\session_api.c(446)      __wt_open_cursor+0x1b
    2017-11-22T17:59:46.460+0300 I CONTROL  [initandlisten] mongod.exe    ...\src\th
    ird_party\wiredtiger\src\meta\meta_table.c(94)           __wt_metadata_cursor+0x
    99
    2017-11-22T17:59:46.461+0300 I CONTROL  [initandlisten] mongod.exe    ...\src\th
    ird_party\wiredtiger\src\conn\conn_api.c(2443)           wiredtiger_open+0xb19
    2017-11-22T17:59:46.461+0300 I CONTROL  [initandlisten] mongod.exe    ...\src\mo
    ngo\db\storage\wiredtiger\wiredtiger_kv_engine.cpp(247)  mongo::WiredTigerKVEngi
    ne::WiredTigerKVEngine+0x7c9
    2017-11-22T17:59:46.462+0300 I CONTROL  [initandlisten] mongod.exe    ...\src\mo
    ngo\db\storage\wiredtiger\wiredtiger_init.cpp(91)        mongo::`anonymous names
    pace'::WiredTigerFactory::create+0x12a
    2017-11-22T17:59:46.463+0300 I CONTROL  [initandlisten] mongod.exe    ...\src\mo
    ngo\db\service_context_d.cpp(202)                        mongo::ServiceContextMo
    ngoD::initializeGlobalStorageEngine+0x5cb
    2017-11-22T17:59:46.464+0300 I CONTROL  [initandlisten] mongod.exe    ...\src\mo
    ngo\db\db.cpp(590)                                       mongo::`anonymous names
    pace'::_initAndListen+0x709
    2017-11-22T17:59:46.464+0300 I CONTROL  [initandlisten] mongod.exe    ...\src\mo
    ngo\db\db.cpp(830)                                       mongo::`anonymous names
    pace'::initAndListen+0x27
    2017-11-22T17:59:46.465+0300 I CONTROL  [initandlisten] mongod.exe    ...\src\mo
    ngo\db\db.cpp(1180)                                      mongoDbMain+0x25c
    2017-11-22T17:59:46.465+0300 I CONTROL  [initandlisten] mongod.exe    ...\src\mo
    ngo\db\db.cpp(868)                                       wmain+0x35
    2017-11-22T17:59:46.466+0300 I CONTROL  [initandlisten] mongod.exe    f:\dd\vcto
    ols\crt\vcstartup\src\startup\exe_common.inl(255)        __scrt_common_main_seh+
    0x124
    2017-11-22T17:59:46.466+0300 I CONTROL  [initandlisten] KERNEL32.DLL
                                                             BaseThreadInitThunk+0x2
    2
    2017-11-22T17:59:46.467+0300 F -        [initandlisten] Got signal: 22 (SIGABRT)
    .
    2017-11-22T17:59:46.467+0300 I CONTROL  [initandlisten] *** unhandled exception
    0x0000000E at 0x00007FFC4DE495FC, terminating
    2017-11-22T17:59:46.468+0300 I CONTROL  [initandlisten] *** stack trace for unha
    ndled exception:
    2017-11-22T17:59:46.485+0300 I CONTROL  [initandlisten] KERNELBASE.dll
                                                               RaiseException+0x6c
    2017-11-22T17:59:46.486+0300 I CONTROL  [initandlisten] mongod.exe      ...\src\
    mongo\util\signal_handlers_synchronous.cpp(237)            mongo::`anonymous nam
    espace'::abruptQuit+0x9b
    2017-11-22T17:59:46.486+0300 I CONTROL  [initandlisten] mongod.exe      d:\th\mi
    nkernel\crts\ucrt\src\appcrt\misc\signal.cpp(516)          raise+0x25e
    2017-11-22T17:59:46.487+0300 I CONTROL  [initandlisten] mongod.exe      d:\th\mi
    nkernel\crts\ucrt\src\appcrt\startup\abort.cpp(71)         abort+0x18
    2017-11-22T17:59:46.487+0300 I CONTROL  [initandlisten] mongod.exe      ...\src\
    mongo\util\assert_util.cpp(172)                            mongo::fassertFailedW
    ithLocation+0x189
    2017-11-22T17:59:46.488+0300 I CONTROL  [initandlisten] mongod.exe      ...\src\
    mongo\db\storage\wiredtiger\wiredtiger_util.cpp(361)       mongo::`anonymous nam
    espace'::mdb_handle_error+0x186
    2017-11-22T17:59:46.488+0300 I CONTROL  [initandlisten] mongod.exe      ...\src\
    third_party\wiredtiger\src\support\err.c(291)              __wt_eventv+0x3b8
    2017-11-22T17:59:46.489+0300 I CONTROL  [initandlisten] mongod.exe      ...\src\
    third_party\wiredtiger\src\support\err.c(318)              __wt_err+0x32
    2017-11-22T17:59:46.490+0300 I CONTROL  [initandlisten] mongod.exe      ...\src\
    third_party\wiredtiger\src\support\err.c(531)              __wt_illegal_value+0x
    5e
    2017-11-22T17:59:46.490+0300 I CONTROL  [initandlisten] mongod.exe      ...\src\
    third_party\wiredtiger\src\meta\meta_turtle.c(273)         __wt_turtle_read+0x1f
    1
    2017-11-22T17:59:46.491+0300 I CONTROL  [initandlisten] mongod.exe      ...\src\
    third_party\wiredtiger\src\meta\meta_table.c(272)          __wt_metadata_search+
    0x289
    2017-11-22T17:59:46.493+0300 I CONTROL  [initandlisten] mongod.exe      ...\src\
    third_party\wiredtiger\src\conn\conn_dhandle.c(261)        __conn_btree_config_s
    et+0x22
    2017-11-22T17:59:46.493+0300 I CONTROL  [initandlisten] mongod.exe      ...\src\
    third_party\wiredtiger\src\conn\conn_dhandle.c(328)        __wt_conn_btree_open+
    0x59
    2017-11-22T17:59:46.494+0300 I CONTROL  [initandlisten] mongod.exe      ...\src\
    third_party\wiredtiger\src\session\session_dhandle.c(523)  __wt_session_get_btre
    e+0xa1
    2017-11-22T17:59:46.494+0300 I CONTROL  [initandlisten] mongod.exe      ...\src\
    third_party\wiredtiger\src\session\session_dhandle.c(514)  __wt_session_get_btre
    e+0x226
    2017-11-22T17:59:46.495+0300 I CONTROL  [initandlisten] mongod.exe      ...\src\
    third_party\wiredtiger\src\session\session_dhandle.c(336)  __wt_session_get_btre
    e_ckpt+0xc3
    2017-11-22T17:59:46.496+0300 I CONTROL  [initandlisten] mongod.exe      ...\src\
    third_party\wiredtiger\src\cursor\cur_file.c(566)          __wt_curfile_open+0x1
    d9
    2017-11-22T17:59:46.497+0300 I CONTROL  [initandlisten] mongod.exe      ...\src\
    third_party\wiredtiger\src\session\session_api.c(391)      __session_open_cursor
    _int+0x2ef
    2017-11-22T17:59:46.497+0300 I CONTROL  [initandlisten] mongod.exe      ...\src\
    third_party\wiredtiger\src\session\session_api.c(446)      __wt_open_cursor+0x1b

    2017-11-22T17:59:46.498+0300 I CONTROL  [initandlisten] mongod.exe      ...\src\
    third_party\wiredtiger\src\meta\meta_table.c(94)           __wt_metadata_cursor+
    0x99
    2017-11-22T17:59:46.499+0300 I CONTROL  [initandlisten] mongod.exe      ...\src\
    third_party\wiredtiger\src\conn\conn_api.c(2443)           wiredtiger_open+0xb19

    2017-11-22T17:59:46.500+0300 I CONTROL  [initandlisten] mongod.exe      ...\src\
    mongo\db\storage\wiredtiger\wiredtiger_kv_engine.cpp(247)  mongo::WiredTigerKVEn
    gine::WiredTigerKVEngine+0x7c9
    2017-11-22T17:59:46.501+0300 I CONTROL  [initandlisten] mongod.exe      ...\src\
    mongo\db\storage\wiredtiger\wiredtiger_init.cpp(91)        mongo::`anonymous nam
    espace'::WiredTigerFactory::create+0x12a
    2017-11-22T17:59:46.501+0300 I CONTROL  [initandlisten] mongod.exe      ...\src\
    mongo\db\service_context_d.cpp(202)                        mongo::ServiceContext
    MongoD::initializeGlobalStorageEngine+0x5cb
    2017-11-22T17:59:46.502+0300 I CONTROL  [initandlisten] mongod.exe      ...\src\
    mongo\db\db.cpp(590)                                       mongo::`anonymous nam
    espace'::_initAndListen+0x709
    2017-11-22T17:59:46.503+0300 I CONTROL  [initandlisten] mongod.exe      ...\src\
    mongo\db\db.cpp(830)                                       mongo::`anonymous nam
    espace'::initAndListen+0x27
    2017-11-22T17:59:46.503+0300 I CONTROL  [initandlisten] mongod.exe      ...\src\
    mongo\db\db.cpp(1180)                                      mongoDbMain+0x25c
    2017-11-22T17:59:46.504+0300 I CONTROL  [initandlisten] mongod.exe      ...\src\
    mongo\db\db.cpp(868)                                       wmain+0x35
    2017-11-22T17:59:46.504+0300 I CONTROL  [initandlisten] mongod.exe      f:\dd\vc
    tools\crt\vcstartup\src\startup\exe_common.inl(255)        __scrt_common_main_se
    h+0x124
    2017-11-22T17:59:46.505+0300 I CONTROL  [initandlisten] KERNEL32.DLL
                                                               BaseThreadInitThunk+0
    x22
    2017-11-22T17:59:46.505+0300 I -        [initandlisten]
    2017-11-22T17:59:46.506+0300 I CONTROL  [initandlisten] writing minidump diagnos
    tic file C:\Program Files\Microsoft Advanced Threat Analytics\Center\MongoDB\bin
    \mongod.2017-11-22T14-59-46.mdmp
    2017-11-22T17:59:46.536+0300 I CONTROL  [initandlisten] *** immediate exit due t
    o unhandled exception

    C:\Program Files\Microsoft Advanced Threat Analytics\Center\MongoDB\bin>

    Any suggestions? 

    Wednesday, November 22, 2017 3:01 PM

All replies

  • Was the snapshot of the vm taken while mongo is stopped or running?
    Wednesday, November 22, 2017 8:12 PM
  • Mongo was running and i think problem occures because of that. Backup was done without stopping service. 

    And i dont have any other backups 

    Thursday, November 23, 2017 1:17 PM
  • That must be it, mongo does not work well when sharpshooting the disk while it's running.
    Thursday, November 23, 2017 1:54 PM
  • I know that, but backup was taken without stopping Mongo,so this broken snapsho my only option. How can i repair it? Official docs useless in this situation
    Friday, November 24, 2017 8:03 AM
  • Did you keep the json file & certificate copy as the ATA disaster recovery guide says?

    If Yes, you can recover the center with the configuration , without having to reinstall the gateways.

    as taking a snapshot of mongo while it's running is not officially supported, I am not sure if anything can be done here.

    You can contact MS support services, and they might be able to consult mongo support as "best effort" to help.

    Friday, November 24, 2017 10:08 AM
  • If i had json and certificate - it wouldn be a problem, but i dont have one. Our backup system specialist relais only on snapshots. And mongo support didnt replay on issue ticket 
    Friday, November 24, 2017 11:45 AM
  • You can try to open an ATA support ticket via Microsoft support for best effort help.

    MS can escalate to mongo, they might not find a solution, but they usually manage to get answers from mongo.

    Friday, November 24, 2017 8:53 PM