none
Log Collector RRS feed

  • Question

  •  We are developing a solution that collects logs from 8 domain controllers - Security event logs. roughly 5000 events are generated on each DC per 5 mins. Single service on one central server can do that.

    option1: separate thread to read events from each DC using EvtSubscribe API call with pull method and then to write them in common database structure like stack or queue and then another thread to write them to one master SQL table. [end searching will be simpler for me as all events in same table]

    option2: we can have separate table for each DC and in application have separate thread to use EvtSubscribe API call with pull method and write to corresponding table. [seems faster as splitting table each independent to other will allow writes in parallel, however end searching will require a concat/join of some sort].

     Need opinion on what should be approach ideal for such a solution.


    Shahid Roofi

    Sunday, November 17, 2019 7:11 PM

Answers

  • I would use option 1 writing to a shared queue. For the inserts, consider a single thread that reads the queued events and inserts in batches using a bulk insert method.


    Dan Guzman, Data Platform MVP, http://www.dbdelta.com

    • Marked as answer by Shahid Roofi Monday, November 18, 2019 8:50 AM
    Sunday, November 17, 2019 9:18 PM
    Moderator

All replies

  • I would use option 1 writing to a shared queue. For the inserts, consider a single thread that reads the queued events and inserts in batches using a bulk insert method.


    Dan Guzman, Data Platform MVP, http://www.dbdelta.com

    • Marked as answer by Shahid Roofi Monday, November 18, 2019 8:50 AM
    Sunday, November 17, 2019 9:18 PM
    Moderator
  • I would use option 1 writing to a shared queue. For the inserts, consider a single thread that reads the queued events and inserts in batches using a bulk insert method.


    Dan Guzman, Data Platform MVP, http://www.dbdelta.com

          Make all sense.

    Shahid Roofi

    Monday, November 18, 2019 8:51 AM