none
Risk of Indexing AD Attributes

    Question

  • We have a client software called cisco unified personal communicator(CUPC) which search AD for each contact's phone number added into it when user log on. Since phone number attribute is not indexed in AD , the search visits every AD user objects and causes heavy load to the CPU of the DC.  I am trying to index phone number attribute in Windows 2008 R2 AD environment. I would like know if there's any risk of doing that. What is behind when indexing AD attribute? Will it increase AD database size ? Can I revert the change after index the attribute ? Since it is a schema change, I asssume I have to do forest recovery if running something horribly wrong, right? Thanks in advance !


    This posting is provided AS-IS with no warranties/guarantees and confers no rights.

    Wednesday, February 22, 2012 7:37 PM

Answers

  • Hi
    First off I would like to know what kind of query and query scope this app are using (e.g how this app are searching for the “phoneNumber” attribute, exactly which attribute are we talking about?)
     
    Determine the need for an index and select the most appreciate index type
     
    Active Directory supports several different index options for attributes defined by the Search-Flags attribute:
    1 (0x00000001): Normal Index
    Create an index for the attribute.
     
    2 (0x00000002): PDNT (Parent Container) + Indexed attribute (Sufficient while you’re doing one-level searches)
    Create an index for the attribute in each container.
     
    4 (0x00000004) (Probably not interesting for you)
    Add this attribute to the Ambiguous Name Resolution (ANR) set. This is used to assist in finding an object when only partial information is given. For example, if the LDAP filter is (ANR=JEFF), the search will find each object where the first name, last name, email address, or other ANR attribute is equal to JEFF. Bit 0 must be set for this index take affect.
     
     
    Supported beginning with Windows Server 2003. Create a tuple index for the attribute. This will improve searches where the wildcard appears at the front of the search string. For example, (sn=*mith).

    See: http://msdn.microsoft.com/en-us/library/windows/desktop/ms679765(v=vs.85).aspx for more information.
     
    I would recommend you to use the stats control and/or turn on the Field Engineering Flag on your DCs to catch inefficient/expensive queries that may qualify for being indexed.
     
    You should test the query your app are using and as peer guidelines in http://msdn.microsoft.com/en-us/library/windows/desktop/ms808539.aspx you can determine if it’s considered efficient or not, as the article states try ovoid end up in the ancestors_index.
     
    How much space dose a particular index allocates in NTDS.dit?
     
    Run esentutl /ms ntds.dit
     
    You can read this blog post for more information:
    http://blogs.msdn.com/b/brettsh/archive/2006/06/12/631516.aspx
     
    ----------------------------------------------------------
    Regards
    Christoffer Andersson – Principal Advisor
    Enfo Zipper

    "Marcin Policht" wrote in message news:12b1a085-9140-41be-9b26-1b1be1873aa4...

    Enabling indexing will have some impact on both short term utilization (during actual process of indexing) and size of DB. Quantifying each would depend on the volume of records being indexed - but , I'd not be overly concerned (note that you can easily test it by duplicating one of your DCs in a lab).

    In the long run, indexing slows down writes and speeds up reads. Considering that majority of AD related usage involves the latter, the former is rarely a consideration.

    The operation is reversible.

    Let's hope that Christoffer Anderssson will chime in... ;)

    hth
    Marcin


    Enfo Zipper Christoffer Andersson – Principal Advisor
    • Proposed as answer by Marcin PolichtMVP Thursday, February 23, 2012 11:48 AM
    • Marked as answer by Bruce-Liu Tuesday, February 28, 2012 6:34 AM
    Thursday, February 23, 2012 10:43 AM
  • Enabling indexing will have some impact on both short term utilization (during actual process of indexing) and size of DB. Quantifying each would depend on the volume of records being indexed - but , I'd not be overly concerned (note that you can easily test it by duplicating one of your DCs in a lab).

    In the long run, indexing slows down writes and speeds up reads. Considering that majority of AD related usage involves the latter, the former is rarely a consideration.

    The operation is reversible.

    Let's hope that Christoffer Anderssson will chime in... ;)

    hth
    Marcin

    • Marked as answer by KungfuPanda2 Wednesday, February 22, 2012 8:36 PM
    Wednesday, February 22, 2012 7:48 PM
  • I don’t believe there is any “risk” of indexing an attribute in AD.  However, it can increase the AD database size.  

    >>> Can I revert the change after index the attribute ?

    You can “disable” the indexing option. 

    Enabling indexing on an attribute is not going to “kill” the entire AD. 


    Santhosh Sivarajan | Houston, TX
    http://www.sivarajan.com/

    FaceBook Twitter LinkedIn SS Tech Forum

    This posting is provided AS IS with no warranties,and confers no rights.


    Wednesday, February 22, 2012 8:08 PM
  • Looks like you’re getting pretty bad results – the index used is the Ancestors_index. You’re ending up visiting the entire DIT and of course that is inefficient.
     
    Do you have any way to change the query the app is using? (Cause the query it self is by it’s nature badly written) it doesn't scope the query to any particular object class(es)

    You probably want to index the following attributes using a “Normal Index”
    • ipPhone
    • mobile
    • telephoneNumber
    Seems like the app is trying to do some kind of medial search, However at a first glance once can think that a tuple index would benefit at this situation, but by default so far I know the tuple has a min length and a max length using the following values:
    Min=3;
    Max=10;
     
    Meaning that the size of tuples in the following query is too short to fall into range:
    ”( | (ipPhone=*7*5*9*0*) (mobile=*7*5*9*0*) (telephoneNumber=*7*5*9*0*) )”
     
    I would actually consider to see if it’s possible to changed the way this app queries Active Directory, if it turns out that it can’t be changed, I would consider hosting a AD LDS instance with the specific data and target the app to that specific AD LDS instance instead of having poorly performance intensive queries running on my production AD.
     
    ----------------------------------------------------------
    Regards
    Christoffer Andersson – Principal Advisor
    Enfo Zipper

    "KungfuPanda2" wrote in message news:354dc2bf-c20c-4d78-8e02-976387854f51...

    Many thanks for your such detailed answer. Here is the result of two typical qurreis I captured from W2K8 builtin performance monitor.

    Instance:NTDS

    Scope :deep

    ObjectName:dc=corp,dc=example,dc=com

    FilterName :(ipPhone=9740)

    Index:Ancestors_index:0:N;

    Status:0

    Visisted:12,227

    Found :0

    Requests/sec :0

    ResponseTime(ms):376

    Instance:NTDS

    Scope :deep

    ObjectName:dc=corp,dc=example,dc=com

    FilterName :( | (ipPhone=*7*5*9*0*) (mobile=*7*5*9*0*) (telephoneNumber=*7*5*9*0*) )

    Index:Ancestors_index:10004:N;

    Status:0

    Visisted:12,227

    Found :46

    Requests/sec :0

    ResponseTime(ms):382


    This posting is provided AS-IS with no warranties/guarantees and confers no rights.


    Enfo Zipper Christoffer Andersson – Principal Advisor
    • Marked as answer by KungfuPanda2 Thursday, February 23, 2012 6:58 PM
    Thursday, February 23, 2012 4:08 PM

All replies

  • Hello,

    if that is only a single attribute then this shouldn't be a problem: http://technet.microsoft.com/en-us/library/cc737526(v=ws.10).aspx

    http://blogs.technet.com/b/ad/archive/2008/04/01/how-to-create-a-mosiac-of-user-thumbnails-in-aduc-dsa-msc.aspx


    Best regards

    Meinolf Weber
    MVP, MCP, MCTS
    Microsoft MVP - Directory Services
    My Blog: http://msmvps.com/blogs/mweber/

    Disclaimer: This posting is provided "AS IS" with no warranties or guarantees , and confers no rights.

    Wednesday, February 22, 2012 7:45 PM
  • Enabling indexing will have some impact on both short term utilization (during actual process of indexing) and size of DB. Quantifying each would depend on the volume of records being indexed - but , I'd not be overly concerned (note that you can easily test it by duplicating one of your DCs in a lab).

    In the long run, indexing slows down writes and speeds up reads. Considering that majority of AD related usage involves the latter, the former is rarely a consideration.

    The operation is reversible.

    Let's hope that Christoffer Anderssson will chime in... ;)

    hth
    Marcin

    • Marked as answer by KungfuPanda2 Wednesday, February 22, 2012 8:36 PM
    Wednesday, February 22, 2012 7:48 PM
  • Howdie!
     
    Am 22.02.2012 20:37, schrieb KungfuPanda2:
    > We have a client software called cisco unified personal
    > communicator(CUPC) which search AD for each contact's phone number added
    > into it when user log on. Since phone number attribute is not indexed in
    > AD , the search visits every AD user objects and causes heavy load to
    > the CPU of the DC. I am trying to index phone number attribute in
    > Windows 2008 R2 AD environment. I would like know if there's any risk of
    > doing that. What is behind when indexing AD attribute? Will it increase
    > AD database size ? Can I revert the change after index the attribute ?
    > Since it is a schema change, I asssume I have to do forest recovery if
    > running something horribly wrong, right? Thanks in advance !
     
    Creating an index will help with querying the data in a more efficient
    way on the client side but the load on the Domain Controller(s) will
    pretty much stay the same.
     
    Not knowing Cisco's product, querying AD with every logon and re-loading
    the whole list of phone numbers seems inefficient - and bad design. But
    then again, I have no clue how it is constructed and how it works.
     
    So -- what are you trying to mitigate? Load on the DC? There may be
    different things you can consider:
     
    (a) Can it be configured to use AD LDS instead of a AD DS DC?
    (b) Can you configure it to use a dedicated DC and promote a new DC and
    separate it from other duties (auth, FSMO, ...)
     
    Cheers,
    Florian
     
     

    The views and opinions expressed in my postings do NOT necessarily correlate with the ones of my friends, family or my employer. If anyone should be allowed to mark a response as an "answer", it should be the thread creator. No one else.
    Wednesday, February 22, 2012 7:54 PM
  • I don’t believe there is any “risk” of indexing an attribute in AD.  However, it can increase the AD database size.  

    >>> Can I revert the change after index the attribute ?

    You can “disable” the indexing option. 

    Enabling indexing on an attribute is not going to “kill” the entire AD. 


    Santhosh Sivarajan | Houston, TX
    http://www.sivarajan.com/

    FaceBook Twitter LinkedIn SS Tech Forum

    This posting is provided AS IS with no warranties,and confers no rights.


    Wednesday, February 22, 2012 8:08 PM
  • Thanks for the prompt reply. Acutally just tested the indexing in Lab and it was a low risk task indeed. I compared the search results before and after indexing phone number attribute. As for each search, the repsonse time was improved from 682 ms to 29 ms and CPU% dropped from 0.7% to 0.0%.   I think the indexing can help with CPU load.

    This posting is provided AS-IS with no warranties/guarantees and confers no rights.

    Wednesday, February 22, 2012 8:36 PM
  • Take a look at below reference too.

    http://blogs.technet.com/b/ad/archive/2008/04/01/how-to-create-a-mosiac-of-user-thumbnails-in-aduc-dsa-msc.aspx

     

     

    Regards

    Awinish Vishwakarma

    MY BLOG:  http://awinish.wordpress.com/


    This posting is provided AS-IS with no warranties/guarantees and confers no rights.

    Thursday, February 23, 2012 5:48 AM
    Moderator
  • Hi
    First off I would like to know what kind of query and query scope this app are using (e.g how this app are searching for the “phoneNumber” attribute, exactly which attribute are we talking about?)
     
    Determine the need for an index and select the most appreciate index type
     
    Active Directory supports several different index options for attributes defined by the Search-Flags attribute:
    1 (0x00000001): Normal Index
    Create an index for the attribute.
     
    2 (0x00000002): PDNT (Parent Container) + Indexed attribute (Sufficient while you’re doing one-level searches)
    Create an index for the attribute in each container.
     
    4 (0x00000004) (Probably not interesting for you)
    Add this attribute to the Ambiguous Name Resolution (ANR) set. This is used to assist in finding an object when only partial information is given. For example, if the LDAP filter is (ANR=JEFF), the search will find each object where the first name, last name, email address, or other ANR attribute is equal to JEFF. Bit 0 must be set for this index take affect.
     
     
    Supported beginning with Windows Server 2003. Create a tuple index for the attribute. This will improve searches where the wildcard appears at the front of the search string. For example, (sn=*mith).

    See: http://msdn.microsoft.com/en-us/library/windows/desktop/ms679765(v=vs.85).aspx for more information.
     
    I would recommend you to use the stats control and/or turn on the Field Engineering Flag on your DCs to catch inefficient/expensive queries that may qualify for being indexed.
     
    You should test the query your app are using and as peer guidelines in http://msdn.microsoft.com/en-us/library/windows/desktop/ms808539.aspx you can determine if it’s considered efficient or not, as the article states try ovoid end up in the ancestors_index.
     
    How much space dose a particular index allocates in NTDS.dit?
     
    Run esentutl /ms ntds.dit
     
    You can read this blog post for more information:
    http://blogs.msdn.com/b/brettsh/archive/2006/06/12/631516.aspx
     
    ----------------------------------------------------------
    Regards
    Christoffer Andersson – Principal Advisor
    Enfo Zipper

    "Marcin Policht" wrote in message news:12b1a085-9140-41be-9b26-1b1be1873aa4...

    Enabling indexing will have some impact on both short term utilization (during actual process of indexing) and size of DB. Quantifying each would depend on the volume of records being indexed - but , I'd not be overly concerned (note that you can easily test it by duplicating one of your DCs in a lab).

    In the long run, indexing slows down writes and speeds up reads. Considering that majority of AD related usage involves the latter, the former is rarely a consideration.

    The operation is reversible.

    Let's hope that Christoffer Anderssson will chime in... ;)

    hth
    Marcin


    Enfo Zipper Christoffer Andersson – Principal Advisor
    • Proposed as answer by Marcin PolichtMVP Thursday, February 23, 2012 11:48 AM
    • Marked as answer by Bruce-Liu Tuesday, February 28, 2012 6:34 AM
    Thursday, February 23, 2012 10:43 AM
  • That’s not true, the only thing that will replicate is the changed value of Search-Flags for the particular attribute in the Schema – Indexing takes place on each DSA/DC as soon as the DSA/DC picks up the replicated value in Search-Flags then the local DSA/DC will create it’s own unique index (The index it self or it’s content is never replicated)
     
    ----------------------------------------------------------
    Regards
    Christoffer Andersson – Principal Advisor
    Enfo Zipper

    "Santhosh Sivarajan-" wrote in message news:ff9269e3-fd9a-4698-b58d-c5cd022478a8...

    I don’t believe there is any “risk” of indexing an attribute in AD.  However, it can increase the AD database size.  Also, when you enable the indexing it will replicate information throughout the domain.  This will increase the replication traffic.

    >>> Can I revert the change after index the attribute ?

    You can “disable” the indexing option. 

    Enabling indexing on an attribute is not going to “kill” the entire AD.


    Santhosh Sivarajan | Houston, TX
    http://www.sivarajan.com/

    FaceBook Twitter LinkedIn SS Tech Forum

    This posting is provided AS IS with no warranties,and confers no rights.


    Enfo Zipper Christoffer Andersson – Principal Advisor
    Thursday, February 23, 2012 10:49 AM
  • Many thanks for your such detailed answer. Here is the result of two typical qurreis I captured from W2K8 builtin performance monitor.

    Instance:NTDS

    Scope :deep

    ObjectName:dc=corp,dc=example,dc=com

    FilterName :(ipPhone=9740)

    Index:Ancestors_index:0:N;

    Status:0

    Visisted:12,227

    Found :0

    Requests/sec :0

    ResponseTime(ms):376

    Instance:NTDS

    Scope :deep

    ObjectName:dc=corp,dc=example,dc=com

    FilterName :( | (ipPhone=*7*5*9*0*) (mobile=*7*5*9*0*) (telephoneNumber=*7*5*9*0*) )

    Index:Ancestors_index:10004:N;

    Status:0

    Visisted:12,227

    Found :46

    Requests/sec :0

    ResponseTime(ms):382


    This posting is provided AS-IS with no warranties/guarantees and confers no rights.

    Thursday, February 23, 2012 3:07 PM
  • Thanks for clarifying that Chris.  Index is local and local to the database. That makes more sense

    Thanks, 


    Santhosh Sivarajan | Houston, TX
    http://www.sivarajan.com/

    FaceBook Twitter LinkedIn SS Tech Forum

    This posting is provided AS IS with no warranties,and confers no rights.


    Thursday, February 23, 2012 3:24 PM
  • Looks like you’re getting pretty bad results – the index used is the Ancestors_index. You’re ending up visiting the entire DIT and of course that is inefficient.
     
    Do you have any way to change the query the app is using? (Cause the query it self is by it’s nature badly written) it doesn't scope the query to any particular object class(es)

    You probably want to index the following attributes using a “Normal Index”
    • ipPhone
    • mobile
    • telephoneNumber
    Seems like the app is trying to do some kind of medial search, However at a first glance once can think that a tuple index would benefit at this situation, but by default so far I know the tuple has a min length and a max length using the following values:
    Min=3;
    Max=10;
     
    Meaning that the size of tuples in the following query is too short to fall into range:
    ”( | (ipPhone=*7*5*9*0*) (mobile=*7*5*9*0*) (telephoneNumber=*7*5*9*0*) )”
     
    I would actually consider to see if it’s possible to changed the way this app queries Active Directory, if it turns out that it can’t be changed, I would consider hosting a AD LDS instance with the specific data and target the app to that specific AD LDS instance instead of having poorly performance intensive queries running on my production AD.
     
    ----------------------------------------------------------
    Regards
    Christoffer Andersson – Principal Advisor
    Enfo Zipper

    "KungfuPanda2" wrote in message news:354dc2bf-c20c-4d78-8e02-976387854f51...

    Many thanks for your such detailed answer. Here is the result of two typical qurreis I captured from W2K8 builtin performance monitor.

    Instance:NTDS

    Scope :deep

    ObjectName:dc=corp,dc=example,dc=com

    FilterName :(ipPhone=9740)

    Index:Ancestors_index:0:N;

    Status:0

    Visisted:12,227

    Found :0

    Requests/sec :0

    ResponseTime(ms):376

    Instance:NTDS

    Scope :deep

    ObjectName:dc=corp,dc=example,dc=com

    FilterName :( | (ipPhone=*7*5*9*0*) (mobile=*7*5*9*0*) (telephoneNumber=*7*5*9*0*) )

    Index:Ancestors_index:10004:N;

    Status:0

    Visisted:12,227

    Found :46

    Requests/sec :0

    ResponseTime(ms):382


    This posting is provided AS-IS with no warranties/guarantees and confers no rights.


    Enfo Zipper Christoffer Andersson – Principal Advisor
    • Marked as answer by KungfuPanda2 Thursday, February 23, 2012 6:58 PM
    Thursday, February 23, 2012 4:08 PM
  • Thanks Chris. That app is supported by another team and I will check with them. This app is actaully called Cisco Unified Personal Communicator(CUPC). Here is a PDF doc tells how to configure AD for it. It recommends to index ipphone attribute too. I am thinking to use Global Catalog query instead of standard LDAP query. We are in single domain environment which means GC NC is actually the same as domain NC if I understand correctly. Do you think if it helps if we specify CUPC to use port 3268 (GC query) instead?

    http://www.cisco.com/en/US/docs/voice_ip_comm/cups/8_0/english/install_upgrade/deployment/guide/dgactivedirconfig.pdf


    This posting is provided AS-IS with no warranties/guarantees and confers no rights.

    Thursday, February 23, 2012 4:37 PM
  • No it won’t matter/help if you’re using the LDAP or the GC port in a single domain.
     
    Seems like you can set a lot of parameters in the app:
    Default search base: Set this to the path where the object’s that contains those attributes resides.
    BaseFilter: Use the suggested one in the DOCs or more a more specific filer, obviously your installation is not.
     
    SearchBase1, SearchBase2,
    SearchBase3, SearchBase4,
    SearchBase5: Same as above but if you need to set multiple locations, this is the place.
     
    UseWildcards: Go a head and disable this

    Seems like the app support AD LDS if you want to go for that option:
    ”Connections to Active Directory Lightweight Directory Services (AD LDS) and Active Directory
    Application Mode (ADAM) servers that implement local and proxy authentication are supported” you can configure accounts to be synced between your AD DS domain and a AD LDS instance using: http://blogs.technet.com/b/efleis/archive/2005/09/23/adamsync-can-also-transform-users-in-to-proxy-users.aspx and http://blogs.msdn.com/b/jeff/archive/2007/04/01/synchronize-active-directory-to-adam-with-adamsync-step-by-step.aspx
     
    ----------------------------------------------------------
    Regards
    Christoffer Andersson – Principal Advisor
    Enfo Zipper

    "KungfuPanda2" wrote in message news:30cfe97e-b1a5-4649-ad35-e624b1145bfb...

    Thanks Chris. That app is supported by another team and I will check with them. This app is actaully called Cisco Unified Personal Communicator(CUPC). Here is a PDF doc tells how to configure AD for it. It recommends to index ipphone attribute too. I am thinking to use Global Catalog query instead of standard LDAP query. We are in single domain environment which means GC NC is actually the same as domain NC if I understand correctly. Do you think if it helps if we specify CUPC to use port 3268 (GC query) instead?

    http://www.cisco.com/en/US/docs/voice_ip_comm/cups/8_0/english/install_upgrade/deployment/guide/dgactivedirconfig.pdf


    This posting is provided AS-IS with no warranties/guarantees and confers no rights.


    Enfo Zipper Christoffer Andersson – Principal Advisor
    Thursday, February 23, 2012 5:14 PM
  • I will contact the team supporting this app to set up filters and do more efficient queries. Again, many thanks for your help !

    This posting is provided AS-IS with no warranties/guarantees and confers no rights.

    Thursday, February 23, 2012 6:58 PM
  • Hi KungfuPanda2;

    We have an issue just like your are experiencing. Could you please share the results and number of users,server specs? Do u have any solution so far?

    Thanks.

    Monday, April 30, 2012 2:56 PM