none
New Users Can't Create Roaming Profiles at Logon

    Question

  • 2016/12/18 malware hits customer network and destroys tons of data and multiple servers – including one domain controller and all of the group policies.

    2016/12/18 – 2016/27  I restore or rebuild from scratch multiple servers, restore user data, domain controller, GPO’s etc.

    2017/01/03 – Customer is hit again with new strain of similar malware, although updated antivirus had been installed, a user with elevated privileges removed it.

    2017/01/04 – 2017/01/08  I again rebuild servers and GPO’s, and restore data, and setup NetLogon debugging, as well as file/folder auditing.

    • No more outbreaks


    Pre-existing users who have been configured to connect to their roaming profiles continue to do so.

    Any new user created cannot populate new roaming user profiles on existing roaming profile share. All new users get the common “User profile cannot be loaded”.

    I have duplicated the same AD / file server / security group / user account relationship in my lab and it works perfectly – in fact difficult to break.

    I have followed exactly these articles:
    https://technet.microsoft.com/en-us/library/jj649079(v=ws.11).aspx
    http://www.mcbsys.com/blog/2010/10/reset-roaming-profile-and-folder-redirection-permissions/
    http://jeffgraves.me/2013/
    http://www.grouppolicy.biz/2010/08/best-practice-roaming-profiles-and-folder-redirection-a-k-a-user-virtualization/

    All new users are in exactly the same security groups as the pre-existing users. All new users attempt to connect/create roaming user profiles within the same file server / folder.

    To isolate, I moved to brand new workstations, brand new file server, brand new accounts – to rule out anything corrupt on the previous objects. Same exact results.
    The new builds were fresh ISO builds, not templates.

    I have run packet captures in my lab (where roaming profiles work) and 2 on customer network (one where pre-existing users succeed, where new users fail) and compared them. The successful logons look identical except for a DFS query (DFS is not running so this is curious). The failed logons don’t even appear to try and create the roaming profile folder:

    Some of the test workstations were physical machines, some were virtual machines – but I made sure to use new builds of both.
    Servers have to be virtual for now.

    Checked VMware guest tools, VMware guest versions, NIC types and drivers, etc. All are legit.

    Of note:  same users that fail the roaming profile logon can logon (without roaming) and map a drive to the  _EXACT_  location of where their roaming profiles  _should_  be. So permissions don’t seem to be an issue.
    I’ve turned up debugging and auditing and the only profile related errors are:
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    Log Name:      Application
    Source:        Microsoft-Windows-Winlogon
    Date:          3/6/2017 4:02:51 PM
    Event ID:      6004
    Task Category: None
    Level:         Warning
    Keywords:      Classic
    User:          N/A
    Computer:      winr2stg.domain.local
    Description:
    The winlogon notification subscriber <Profiles> failed a critical notification event.
    Event Xml:
    <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
      <System>
        <Provider Name="Microsoft-Windows-Winlogon" Guid="{DBE9B383-7CF3-4331-91CC-A3CB16A3B538}" EventSourceName="Wlclntfy" />
        <EventID Qualifiers="32768">6004</EventID>
        <Version>0</Version>
        <Level>3</Level>
        <Task>0</Task>
        <Opcode>0</Opcode>
        <Keywords>0x80000000000000</Keywords>
        <TimeCreated SystemTime="2017-03-07T00:02:51.000000000Z" />
        <EventRecordID>1476</EventRecordID>
        <Correlation />
        <Execution ProcessID="0" ThreadID="0" />
        <Channel>Application</Channel>
        <Computer>winr2stg.domain.local</Computer>
        <Security />
      </System>
      <EventData>
        <Data>Profiles</Data>
        <Binary>F4010000</Binary>
      </EventData>
    </Event>


    Log Name:      Application
    Source:        Microsoft-Windows-User Profiles Service
    Date:          3/6/2017 4:02:51 PM
    Event ID:      1520
    Task Category: None
    Level:         Error
    Keywords:     
    User:          domain\pac.man
    Computer:      winr2stg.domain.local
    Description:
    Windows cannot log you on because your roaming mandatory profile is not available. This error may be caused by incorrect file system permissions or network problems.

    DETAIL - The system cannot find the file specified.

    Event Xml:
    <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
      <System>
        <Provider Name="Microsoft-Windows-User Profiles Service" Guid="{89B1E9F0-5AFF-44A6-9B44-0A07A7CE5845}" />
        <EventID>1520</EventID>
        <Version>0</Version>
        <Level>2</Level>
        <Task>0</Task>
        <Opcode>0</Opcode>
        <Keywords>0x8000000000000000</Keywords>
        <TimeCreated SystemTime="2017-03-07T00:02:51.817433400Z" />
        <EventRecordID>1477</EventRecordID>
        <Correlation />
        <Execution ProcessID="788" ThreadID="1328" />
        <Channel>Application</Channel>
        <Computer>winr2stg.domain.local</Computer>
        <Security UserID="S-1-5-21-2901645698-1785784430-4210207855-17122" />
      </System>
      <EventData>
        <Data Name="Error">The system cannot find the file specified.
    </Data>
      </EventData>
    </Event>


    Log Name:      Application
    Source:        Microsoft-Windows-User Profiles Service
    Date:          3/6/2017 4:02:51 PM
    Event ID:      1500
    Task Category: None
    Level:         Error
    Keywords:     
    User:          domain\pac.man
    Computer:      winr2stg.domain.local
    Description:
    Windows cannot log you on because your profile cannot be loaded. Check that you are connected to the network, and that your network is functioning correctly.

    DETAIL - The system cannot find the file specified.

    Event Xml:
    <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
      <System>
        <Provider Name="Microsoft-Windows-User Profiles Service" Guid="{89B1E9F0-5AFF-44A6-9B44-0A07A7CE5845}" />
        <EventID>1500</EventID>
        <Version>0</Version>
        <Level>2</Level>
        <Task>0</Task>
        <Opcode>0</Opcode>
        <Keywords>0x8000000000000000</Keywords>
        <TimeCreated SystemTime="2017-03-07T00:02:51.817433400Z" />
        <EventRecordID>1478</EventRecordID>
        <Correlation />
        <Execution ProcessID="788" ThreadID="1328" />
        <Channel>Application</Channel>
        <Computer>winr2stg.domain.local</Computer>
        <Security UserID="S-1-5-21-2901645698-1785784430-4210207855-17122" />
      </System>
      <EventData>
        <Data Name="Error">The system cannot find the file specified.
    </Data>
      </EventData>
    </Event>


    Log Name:      Application
    Source:        Microsoft-Windows-Winlogon
    Date:          3/6/2017 4:02:59 PM
    Event ID:      6001
    Task Category: None
    Level:         Warning
    Keywords:      Classic
    User:          N/A
    Computer:      winr2stg.domain.local
    Description:
    The winlogon notification subscriber <Sens> failed a notification event.
    Event Xml:
    <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
      <System>
        <Provider Name="Microsoft-Windows-Winlogon" Guid="{DBE9B383-7CF3-4331-91CC-A3CB16A3B538}" EventSourceName="Wlclntfy" />
        <EventID Qualifiers="32768">6001</EventID>
        <Version>0</Version>
        <Level>3</Level>
        <Task>0</Task>
        <Opcode>0</Opcode>
        <Keywords>0x80000000000000</Keywords>
        <TimeCreated SystemTime="2017-03-07T00:02:59.000000000Z" />
        <EventRecordID>1479</EventRecordID>
        <Correlation />
        <Execution ProcessID="0" ThreadID="0" />
        <Channel>Application</Channel>
        <Computer>winr2stg.domain.local</Computer>
        <Security />
      </System>
      <EventData>
        <Data>Sens</Data>
        <Binary>F0030000</Binary>
      </EventData>
    </Event>
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

    I also had other IT staff double-check my work and they agree it aligns with best practice.

    I’ve done this many times in the past and never had this kind of issue before.

    Thanks in advance for anyone’s insight,

    Tuesday, March 7, 2017 6:52 AM

All replies

  • Have you considered that this might be an undesired side effect of your new AV implementation? If so, maybe AV-specific logs might reveal something helpful

    hth
    Marcin

    Tuesday, March 7, 2017 12:33 PM
  • I have considered that. I turned off all A/V and ran the same tests again - same results. Also, this same AV was running prior to these issues, albeit the AV fell behind on signatures for a while. Also, the logs don't show any problems at all. Thanks for your reply.
    Tuesday, March 7, 2017 2:00 PM
  • That's a challenging one. If the packet captures don't reveal anything conclusive, you might want to use Process Monitor to look at the local registry/file system on both the client (where the user is logging on) and the server (where the roaming profiles are hosted) - and trying to correlate this with the behavior you are seeing in the lab.

    You might want to also consider trying Microsoft Message Analyzer (if you haven't yet) available at https://www.microsoft.com/en-us/download/details.aspx?id=44226 which simplifies analysis of system logs/traces from different sources

    hth
    Marcin

    Tuesday, March 7, 2017 2:23 PM
  • SOLVED.

    What started all of this was a new user whose last name was "Man". Since our naming convention in Active Directory is "firstname.lastname" - you can see where this is going. A username that ends in ".man" - when configured to have a roaming profile - will attempt to auto-create a roaming profile folder that also ends in .man - which will trick the system into thinking it's a mandatory profile, which is supposed to be read-only. I researched this for a while and couldn't actually find any article or mention to _NOT_ mix usernames that end in ".man" with roaming profiles - but the proof is in the practice. Changing the username to literally anything else got it to work. Note that this was a username change, not a display name, or SID, or anything else.

    Cheers,

    Wednesday, March 8, 2017 5:07 AM
  • Hi,
    Great share and update, could you help to mark it as answer? It will be greatly helpful to others who have the same question. Appreciate for your feedback.
    Best regards,
    Wendy

    Please remember to mark the replies as answers if they help.
    If you have feedback for TechNet Subscriber Support, contact tnmff@microsoft.com

    Thursday, March 9, 2017 1:39 AM
    Moderator