locked
x-ms-forwarded-client-ip not showing in ADFS logs! Under attack externally. RRS feed

  • Question

  • Hi clever people!

    Using exchange online with ADFS on server 2012 (850 mailboxes) and we are getting thousands of bad password attempts.

    I am trying desperately to get "x-ms-forwarded-client-ip" (the hacker/bots originating IP) to show in the ADFS logs, all I am seeing is a load of Microsoft IP addresses which is totally useless.

    Once I have the IP addresses hammering ADFS I can block them very quickly with Client Access Rules but I just can't get the originating ip's to show no matter what I do.

    I have run through both of the below and still no luck, can some clever person please help?

    https://blogs.technet.microsoft.com/pie/2016/02/02/ad-fun-services-track-down-the-source-of-adfs-lockouts/

    https://blogs.technet.microsoft.com/tspring/2017/01/20/federated-to-microsoft-cloud-and-account-lockouts/

    Friday, October 19, 2018 9:20 PM

All replies

  • First, upgrade to ADFS 2012 R2 and enable Extranet Lockout Polkicy. Or event better, to Windows Server 2016 and use the Smart Lockout Policy.

    Then, if you don't see the actual IP in the logs it is probably because you have a network device in the front of ADFS spoofing the IP. So you should configure it to either add the header or spoof the actual IP of the external client.

    Also, if those access are performed using a legacy client (POP, IMAP), you can simply disable the support of such protocols.

    If it is coming from Active Sync client, the request are proxied by EXO. So it will show both IPs in the logs (the one from the clients and the one from EXO).

    What's your case?


    Note: Posts are provided “AS IS” without warranty of any kind, either expressed or implied, including but not limited to the implied warranties of merchantability and/or fitness for a particular purpose.

    Sunday, October 21, 2018 5:38 PM
  • Sorry missed R2 out, already on that!

    So I solved the issue late last night and can now see the x-ms-forwarded-client-ip in the logs.

    Now here's the big question, there appears to be no actual way of blocking authentication attempts before they happen, please correct me if i'm wrong but the following do not do what I want:

    Extranet Lockout

    Claim Rules

    EO Client Access Rules

    Conditional Access

    From everything I have read (a lot) none of the above actually happen before an attempt is made therefore lockouts will always happen!

    What I want to do is geoip block the x-ms-forwarded-client-ip after the WAP but before it actually hits the ADFS server, in other words if someone is in say China, goes to exchange online and puts in an email address with our UPN they get redirected to our ADFS signin page but at that point it says webpage unavailable therefore they cannot trigger a bad password attempt!

    Is this possible and if so how?

    (i'm thinking something like Untangle SSL Inspector in between the WAP and ADFS servers?)

    Sunday, October 21, 2018 7:56 PM
  • Sorry missed R2 out, already on that!

    So I solved the issue late last night and can now see the x-ms-forwarded-client-ip in the logs.

    Now here's the big question, there appears to be no actual way of blocking authentication attempts before they happen, please correct me if i'm wrong but the following do not do what I want:

    Extranet Lockout

    Claim Rules

    EO Client Access Rules

    Conditional Access

    From everything I have read (a lot) none of the above actually happen before an attempt is made therefore lockouts will always happen!

    What I want to do is geoip block the x-ms-forwarded-client-ip after the WAP but before it actually hits the ADFS server, in other words if someone is in say China, goes to exchange online and puts in an email address with our UPN they get redirected to our ADFS signin page but at that point it says webpage unavailable therefore they cannot trigger a bad password attempt!

    Is this possible and if so how?

    (i'm thinking something like Untangle SSL Inspector in between the WAP and ADFS servers?)

    One potential option:

    https://blogs.technet.microsoft.com/exchange/2018/10/17/disabling-basic-authentication-in-exchange-online-public-preview-now-available/

    When using Basic Auth clients, not sure there is much you can do but be reactive otherwise . I would be curious to hear Pierre's thoughts.
    Sunday, October 21, 2018 8:02 PM
  • Attached is all the IP's that caused bad password attempts between 02.30am and 07:15am this morning for instance and MICROSOFT need to seriously sort this out as its not acceptable.  Why an earth should someone/thing be able to attempt authentication from these places if we don't want them to, we should be able to stop them even getting past the email address point..........crazy!

    Sunday, October 21, 2018 8:24 PM
  • Sunday, October 21, 2018 8:28 PM
  • You can block those with Conditional Access Policies in Azure.

    Sunday, October 21, 2018 9:17 PM
  • Conditional Access happens after authentication attempt which is too late, you still get bad password attempts and therefore lockouts!

    There must be a way to block authentication attempts beforehand?

    Sunday, October 21, 2018 9:21 PM
  • If the lockouts are the issues, then the correctly configured the built-in mechanisms will help. 

    Blocking authentication attempts is by tricky by nature as we don't know who the user is yet. Well, there are couple of actions you can do while keeping ADFS. I'll add number to ease the follow-up questions.

    1. First of, make sure our password will never be guessed with such attempts. You can do that using the Azure AD Password Protection which has a password filter you can deploy on-prem:

    • Preview: Enforce Azure AD password protection for Windows Server Active Directory https://docs.microsoft.com/en-us/azure/active-directory/authentication/concept-password-ban-bad-on-premises

    This requires a license P1 for each synchronized users. If you can't afford it, well, you can try a custom solution mixing your own password filter and a regular scan using tools such as DSInternals (not a Microsoft tool and can be seen as a hacking tools by A/V and A/M so be careful when using such tools for audit).

    2. Then, make sure you are not using legacy clients applications and disable the feature in Exchange Online (POP, IMAP, SMTP).

    3. Using ADFS 2016, you can use Azure MFA as a first factor for authentication. As a result, knowing or trying to guess the password is voided.

    4. On the same note as 3, you can use Certificate Based Authentication and disable legacy endpoint for authentication. 

    5. ADFS 2019 have a way to check the "reputation of IPs". But I ain't sure if it will be before the authentication or after. In the meantime, you can combine point 1 and the Smart Lockout Policy of ADFS 2016 (see here: https://docs.microsoft.com/en-us/windows-server/identity/ad-fs/operations/configure-ad-fs-extranet-smart-lockout-protection).

    You can also not use ADFS and have Azure AD handle the things for you. 

    6. In that case you can still achieve on-prem SSO with the feature Azure AD Connect Seamless SSO (free): https://docs.microsoft.com/en-us/azure/active-directory/hybrid/how-to-connect-sso

    7. Also, review the logs of your VPN and other point of entries on your network as they are also subjects to the same attacks that you are seeing here. 


    Note: Posts are provided “AS IS” without warranty of any kind, either expressed or implied, including but not limited to the implied warranties of merchantability and/or fitness for a particular purpose.

    Sunday, October 21, 2018 10:03 PM
  • And Microsoft as well as many tech companies are also involved in tracking and taking down botnets (such the ones hammering you now). You can read about those efforts on the Microsoft Digital Crime Unit page.

    Note: Posts are provided “AS IS” without warranty of any kind, either expressed or implied, including but not limited to the implied warranties of merchantability and/or fitness for a particular purpose.


    Sunday, October 21, 2018 10:05 PM
  • Conditional Access happens after authentication attempt which is too late, you still get bad password attempts and therefore lockouts!

    There must be a way to block authentication attempts beforehand?

    That link I posted earlier should block before the auth attempts. Havent tested it though

    https://blogs.technet.microsoft.com/exchange/2018/10/17/disabling-basic-authentication-in-exchange-online-public-preview-now-available/

    P.S. Alot of people are in the same boat. The attacks have definitely increased recently
    Sunday, October 21, 2018 10:34 PM
  • Oh, and you can also block IPs with ADFS 2016: https://docs.microsoft.com/en-us/windows-server/identity/ad-fs/operations/configure-ad-fs-banned-ip

    Note: Posts are provided “AS IS” without warranty of any kind, either expressed or implied, including but not limited to the implied warranties of merchantability and/or fitness for a particular purpose.

    Monday, October 22, 2018 3:16 PM
  • Hi Andy

    Thanks for the prize link for basic authentication, I have been working through Pierre's list and will respond to him shortly (manic day today)

    I have created the authentication policy to turn of 3 things to start with and I have managed to set the new policy as the default policy just fine.

    However I am now STUCK at the last hurdle to assign the policy to ALL existing users with what is probably a very simple powershell command but I cannot get it to work and it as per the Microsoft documentation.  Please help? (i have tried it in a .ps1, tried ($User in $Users), tried the { next to foreach and it doesn't like me...) 

    PS C:\> $Users = Get-User -ResultSize unlimited
    PS C:\> $Users | foreach {Set-User -AuthenticationPolicy "Block Basic Auth for I
    map4/Pop3/Outlook Service(win10 apps)"}

    cmdlet Set-User at command pipeline position 1
    Supply values for the following parameters:
    Identity:


    Monday, October 22, 2018 8:17 PM
  • Hi Andy

    Thanks for the prize link for basic authentication, I have been working through Pierre's list and will respond to him shortly (manic day today)

    I have created the authentication policy to turn of 3 things to start with and I have managed to set the new policy as the default policy just fine.

    However I am now STUCK at the last hurdle to assign the policy to ALL existing users with what is probably a very simple powershell command but I cannot get it to work and it as per the Microsoft documentation.  Please help? (i have tried it in a .ps1, tried ($User in $Users), tried the { next to foreach and it doesn't like me...) 

    PS C:\> $Users = Get-User -ResultSize unlimited
    PS C:\> $Users | foreach {Set-User -AuthenticationPolicy "Block Basic Auth for I
    map4/Pop3/Outlook Service(win10 apps)"}

    cmdlet Set-User at command pipeline position 1
    Supply values for the following parameters:
    Identity:


    $Users = Get-User -ResultSize unlimited

    $users =$users.WindowsEmailAddress

    $users | %{Set-User -Identity $_ -AuthenticationPolicy "Block Basic Auth for I
    map4/Pop3/Outlook Service(win10 apps)"}

    Tuesday, October 23, 2018 10:46 AM
  • You nailed it Andy David thankyou very much!

    Ended up with this to make it work if anyone else is interested:

    Create the authentication policy:

    Set-AuthenticationPolicy -Identity "Block Basic Auth for Imap4/Pop3/Outlook Service(win10 apps)" -AllowBasicAuthActiveSync -AllowBasicAuthAutodiscover -AllowBasicAuthImap:$false -AllowBasicAuthMapi -AllowBasicAuthOfflineAddressBook -AllowBasicAuthOutlookService:$false -AllowBasicAuthPop:$false -AllowBasicAuthReportingWebServices -AllowBasicAuthRest -AllowBasicAuthRpc -AllowBasicAuthSmtp -AllowBasicAuthWebServices -AllowBasicAuthPowerShell

    Configure the default authentication policy:

    Set-OrganizationConfig -DefaultAuthenticationPolicy "Block Basic Auth for Imap4/Pop3/Outlook Service(win10 apps)"

    Assign the authentication policy to users

    Individual user accounts:

    Set-User -Identity user@domain.com -AuthenticationPolicy "Block Basic Auth for Imap4/Pop3/Outlook Service(win10 apps)"

    All user accounts (my own creation with Andy Davids powershell brain help):

    $Users = Get-User -ResultSize unlimited
    $users =$users.WindowsEmailAddress
    $users | %{Set-User -Identity $_ -AuthenticationPolicy "Block Basic Auth for Imap4/Pop3/Outlook Service(win10 apps)"}

    Immediately apply the authentication policy to users within 30 minutes (despite other websites saying it's only 24 hrs):

    Individual user accounts:

    Set-User -Identity user@domain.com -STSRefreshTokensValidFrom $([System.DateTime]::UtcNow)

    All user accounts (my own creation with Andy Davids powershell brain help):

    $Users = Get-User -ResultSize unlimited
    $users =$users.WindowsEmailAddress
    $users | %{Set-User -Identity $_ -STSRefreshTokensValidFrom $([System.DateTime]::UtcNow)}

    Note: The very last command underlined above to immediately apply to all users threw some weird errors for every user as it went through but it did change the -STSRefreshTokensValidFrom on each user so I think it worked.

    To check it had assigned to users and was visible I ran this against few users:

    Get-User -Identity "Display Name" | Format-List

    Look through the output to find these entries:

    AuthenticationPolicy                : Block Basic Auth for Imap4/Pop3/Outlook Service(win10 apps)
    StsRefreshTokensValidFrom      : 23/10/2018 14:04:47

    Next thing to do is work out how to see it actually working.....should be fun!!

    Tuesday, October 23, 2018 8:50 PM
  • Forgot to say I had a sudden brain wave about the bad password attempts that still get through to ADFS if the user doesn't exist.

    What about creating an OU in AD called Blocked ADFS Usernames and create the bad username entered in a 411 in that OU, do a delta sync to get them up to Azure and then assign/apply the Basic Authentication policy so they get blocked too.

    Just a thought??

    Tuesday, October 23, 2018 8:56 PM
  • Conditional Access happens after authentication attempt which is too late, you still get bad password attempts and therefore lockouts!

    There must be a way to block authentication attempts beforehand?

    That link I posted earlier should block before the auth attempts. Havent tested it though

    https://blogs.technet.microsoft.com/exchange/2018/10/17/disabling-basic-authentication-in-exchange-online-public-preview-now-available/

    P.S. Alot of people are in the same boat. The attacks have definitely increased recently
    On the money Andy well done thankyou!  But be aware everyone there are Powershell errors in the documentation.  See further down this page...
    • Edited by KevinTalbot Tuesday, October 23, 2018 8:58 PM
    Tuesday, October 23, 2018 8:57 PM
  • If the lockouts are the issues, then the correctly configured the built-in mechanisms will help. 

    Blocking authentication attempts is by tricky by nature as we don't know who the user is yet. Well, there are couple of actions you can do while keeping ADFS. I'll add number to ease the follow-up questions.

    1. First of, make sure our password will never be guessed with such attempts. You can do that using the Azure AD Password Protection which has a password filter you can deploy on-prem:

    • Preview: Enforce Azure AD password protection for Windows Server Active Directory https://docs.microsoft.com/en-us/azure/active-directory/authentication/concept-password-ban-bad-on-premises

    This requires a license P1 for each synchronized users. If you can't afford it, well, you can try a custom solution mixing your own password filter and a regular scan using tools such as DSInternals (not a Microsoft tool and can be seen as a hacking tools by A/V and A/M so be careful when using such tools for audit).

    2. Then, make sure you are not using legacy clients applications and disable the feature in Exchange Online (POP, IMAP, SMTP).

    3. Using ADFS 2016, you can use Azure MFA as a first factor for authentication. As a result, knowing or trying to guess the password is voided.

    4. On the same note as 3, you can use Certificate Based Authentication and disable legacy endpoint for authentication. 

    5. ADFS 2019 have a way to check the "reputation of IPs". But I ain't sure if it will be before the authentication or after. In the meantime, you can combine point 1 and the Smart Lockout Policy of ADFS 2016 (see here: https://docs.microsoft.com/en-us/windows-server/identity/ad-fs/operations/configure-ad-fs-extranet-smart-lockout-protection).

    You can also not use ADFS and have Azure AD handle the things for you. 

    6. In that case you can still achieve on-prem SSO with the feature Azure AD Connect Seamless SSO (free): https://docs.microsoft.com/en-us/azure/active-directory/hybrid/how-to-connect-sso

    7. Also, review the logs of your VPN and other point of entries on your network as they are also subjects to the same attacks that you are seeing here. 


    Note: Posts are provided “AS IS” without warranty of any kind, either expressed or implied, including but not limited to the implied warranties of merchantability and/or fitness for a particular purpose.

    Hi Pierre, thankyou for the comprehensive list.  I have been gradually working my way through it point by point but got also got wrapped up in the new Disable Basic Authentication feature.

    Lockouts are/were definitely the immediate issue but the fack that the spray attack is able to create bad authentication attempts in the first place is the actual issue and all of the above only "helps" like you say.

    However, I implemented your point 2 yesterday and disable IMAP and POP3 on all mailboxes and new mailboxes and "touch wood" that appears to have stopped a great deal of lockouts on its own.  Not sure how as this is after an authentication attempt, the only other thing I noticed different today was the fact the 411 usernames being hit were staff that left 3-6 years ago.  Therefore don't exist and don't lockout.

    It's as if the Botnet is using a different list to the last few days.

    Thought of this idea earlier:

    Forgot to say I had a sudden brain wave about the bad password attempts that still get through to ADFS if the user doesn't exist.

    What about creating an OU in AD called Blocked ADFS Usernames and create the bad username entered in a 411 in that OU, do a delta sync to get them up to Azure and then assign/apply the Basic Authentication policy so they get blocked too.

    Just a thought??

    One more question, I want to turn off SMTP and Diasable Basic Authentication for SMTP.  Question is what is that going to break? Our 150 strong fleet of big Ricoh's use SMTP for scan to email so I don't really want to turn it off until I find another way of doing it?

    Tuesday, October 23, 2018 9:18 PM
  • Conditional Access happens after authentication attempt which is too late, you still get bad password attempts and therefore lockouts!

    There must be a way to block authentication attempts beforehand?

    That link I posted earlier should block before the auth attempts. Havent tested it though

    https://blogs.technet.microsoft.com/exchange/2018/10/17/disabling-basic-authentication-in-exchange-online-public-preview-now-available/

    P.S. Alot of people are in the same boat. The attacks have definitely increased recently

    On the money Andy well done thankyou!  But be aware everyone there are Powershell errors in the documentation.  See further down this page...
    Yes, saw that! Its been reported so hopefully it will get fixed soon.
    Wednesday, October 24, 2018 11:09 AM
  • Also just noticed the Default Policy although set does not get applied to new users?
    Wednesday, October 24, 2018 11:18 AM
  • One more question, I want to turn off SMTP and Diasable Basic Authentication for SMTP.  Question is what is that going to break? Our 150 strong fleet of big Ricoh's use SMTP for scan to email so I don't really want to turn it off until I find another way of doing it?

    Are they authenticating to 365? Typically those are just sent anonymously from on-prem > 365...

    Wednesday, October 24, 2018 11:20 AM