Ensure that the identifier of the RP matches exactly the identifier that the actual application has in its configuration.
Ensure that time is also in sync. The ADFS should not provide a token which is not yet valid for the RP.
A fiddler trace might help. Try taking one, sanitize it (remove credentials) and share it over here!
Note: Posts are provided “AS IS” without warranty of any kind, either expressed or implied, including but not limited to the implied warranties of merchantability and/or fitness for a particular purpose.