The Work Management Service needs
Important: The service account used for the Work Management Service Application needs Full Control permissions on the User Profile Service Application! Also, when setting up the Work Management Service Application
be sure to use the same service account as your Web Applications for the application pool identity. If this permission is lacking you will get "We are having trouble refreshing your tasks". If it is being used in a farm where you are required
to use separate accounts the following items need to be done:
I have also discovery that in order to get this to work I needed to use the same service account as my Web Application pool when setting up the Work Management Service Application Pool.