Check the event log on the Hyper-V host system in question. We ran into similar issue.
I would try:
Load all of the latest SCVMM updates for server for clients
Check for network issues (our server had dual networks with dual DNS records on our DNS server causing an issue. Unregistering the 2nd DNS record and blocking the DNS registration on the multiple network cards fixed it.)
Make sure the system is not taxed or over subscribed (slow responses trigger this error too.)