Remote Procedure Call (RPC) is an inter-process communication technique to allow client and server software to communicate on a network. The RPC protocol is based on a client/server model. The client makes a procedure call that appears to be local but is
actually run on a remote computer. During this process, the procedure call arguments are bundled and passed through the network to the server. The arguments are then unpacked and run on the server. The result is again bundled and passed back to the client,
where it is converted to a return value for the client's procedure call.
RPC is used by several components in Windows Server, such as the File Replication Service (FRS), Active Directory Replication, Certificate services, DCOM, domain join, DCPromo and RDP, NLB and Cluster, Microsoft Operations Master, Exchange and SQL.
An RPC server is a communications interface provided by an application or service that allows remote clients to connect, pass commands, and transfer data using the RPC protocol. A typical example of an RPC server is Microsoft Exchange Server. Microsoft Exchange
Server is an application running on a computer that supplies an RPC communications interface for an RPC client.
An application will register its RPC server with the operating system’s End Point Mapper (EPM) service so that the remote client can locate the RPC server. When the application registers with the EPM it will indicate the IP address and TCP port that it is
An RPC client is an application running on any given computer that uses the RPC protocol to communicate with an RPC server. An example of a typical RPC client is the Microsoft Outlook application.
NOTE: In this document the terms RPC server and
RPC client refer to the application running at both ends of an RPC communication.
Common causes of RPC errors include:
Use the following procedures to diagnose and repair common causes of RPC errors.
Verify the status and startup type for the RPC and RPC locator services on the server that gets the error:
Verify that ports greater than 1024 are not blocked. Clients connect to RPC Endpoint Mapper on port 135. RPC Endpoint Mapper then tells the client which randomly assigned port between 1024-65535 a requested service is listening on.
Ports may be blocked by a hardware firewall or a software firewall. Software firewalls include Internet Connection Firewall on computers running Windows Server 2003 or Windows XP, and Windows Firewall on computers running Windows Vista, Windows 7, Windows
Server 2008 and Windows Server 2008 R2. A computer might also have third-party firewall software installed, or antivirus software with built-in firewall functionality. By default, port 135 TCP/UDP and ports 1024-65535 TCP must be open for RPC to work. You
can restrict the ports greater than 1024 that RPC uses. However, RPC Endpoint Mapper is always on port 135.
File and Printer sharing for Microsoft Networks will produce the error �������RPC Server is unavailable” when you try to view or manage services on a remote computer using the Services snap-in. See the following example:
Unable to open service control manager database on \\<computer>.
Error 1722: The RPC server is unavailable.
This error message may occur if the File and Printer Sharing for Microsoft Networks component is not enabled on the remote computer.
The process of an RPC client connecting to an RPC server can be broken down into four phases. This troubleshooting guide will discuss the events that occur at each phase, how to test these events, and how to identify if the phase completed successfully.
Phase 1: Name Resolution: Name resolution is the act of resolving a name to an IP address. This normally takes two forms: NetBIOS Name Resolution or the more common DNS Name Resolution.
Phase 2: TCP session establishment: TCP session establishment is the act of establishing a TCP connection between the RPC client and the RPC server. TCP sessions will be initiated by the RPC client via a TCP 3-way handshake with the RPC
Phase 3: RPC Discovery: When a client wants to connect to the RPC server supplied by the application it will contact the computer that hosts the RPC Server and discover how to connect to the RPC Server.
Phase 4: RPC Communication: RPC Communication is the act of making RPC requests to the application endpoint and receiving RPC responses from this application.
Data needed to troubleshoot the issue:
Note: You can also obtain this information by opening Control Panel\Network and Sharing Center, clicking Local Area Connection and selecting Properties.
Name Resolution consists of one or possibly more NetBIOS or DNS queries to locate the IP address for the RPC Server. Troubleshooting this phase requires verifying that a response is received to the name resolution request and that the response contains the
correct IP address for the RPC server. Compare the IP address reported by DNS or NetBIOS in the network trace for the server with the IP addresses you noted earlier. If it does not match then check DNS and WINS and note if there is a difference.
To identify DNS Name Resolution in a network trace use the following filter in Network Monitor or Wireshark: dns. DNS resolution will be occurring at the client so open the network trace taken from the RPC client machine. You will be looking for one packet
that is the query from the client to the DNS server and then the response packet from the DNS server. It will look similar to this:
If the trace shows the correct IP address for the RPC server was returned by the DNS server proceed to TCP Session Establishment.
If the trace does not show a correct IP address returned or you do not see any answer from the DNS server then reference the following resources to help with DNS name resolution troubleshooting.
For details on troubleshooting Active Directory related DNS issues go
For general DNS troubleshooting:
NetBIOS queries come in two forms, WINS or NetBIOS Broadcasts. WINS will consist of a unicast query to a WINS server and a response from the WINS server.
NetBIOS broadcasts are queries broadcast to all hosts on the local subnet so name resolution is limited to only hosts on the subnet. The host with the name listed in the NetBIOS Broadcast will respond with its IP address.
To identify NetBIOS Name Resolution in a network trace, use the following filter in Network Monitor - “nbtns”. For Wireshark, use the following filter - nbns”. If the trace shows a successful resolution using WINS or NetBIOS queries proceed to TCP Session
For details on troubleshooting this NetBIOS Name Resolution further:
TCP Sessions always begin with a TCP 3-way handshake. The handshake should look similar to what is shown below. The RPC Client will send the first packet, known as the SYN packet. The computer hosting the RPC Server will send a SYN/ACK response, and then
the RPC Client will send an ACK packet.
Scenarios that may cause the TCP session to fail
The RPC Server is not actively listening.
It was noted earlier that an RPC Server will register itself and listen on a particular port and IP address of the host computer. If for some reason that fails the TCP layer will answer the SYN packet from the client with a Reset packet.
A device in the middle between the RPC Client and RPC Server will be resetting the connection attempt.
In the client side trace it will appear as if the server sent the TCP Reset while the trace from the server indicates the client is the source of the TCP Reset.
For both these scenarios, check for the presence of a Reset packet in the TCP three way handshake by using the display filter specification of “TCP.flags.reset==1”.
For troubleshooting this step see the following sections in this document:
If the 3-way handshake is successful, continue to the RPC Discovery phase.
The RPC Discovery phase will occur one of two ways. In both methods the client will know the identifier for the RPC Server it wants to contact and will supply that to the computer hosting the RPC Server and ask for information on how to contact the RPC Server.
The identifier is different depending on which method is used and the RPC client will know ahead of time which method it wishes to use.
This method is a two-step process. First the RPC client will contact the End Point Mapper (EPM) on the machine hosting the RPC Server to find out what port and IP address that Server is listening on. Upon successful completion of this the RPC client will
contact the RPC Server directly on the indicated IP address and Port. Below is a sample of what this would look like and a step by step explanation below it. This step depends on the successful TCP session establishment twice, first to the EPM and then to
the RPC Server.
The second method an RPC Client may use to contact an RPC Server is RPC over SMB. This method depends upon first establishing an SMB session with the computer hosting the RPC Server and then using the Named Pipes protocol to communicate using RPC. So in
effect there are several levels of encapsulation – RPC over Named Pipes over SMB over TCP. We will not address the SMB session setup in this document and the TCP session establishment has already been discussed.
With a successfully opened TCP and SMB session, next:
EVENTLOG = The Event log service
winreg = Remote Registry
svcctl = Service Control Manager
srvsvc = Server Service
At this point RPC communication is occurring between the RPC Client and RPC Server. The troubleshooting steps involved at this stage are largely based on the application reporting the RPC failure.
For Active Directory processes or services please see
Active Directory Symptoms.
For Microsoft Exchange related RPC errors please see:
Analyzing Exchange RPC traffic over TCP/IP
RPC network traffic can take multiple forms. It is important to understand which form is in use in order to identify which TCP session is responsible for the RPC communication.
This is sometimes referred to as Traditional RPC or Sockets based RPC. An example of this is Outlook without “Outlook anywhere” or without http settings configured. A TCP session on TCP port 135 is established with the RPC server. To view this traffic in
a trace use the filter: “tcp.port==135”. This session will be used in the RPC Discovery phase to locate the endpoint of the desired application.
RPC over HTTP
RPC connectivity for Internet connected hosts will typically use RPC over HTTP in order to traverse firewalls. Some examples of this can be seen with Terminal Services Gateway, Outlook Web Access, Outlook via “Outlook Anywhere”. This communication will be
established on one or more connections to either TCP port 80 or 443(SSL). Since this typically traverses a public network, SSL or TCP port 443 is the more common method. Use the filter “tcp.port==80 or tcp.port==443” to locate either form inside network trace.
For sessions over TCP port 80, the HTTP requests associated with RPC over HTTP will include a UserAgent header that contains the text “OutlookConnectorDS” and the version number of the connector.
Sessions using TCP port 443 will initially establish a TLS session. After this TLS negotiation, the TCP Payload will be encrypted in TLS/SSL and the contents of the frames will not be readable in the trace. In this phase, look for failures due to improper
certificates, inaccessible Certificate Revocation Lists, or untrusted certificate chains.
For more information on troubleshooting SSL/TLS see:
RPC can also take advantage of SMB sessions for the purpose of RPC communication. Some examples of this can be seen with Computer Management or the Remote Registry service. With the use of RPC over SMB:
If a failure in step 1 occurs, see additional troubleshooting steps see:
File and Printer Sharing.
If Kerberos is used, and the client doesn’t currently have a Kerberos ticket for the RPC server, just after the Negotiate Dialect response is received, the client will obtain a Kerberos ticket for the Servername/cifs SPN of the RPC server. This exchange
will occur over the Kerberos ports TCP or UDP port 88 between the client and a Domain Controller. SessionSetupANDX follows and will consist of a single SessionSetupANDX request which includes the Kerberos ticket, followed by a SessionSetupANDX Response indicating
success or failure of the authentication.
For additional troubleshooting steps during authentication, see
If NTLM is used, SessionSetup will result in a SessionSetupANDX response with a status of STATUS_MORE_PROCESSING_REQUIRED. This response includes the NTLM challenge. The subsequent SessionSetupANDX Request will include the hashed credentials of the client.
At this time, the RPC server must validate the credentials supplied by the user. To do this, the RPC server will contact a domain controller, and validate the credentials with the netlogon service, via RPC, on the domain controller. If this is successful,
the RPC server will then respond to the client with a SessionSetupANDX Response indicating STATUS_SUCCESS.
For additional troubleshooting steps during authentication, see
Verify that authentication is working correctly by checking for Time skew, UDP Fragmentation or an Invalid Kerberos Realm.
Knowledge base article 244474 - "How to force Kerberos to use TCP instead of UDP in Windows Server 2003, in Microsoft Windows and XP, and in Microsoft Windows 2000" provides the steps to resolve this
Naming Convention could not be located because: No authority could be contacted for authentication. Contact your system administrator to verify that your domain is properly configured and is currently online.
Naming information cannot be located because: No authority could be contacted for authentication. Contact your system administrator to verify that your domain is properly configured and is currently online.
To verify that the correct Kerberos realm is configured, follow the steps in 837513 - "Domain controller is not functioning correctly".
1. If you are experiencing replication problems and getting RPC server is unavailable errors as is reported in repadmin /showreps below, use Portqry or Network Monitor to determine if RPC traffic is being blocked is the first step when attempting
to troubleshoot RPC Server is unavailable errors.
[Replications Check,DC2] A recent replication attempt failed:
From DC1 to DC2
Naming Context: CN=Schema,CN=Configuration,DC=xl
The replication generated an error (1722):
The RPC server is unavailable.
The failure occurred at 2003-10-30 11:59.47.
The last success occurred at 2003-10-28 20:50.22.
26 failures have occurred since the last success.
[DC1] DsBind() failed with error 1722,
The RPC server is unavailable..
The source remains down. Please check the machine.
Bermuda\DC1 via RPC objectGuid: 28c78c72-3c95-499a-bcda137a250f069f
Last attempt @ 2003-10-30 11:58.15 failed, result 1722:
The RPC server is unavailable.
Troubleshooting: If IP Security Policies in Active Directory had the Assigned Value to Server (Request Security) set to Yes then these errors will result. Knowledge base article 313190
- "How to use IPSec IP filter lists in Windows 2000" provide details about where to check these settings and more information about their impact.
2. If you are blocking all ICMP traffic between separate AD sites, you will receive the errors below in the output of DCDIAG when trying to replicate inter-site:
Testing server: contoso\DC1
Starting test: Replications
* Replications Check
[Replications Check,DC1] A recent replication attempt failed:
From DC2 to DC1
Naming Context: CN=Schema,CN=Configuration,DC=litware,DC=com
The failure occurred at 2003-08-24 23:00.51.
The last success occurred at (never).
553 failures have occurred since the last success.
[DC2] DsBind() failed with error 1722,
REPLICATION LATENCY WARNING
DC1: A full synchronization is in progress
from DC2 to DC1
Replication of new changes along this path will be delayed.
[DC2] LDAP connection failed with error 58,
The specified server cannot perform the requested operation.
Troubleshooting: To resolve this issue, remove the ICMP traffic restriction between domain controllers. When establishing an RPC session prior to AD replication, ICMP traffic is used. If the ICMP fails, so does the RPC session establishment,
and hence AD replication also fails. ISA 2004 can prevent ICMP traffic with the exception of computers specified in the Remote Management Computers computer set which can be configured in system policy.
3. The following error will appear when attempting to connect to the computer.
"computer <\\servername.domain.local> cannot be managed. The network path was not found. RPC server is unavailable.
Or when viewing the properties of the remote computer you will receive the error:
"Win32: The RPC server is unavailable".
Troubleshooting: Computer management is one of the better tools for testing RPC connectivity. When RPC traffic is being blocked, connections to other computers using the computer management console will fail.
4. When attempting to promote an additional domain controller in an Active Directory domain while the RPC service is blocked or not running, the following error will appear:
"The domain "domain.local" is not an Active Directory domain, or an Active Directory domain controller for the domain could not be contacted.
5. Connections to computers via Remote Desktop may fail if RPC connectivity cannot be established. When attempting to logon on to the domain via Remote Desktop the following error will be produced in the form of a popup error message if RPC connectivity
is the root of the problem:
"The system cannot log you on due to the following error: The RPC server is unavailable.”
You may also see the following errors on the Terminal server:
Error 1727: The remote procedure call failed and did not execute
Error 1722: The RPC server is unavailable.
Error 1723: The RPC server is too busy to complete this operation.
Error 1721: Not enough resources are available to complete this operation.
Event ID 5719:
Description: No Windows NT Domain Controller is available for domain domain_name.
The following error occurred: There are currently no logon servers available to
service the logon request.
Event ID: 1219
Details: Logon rejected for CONTOSO\<computername>. Unable to obtain Terminal Server
Configuration. Error: The RPC server is unavailable.
Troubleshooting: These errors can be a result of the TCP/IP NetBIOS Helper service being disabled on the Terminal server or NetBIOS over TCP/IP being disabled on one of the NIC's used to access the Terminal server. You should also verify
that the Client for Microsoft networks is bound to the adapter used to access the Terminal server. You can tell if this is happening by looking at a Netdiag /v from the box for the following output:
Testing redirector and browser... Failed
NetBT transports test. . . . . . . : Failed
List of NetBt transports currently configured:
[FATAL] No NetBt transports are configured.
Redir and Browser test . . . . . . : Failed
List of transports currently bound to the Redir
[FATAL] The redir isn't bound to any NetBt transports.
List of transports currently bound to the browser
[FATAL] The browser isn't bound to any NetBt transports.
Computer Management MMC to a remote host
Outlook to an Exchange server
RPCPing - http://support.microsoft.com/kb/831051
RPCPing - http://support.microsoft.com/kb/831051
Network Monitor -
Wireshark - Download
You can use the Portqry tool to verify that the required ports are open. You should run the Portqry tool on a computer that is not receiving any RPC errors against a computer that is receiving RPC errors by using the -n switch. To this, follow these steps:
a. Click "Start", click "Run", type "cmd" in the "Open" box, and then click OK".
b. Type "portqry -n <problem_server> -e 135" (without the quotation marks).
The output will appear similar to the following examples:
Querying target system called:
Attempting to resolve name to IP address...
Name resolved to 169.254.1.1
TCP port 135 (epmap service): LISTENING
Using ephemeral source port
Querying Endpoint Mapper Database...
UUID: f5cc59b4-4264-101a-8c59-08002b2f8426 NtFrs Service
UUID: e3514235-4b06-11d1-ab04-00c04fc2dcd2 MS NT Directory DRS Interface
If port 135 is blocked, the following will appear:
TCP port 135 (epmap service): NOT LISTENING However, for these RPC Endpoint Mapper errors it is likely that ports greater than 1024 are blocked, and not port 135.From the output, you know the DC is using port 1094 for FRS and 1025, 1029, and 6004 for Active
Directory replication. You can use the Portqry tool again to check those ports. For example, you can test all the ports at the same time by using the Portqry tool with the -o switch. For example, type
"portqry -n <problem_server> -o 1094,1025,1029,6004"(Without the quotation marks)
If the ports all respond as "LISTENING," it's likely that blocked ports are not causing this problem. If any ports respond as "NOT LISTENING," the ports are probably blocked.
Basics of RPC are covered here:
RPC to Go v.1:
Architecture and a closer look at a connection to the RPC Endpoint mapper in a network capture.
RPC to Go v.2:
This describes how RPC commands can be sent over Named Pipes in SMB via the IPC$ Tree.
RPC to Go v.3:
Troubleshooting “RPC server is unavailable” error, reported in failing
AD replication scenario.
This one is good. It lays out RPC basics really quickly and then moves on RPC errors. The information on MaxUserPort would need to be updated with the information about the dynamic port ranges that are used in Vista/W2008 are the high range of ports compared
to the 1025-5000 for W2003.
How IT Works, Troubleshooting RPC Errors by Zubair Alexander:
Troubleshooting RPC Endpoint Mapper errors using the Windows Server 2003 Support Tools from the product CD