Friday, August 14, 2015

MAPI NSPI Issues with Exchange 2013

A customer running two Exchange 2013 CU9 multi-role servers in a cluster with an F5 load balancer contacted me this morning reporting issues where they could not create new Outlook Profiles for users.  Users with existing Outlook profiles were not affected by the issue.

Users primarily connect with MAPI over HTTP which is enabled on the Exchange Organization configuration.

When a user attempted to create a new profile, they would receive the following error:

The connection to Microsoft Exchange is unavailable.  Outlook must be online or connected to complete this action.


When running the Microsoft Exchange Remote Connectivity Analyzer against the Exchange organisation, the following errors were captured:

Testing MAPI over HTTP connectivity to server webmail.company.com
MAPI over HTTP connectivity failed.
  Additional Details
  HTTP Response Headers:
request-id: 1f79419f-0c42-4a38-bf07-99b1c6882928
Set-Cookie: ClientId=MUZOXYKGXN9GLA; expires=Sat, 13-Aug-2016 02:50:02 GMT; path=/; HttpOnly
Server: Microsoft-IIS/8.5
WWW-Authenticate: Negotiate,NTLM
X-Powered-By: ASP.NET
X-FEServer: SERVER2
Date: Fri, 14 Aug 2015 02:50:02 GMT
Content-Length: 0
Elapsed Time: 7764 ms. 


Testing the MAPI Address Book endpoint on the Exchange server.
An error occurred while testing the address book endpoint.
  Additional Details
  Elapsed Time: 5784 ms. 

  Test Steps
  Testing the address book "Check Name" operation for user TestUser@company.com against server webmail.company.com.
  An error occurred while attempting to resolve the name.
  Additional Details
  A protocol layer error occured. HttpStatusCode: 500
FailureLID: 47372
FailureInfo: 

###### REQUEST [2015-08-14T02:50:03.1064218Z] ######

POST /mapi/nspi/?mailboxId=e110343c-c351-4dac-b066-3b552417a51b@company.com HTTP/1.1
Content-Type: application/octet-stream 
User-Agent: MapiHttpClient 
X-RequestId: 57959cb1-fd5d-4ae1-ac55-4494f0a61748:1 
X-ClientInfo: 6016fb10-ca94-4a67-847c-549dd99bacb8:1 
X-ClientApplication: MapiHttpClient/15.1.243.0 
X-RequestType: Bind 
Authorization: Negotiate [truncated] 
Host: webmail.company.com 
Cookie: ClientId=SGMTABEYKDADLXOTCG 
Content-Length: 45 

--- REQUEST BODY [+0:05.159] ---
..[BODY SIZE: 45]

--- REQUEST SENT [+0:05.159] ---

###### RESPONSE [+0:05.781] ######

HTTP/1.1 500 Internal Server Error
request-id: 9653be79-d895-4cf3-8169-98ceeb218197 
X-CalculatedBETarget: Server1.company.local 
X-DiagInfo: Server1 
X-BEServer: Server1 
X-FailureContext: BackEnd;500;NTAw;U3lzdGVtLk5ldC5XZWJFeGNlcHRpb246IFRoZSByZW1vdGUgc2VydmVyIHJldHVybmVkIGFuIGVycm9yOiAoNTAwKSBJbnRlcm5hbCBTZXJ2ZXIgRXJyb3IuDQogICBhdCBTeXN0ZW0uTmV0Lkh0dHBXZWJSZXF1ZXN0LkVuZEdldFJlc3BvbnNlKElBc3luY1Jlc3VsdCBhc3luY1Jlc3VsdCkNCiAgIGF0IE1pY3Jvc29mdC5FeGNoYW5nZS5IdHRwUHJveHkuUHJveHlSZXF1ZXN0SGFuZGxlci48PmNfX0Rpc3BsYXlDbGFzczJjLjxPblJlc3BvbnNlUmVhZHk+Yl9fMmIoKQ==;;ProtocolError; 
Persistent-Auth: true 
X-FEServer: Server1 
Transfer-Encoding: chunked 
Cache-Control: private 
Content-Type: text/html; charset=utf-8 
Date: Fri, 14 Aug 2015 02:50:08 GMT 
Set-Cookie: X-BackEndCookie=e110343c-c351-4dac-b066-3b552417a51b=u56Lnp2ejJqBz5nPyM2cxpnSncebmdLLyZue0sfJnJ7SnM2Znc7Iys7LnJnMgYHNz87K0s/G0s7Mq8/NxcrPxc/H; expires=Sun, 13-Sep-2015 02:50:08 GMT; path=/mapi; secure; HttpOnly 
Server: Microsoft-IIS/8.5 
X-AspNet-Version: 4.0.30319 
X-Powered-By: ASP.NET 



Also on only on Server1, the following error was experienced in the event log.


Log Name:      Application
Source:        ASP.NET 4.0.30319.0
Date:          14/08/2015 5:54:03 PM
Event ID:      1309
Task Category: Web Event
Level:         Warning
Keywords:      Classic
User:          N/A
Computer:      Server1
Description:
Event code: 3005 
Event message: An unhandled exception has occurred. 
Event time: 14/08/2015 5:54:03 PM 
Event time (UTC): 14/08/2015 9:54:03 AM 
Event ID: 6e8a15487387414b9279078f9f6aba51 
Event sequence: 2 
Event occurrence: 1 
Event detail code: 0 

Application information: 
    Application domain: /LM/W3SVC/2/ROOT/mapi/nspi-3615-130840196323691514 
    Trust level: Full 
    Application Virtual Path: /mapi/nspi 
    Application Path: C:\Program Files\Microsoft\Exchange Server\V15\ClientAccess\mapi\nspi\ 
    Machine name: Server1

Process information: 
    Process ID: 13528 
    Process name: w3wp.exe 
    Account name: NT AUTHORITY\SYSTEM 

Exception information: 
    Exception type: HttpException 
    Exception message: RpcObjectSetType
   at System.Web.HttpApplicationFactory.EnsureAppStartCalledForIntegratedMode(HttpContext context, HttpApplication app)
   at System.Web.HttpApplication.RegisterEventSubscriptionsWithIIS(IntPtr appContext, HttpContext context, MethodInfo[] handlers)
   at System.Web.HttpApplication.InitSpecial(HttpApplicationState state, MethodInfo[] handlers, IntPtr appContext, HttpContext context)
   at System.Web.HttpApplicationFactory.GetSpecialApplicationInstance(IntPtr appContext, HttpContext context)
   at System.Web.Hosting.PipelineRuntime.InitializeApplication(IntPtr appContext)

RpcObjectSetType
   at Microsoft.Exchange.Rpc.RpcServerBase.ThrowRpcException(String message, Int32 rpcStatus)
   at Microsoft.Exchange.Rpc.ProcessAccess.ProcessAccessRpcServer.RegisterInterface(Void* ifSpec, ValueType mgrTypeGuid, _GUID* pMgrTypeUuid, Void* pMgrEpv, UInt32 flags, UInt32 maxCalls)
   at Microsoft.Exchange.Rpc.RpcServerBase.RegisterServer(Type type, ObjectSecurity sd, UInt32 desiredAccess, ValueType mgrTypeGuid, Void* mgrEpv, String annotation, Boolean isLocalOnly, Boolean autoListen, UInt32 maxCalls)
   at Microsoft.Exchange.Data.ApplicationLogic.ProcessAccessManager.RegisterComponent(IDiagnosable diagnosable)

Request information: 
    Request URL: https://localhost:444/mapi/nspi 
    Request path: /mapi/nspi 
    User host address: ::1 
    User:  
    Is authenticated: False 
    Authentication Type:  
    Thread account name: NT AUTHORITY\SYSTEM 

Thread information: 
    Thread ID: 81 
    Thread account name: NT AUTHORITY\SYSTEM 
    Is impersonating: False 
    Stack trace:    at System.Web.HttpApplicationFactory.EnsureAppStartCalledForIntegratedMode(HttpContext context, HttpApplication app)
   at System.Web.HttpApplication.RegisterEventSubscriptionsWithIIS(IntPtr appContext, HttpContext context, MethodInfo[] handlers)
   at System.Web.HttpApplication.InitSpecial(HttpApplicationState state, MethodInfo[] handlers, IntPtr appContext, HttpContext context)
   at System.Web.HttpApplicationFactory.GetSpecialApplicationInstance(IntPtr appContext, HttpContext context)
   at System.Web.Hosting.PipelineRuntime.InitializeApplication(IntPtr appContext)

After further investigation, we discovered that only mailboxes which resided on an Active database on Server1 experienced the issue creating Outlook profiles, the same server which experienced EventID 1309.  After failing mailbox databases over to Server2, users were able to login to Outlook without issues.

After this we identified the issue was only present with Server1.

On Server1 we tested the NSPI website by navigating to the Exchange backend in IIS Manager and selecting Browse:444 (https).


The browse failed on Server1 with an "unhandled exception error".  This was aligned with the HTTP 500 exception shown in the Event Logs on Server1 and the Exchange Remote Connectivity Analyzer.

RpcObjectSetType 
Description: An unhandled exception occurred during the execution of the current web request. Please review the stack trace for more information about the error and where it originated in the code. 

Exception Details: Microsoft.Exchange.Rpc.RpcException: RpcObjectSetType

Source Error: 
An unhandled exception was generated during the execution of the current web request. Information regarding the origin and location of the exception can be identified using the exception stack trace below. 


On Server2 when repeat the same process.


On Server2 it prompted as authentication as normal.


This shows the MAPI/NSPI results as expected.



Resolution:

We rebooted Exchange Server 1 and after a reboot, it started responding to MAPI/NSPI requests as expected.  We believe it was a side affect after the CU9 upgrade, however will monitor the situation and update this blog thread in the event the issue reoccurs.

6 comments:

  1. Interestingly, we had the same issue at approximately the same date (started on Aug 13rd) and on servers that were upgraded from CU8 to CU9 as well. However CU9 was installed almost two months ago and server has been restarted multiple times in between. In our case, recycling the MSExchangeMapiAddressBook application pool immediately solved the issue. We have 24 servers and only one was affected. All servers run CU9 but only 4 of them were installed on CU8 and later upgraded.

    ReplyDelete
    Replies
    1. cheap windows 7 anytime upgrade key , windows 10 product key , windows 7 sp1 ultimate 64 bit product key , windows 7 key code , windows 7 product key , windows 7 home premium wholesale , key upgrade windows 7 home premium to professional 64 bit , buy windows 10 product key , lpiz9R

      Delete
  2. Instead of rebooting the server a short "iisreset" helps also. I have the same issue on both servers in the DAG

    ReplyDelete
  3. Hi Clint,
    thanks for sharing this solution. It helped us a lot. We have multi-role Exchange 2013 CU9 cluster as well.

    ReplyDelete
  4. IISRESET on the CAS servers did the trick for me. Thanks.

    ReplyDelete
  5. Just experienced this same problem on one of our four Exchange 2013 CU10 servers. Everything was working fine and then one day out of the blue, outlook users couldn't connect to the directory and had issues creating profiles and looking up GAL entries. Your blog / solution saved the day! Thx again. We restarted IIS on the server with the issue and all worked fine again. Isn't managed availability suppose to fix issues like this?!

    ReplyDelete