After a network outage, all systems and services come back online except for the BlackBerry smartphones.
Troubleshooting identifies the following event from the Application log of the Exchange 2010 server:
Source: MSExchange ADAccess
Event ID: 2915
Process Microsoft.Exchange.RpcClientAccess.Service.exe (PID=...). User '...BESAdmin~RCA~false' has gone over budget '...' times for component 'RCA' within a one minute period. Info: 'Policy:[Fallback], Parts:MaxConcurrency:...;'. Threshold value: '100'.
1: Restart the Exchange 2010 server which holds the BESAdmin mailbox (*See Note 1 below)
2: Restart the BlackBerry Controller service on the BES Server
*Note 1: Potentially restarting the Microsoft Exchange RPC Client Access service on each CAS server may have been sufficient (please comment if you know the answer to this.) UPDATE: received comment from Anonymous that this is sufficient
*Note 2: Also works in a DAG environment; restart (assuming database copy statuses are either mounted or healthy) whichever Exchange holds the mounted database copy containing the BESAdmin mailbox
“The throttling framework is intended to protect Exchange resources, so if it is going to "fail", it needs to do so in a safe and predictable way. …. When ... tries to load policy … it fails. ... it fails along a fallback path.
If the non-default policy is corrupt or missing, it will first fall back to the default throttling policy for the organization in question. … then it falls back to a special policy defined in code called the "fallback policy". … this policy is embedded in the Exchange assemblies....”
In above scenario, it appears that – due to the network outage – there was trouble contacting a domain controller, trouble reading the BESPolicy throttling policy, trouble reading the default throttling policy, hence it passed to the fallback throttling policy which is hard coded in Exchange 2010 (even though the GetThrottlingPolicy cmdlet indicated the BESAdmin mailbox was using the correct policy.) Recreating the BESPolicy from scratch and re-applying to the BESAdmin mailbox did not fix, nor did a restart of the BES 5.0 server.
The above issue was seen with an Exchange 2010 SP1 DAG and BES 5.0 setup.
*For configuring Exchange 2010 BESPolicy throttling policy correctly, see: