By:Manvendra Singh || Related Tips:More > AlwaysOn Availability
ProblemRecently we had a project to configure SQL Server AlwaysOn Availability Groups between three nodes which were hosted on a multi-subnet environment. A multi-subnet environment requires the windows cluster to be used as the backbone for AlwaysOn and each of the server nodes are located in multiple/different subnets. If you want to learn how to configure SQL Server AlwaysOn between a multi-subnet cluster then I would recommend youread this tip.
In this tip, I will explain how to fix an issue which I faced when configuring SQL Server AlwaysOn. This tip will help you fix this issue and help you reconfigure SQL Server AlwaysOn in a multi-subnet network.
SolutionAs per Books Online "The AlwaysOn Availability Groups feature is a high-availability and disaster-recovery solution that provides an enterprise-level alternative to database mirroring. Introduced in SQL Server 2012, AlwaysOn Availability Groups maximizes the availability of a set of user databases for an enterprise. An availability group supports a failover environment for a discrete set of user databases, known as availability databases that fail over together. An availability group supports a set of read-write primary databases and one to eight sets of corresponding secondary databases. Optionally, secondary databases can be made available for read-only access and/or some backup operations."
Configuration Scenario and SetupWe have three machines named PRI-DB1 ( IP: 10.X.3.XXX ), PRI-DB2 ( IP: 10.X.4.XXX ) and SEC-DB2 ( IP: 172.X.15.XXX ). The IP addresses of the machines reflects their subnet indicating they belong to a different series from each other. PRI-DB1 and PRI-DB2 are hosted in a corporate data center whereas SEC-DB2 is hosted by the Amazon cloud platform. All three machines are running Windows Server 2012 R2 Enterprise Edition and SQL Server 2014 Enterprise Edition. PRI-DB1 will be the primary replica and the remaining two nodes will be the secondary replicas. Data replication between PRI-DB1 and PRI-DB2 will use synchronous-commit mode and the failover mode will be Automatic with no data loss which can be used for HA in case the primary replica goes down. Data replication between PRI-DB1 and SEC-DB2 uses asynchronous mode and the failover mode is Manual which can cause some data loss in case of a disaster.
Before going ahead, I would like to let you know that I have followed the step by step process to configure SQL Server AlwaysOn between multi-subnet cluster , but ended with an error 35250 at the end of the SQL Server AlwaysOn configuration window. I am unable to configure SQL Server AlwaysOn because of the error Failed to join the database 'DRTest' to the availability group 'DBAG' on the availability replica 'PRI-DB2' (Error: 35250). Although you can see the details like secondary replicas, databases, etc. created under the AlwaysOn High Availability folder despite throwing this error, but the databases will not be joined to the Availability Group which we will do later in this tip to fix the issue.
I have not covered SQL Server and Windows Server Failover Cluster installation and the initial SQL Server AlwaysOn configuration steps in this tip. You can use my last tips to configure an AlwaysOn Availability Group . I am assuming that these steps will be taken care of by you prior to fixing this issue with the help of this tip.
NOTE : MAKE SURE TO IMPLEMENT THIS SOLUTION IN A LOWER LIFE CYCLE FIRST. DO NOT MAKE ANY CHANGES IN PRODUCTION WITHOUT PROPER TESTING IN LOWER-LIFE CYCLE ENVIRNOMENTS.
Troubleshooting Error 35250Step 1
As I said above, I will not show you the steps to configure SQL Server AlwaysOn, because I assume that you have already gone through the steps mentionedin my last tip. Here I will go directly to the last window of the SQL Server AlwaysOn configuration where we get the errors (if any) post execution. You can click on the "Error" link to get the details of the error as shown in the below screenshot.
Step 2
I did some research on social media and on the web to rectify this issue. I found this MSDN link where many solutions were suggested. I decided to check the endpoint status on all replicas first by running the below command.
--Run below command to check endpoint stateselect r.replica_server_name, r.endpoint_url,
rs.connected_state_desc, rs.last_connect_error_description,
rs.last_connect_error_number, rs.last_connect_error_timestamp
from sys.dm_hadr_availability_replica_states rs join sys.availability_replicas r
on rs.replica_id=r.replica_id
where rs.is_local=1
First I executed the above command on the primary replica and the output is noted as "Connected".
Next I executed the same command on both secondary replicas where I learned that the endpoint state is "Disconnected" with an error which is shown below.
Connection attempt failed with error: '10060(A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.)'.
The endpoints are not connected on the secondary replicas which caused the connection failure. The target primary and secondary replicas are unable to communicate because of a firewall rule like inbound traffic for port number 5022 on which the endpoint needs to be checked and unblocked because by default inbound traffic is blocked by the Windows firewall.
Step 3
By default, AlwaysOn configures endpoints to use port 5022 which we kept in our configuration. You can see the endpoint port number 5022 is used in the above image. You can query sys.tcp_endpoints on each replica to confirm which port is being used in your environment. As per MSDN "Port 5022 is used by the primary and secondary replicas for synchronization and communication purposes. Traffic must be allowed inbound on this port. Testing has shown that if port 5022 inbound traffic is blocked at the primary, at the secondary or at both, you will be unable to create the availability group and message 35250 will be reported."
So here we will create a new Inbound Port Rule for port 5022 on the Primary and both Secondary servers. Launch "Windows Firewall with Advanced Security" console to create a new Inbound Port Rule. You can type "Windows Firewall with Advanced Security" in the Windows search option to get this console as shown in the below image.
Click on the "Windows Firewall with Advanced Security" as shown above.
The "Windows Firewall with Advanced Security" console will appear. You can see the "Windows Firewall with Advanced Security" console in the below screenshot.
Step 4
Since we have to create an inbound traffic rule, right click on "Inbound R