Load Balanced Deployment of MachPanel

Summary

This article provides information for the load balanced deployment of MachPanel.

Applies To

This article applies to MachPanel build 5.0.11. and above.

MachPanel high availability / load balanced setup:

Please review below KB for HA setup:

https://kb.machsol.com/Knowledgebase/55735/How-to-accomplish-MachPanel-high-availability-load-balanced-setup

Load Balanced Deployment of MachPanel

Pre-requistes:

1. Hardware/Software load balancer to distribute requests among MachPanel servers
2. SQL replication and connection string modification
3. Same hardware/windows should be used for load balanced servers

Important Notes:

1. If your MachPanel install directory does not contain LB-Setup, P2PStateService.exe and p2pStateService.exe.config, then download from zip file below. If you use the latest installer to install MachPanel, these files will already be included in the install directory.
2. LB-Setup utility must be run on primary load balanced server of MachPanel first and then subsequently on each of the secondary servers(s).
3. LB-Setup utility will not import/sync any IIS bindings and client is required to manually sync/import the IIS bindings from primary control server to all other secondary servers.
4. MachPanel Secondary Control server must not be accessed via web interface until this utility is successfully executed and control server files and registry entries and custom state service configuration files are synchronized between primary server and all the secondary servers.
5. Database level load balancing is out of the scope of this document and supporting LB-Setup utility application.

Components

Load balanced deployment of MachPanel consists of following components

• Primary Server
• Secondary Server(s)

Both components have many common roles, but there are some specific roles which are assigned only to the primary server. Both the components are assumed to share same database at backend for data processing and presentation.

1. Primary Server:

Primary server is MachPanel Control server that not only hosts the MachPanel Web application but also run MachPanel services and desktop application for MachPanel configuration (Configuration Studio). There can only be one machine/server designated as primary server for MachPanel Control Server and not more than this.

Primary server is responsible for license validation and also enables control server administrator to look for future updates, download these updates and install these updates.

2. Secondary Server:

Secondary server has to be installed using MachPanel Installer with an option selected to install as secondary server. This server will also host all the backend services and web application but not desktop application for configuring the panel. This server will be using licenses of primary server.

There can be arbitrary number of secondary servers in a Load Balanced deployment of MachPanel and there is no limit on number of servers.

Installation & Configuration

Once MachPanel control server is installed and licenses are activated, a utility “LB-Setup.exe” is required to be executed to designate the server machine as PRIMARY/SECONDARY SERVER in a Load Balanced Deployment of the MachPanel.

Above screen depicts the utility interface. This is a simple utility which is run for setting up both primary and secondary servers for load balanced setup. Below are each of the input fields are defined.

a. Peer IP

b. Peer Port

A valid and unused port number which will be used for communication between Load Balance peer servers. This port number shall greater than 256 and less than 65536 and must not be used in any other application previously. Also remember 42424 is a reserved port for internal communication and cannot be assigned as peer port.

c. Is Primary

This is a flag to tell the setup utility whether it is run for Primary server or secondary server when running for secondary server the utility requires two more inputs which are defined below:

d. Connection String

When running the utility for secondary server, the utility requires database access to find out other peers servers and validate IP address and port and also perform some internal sync operations.

e. Select Primary Configuration

This is needed for secondary peer server setup for MachPanel Load balance server. This is a “Web.Config” file placed inside installation folder of the MachPanel. The file shall be copied from primary load balanced server of MachPanel to all the secondary servers.

Additional Steps:

• Copy Files from Primary Server (..\MachSol\MachPanel Control Server)
• Stop services on Secondary Server, Take backup.
• Replace copied files on Secondary server (..\MachSol\MachPanel Control Server).
• Take backup of registry on Primary Server
  
  
  Copy this registry file on secondary server and Restore (follow these steps)
  
  
  
  

Operations Performed

This utility performs following steps when executed successfully

1. Create “MachPanel_Web” user account if not exists previously (Secondary server(s) only) or change its password if it already exists from “Computer Management” >> System Tools >> Local Users and Groups >> Users”.
2. Set this password in IIS for MachPanel Application Pool.
3. Install custom state service to manage sessions between multiple servers. It is named “P2PStateSvc.exe”. And once installed it is set to auto start configuration in windows services
4. Disable default ASP Net State Service to avoid any conflicts in session management
5. IP and Port of the peer [provided through utility interface] is configured in custom state service configuration. IP is local server IP configured in load balancer and IIS, port is used to share session data among load balanced server e.g. 42425. A separate port is needed to be configured for each load balanced server, like 42425 for Primary, 42426 for Secondary. Port 42424 is used for web server port on all servers.
6. Copy provided [provided through utility interface] control server configuration into secondary server installation directory  (Secondary server(s) only)
7. Validation key and encryption key are generated for session security and saved in control server web configuration  (primary server only)
8. Provisioning service identity is set to “MachPanel_Web” and is restarted.
9. Custom state service is started.
10. IIS app pool identity is set to “MachPanel_Web” user.
11. Existing database is linked by saving connection string [provided through utility interface] in MachPanel Registry settings  (Secondary server(s) only)
12. Load Balance Peer details are saved in MachPanel Database

How It Works?

How Control Server Web application works and manages sessions on multiple machines?

There exists three methods how an ASP.net application can manage its session state and these are

• In-Process Management
• Management through SQL
• Management through State Service Server

State Service Server is an out of process session state storage option that works with web server farms. It stores session data in memory and delivers better performance than SQL Server. This seems like a good compromise between the in-process option and the SQL server options. It has some drawbacks, however.

1. Firstly, several web servers typically depend on one state server for session state which introduces a critical single point of failure.
2. Secondly, in a load balanced environment, the load balancer may redirect a user’s request to a web server that is different from the web server that served the user’s previous request. If the new web server communicates with a different state server, the user’s original session state will not be found and the web application may not work properly.This problem occurs even in persistence-based (a.k.a. sticky) load balancers either erroneously or due to server failure.
3. Thirdly, an issue the default ASP.Net web server and state server communicate in plain text. An eavesdropper can easily get hold of session state data traveling on the network. This may not be a threat if all servers are running in an internal network but it is certainly cause for concern when web servers and state servers are spread across the internet.

A custom state service is installed to manage session data across different servers and also address all the above drawbacks of default ASP.Net default State Service Server. Using this service session data is securely shared across two or more load balanced servers and threat of single point of failure has been removed as all the peers share their session state with other peers in the environment.

License validation, Configuration Studio and software updates:

All the secondary servers for MachPanel Control Server are validated against the licenses issued for primary server of MachPanel. Hence it comes necessary to first completely deploy primary peer server of MachPanel and then any number of subsequent secondary servers.

Configuration studio is only available on primary control server and using this configuration studio software updates are also downloaded and installed on primary server only. There is a mechanism implemented to synchronize files and registry on all peer servers from all other servers with latest files and registry entries, so once primary peer server is updated all the other peer servers will update themselves within few minutes.

MachPanel Services:

This is also very crucial part of this load balanced deployment as how MachPanel backend services such as Provisioning-Svc, Billing-Svc will process their tasks.
As each service has multiple threads working inside a single process, a mechanism has been implemented to lock down a thread for a given request in database and on completion of the request (success or failure) the lock has been released and hence more than once execution of a pending task has been avoided.

How to change a secondary node to primary node in case Primary server is de-commissioned?

Following steps are needed to change a secondary server to primary Server.

1. Open MachPanel database table Hb_tblServerRoles. Update IsPrimary = 0 and IsEnabled = 0 for primary server (where IsPrimary = 1).
2. Update IsPrimary = 1 for secondary server which you want to make primary.
3. Remove License file (MachPanel.dll) from Auth folder in MachPanel installation directory example: C:\Program Files\MachSol\MachPanel Control Server\Auth.
4. Update SyncState = 0 for all load balanced servers in table Hb_tblServerRoles.
5. Open MachPanel, refill Setup form to generate new licenses against new Primary server.
6. Contact MachSol sales to update licenses against new Installation Id.