Tuesday, April 19, 2011

weblogic clustering on different host

Hello Everyone,
Today we will discuss about weblogic clustering on different host, lets discuss about clustering.
I have posted some screenshots as well, if any mismatch in that then please correct me.... :)

"A WebLogic Server cluster consists of multiple WebLogic Server server instances running simultaneously and working together to provide increased scalability and reliability. A cluster appears to clients to be a single WebLogic Server instance. The server instances that constitute a cluster can run on the same machine, or be located on different machines. You can increase a cluster's capacity by adding additional server instances to the cluster on an existing machine, or you can add machines to the cluster to host the incremental server instances. Each server instance in a cluster must run the same version of WebLogic Server."

There are also several methods for setting up the clustered domain, namely using the Fusion Middleware Configuration Wizard, the Administration Console, the wlst command line utility and even programmatically. In this example we will combine the first three: we will create the domain using the Configuration Wizard and setup the domain cluster using the Administration Console and the wlst utility.

Briefly the steps involved in setting up the cluster are:

1. Install WebLogic on both physical machines
2. Create the WebLogic Domain on one physical machine
3. Configure the domain by adding Managed Servers and Machines on one physical machine
4. Copy the domain file structure to the other physical machine and enroll it to the WebLogic Domain
5. Ensure that both WebLogic Machines are operational
6. Create and configure the Cluster
7. Ensure that the Cluster is operational

1. Install WebLogic on both physical machines

The same exact version of WebLogic must be installed on both physical machines. Also, WebLogic must be installed on the same Middleware Home in both machines. This is necessary since we will be copying the domain file structure from one physical machine to the other - step 4 above.

Start by going to the Oracle WebLogic downloads page and download the latest version of WebLogic for Linux.

The 32-bit Linux WebLogic version 10.3.3. file that is downloaded is called wls1033_oepe111150_linux32.bin. For installing and administering this installation create a weblogic Linux user account and use it to login and perform the installation. Make sure that the maximum number of open files is set to 4096 for the user account.

Ensure that the wls1033_oepe111150_linux32.bin has execute permission and start the installation by typing ./wls1033_oepe111150_linux32.bin. The installation will self-unpack and present the Welcome page.

On the Choose Middleware Home Directory page accept the default Oracle/Middleware under the /home/weblogic and press Next.

On the Register for Security Updates page click Next. On the Choose Install Type page select Custom and click Next. On the Choose Products and Components page you may deselect Oracle Enterprise Pack for Eclipse and click Next. Accept the defaults on the JDK Selection page - both the Java and JRockit SDKs are selected - and click Next. On the Choose Product Installation Directories page accepts the defaults and click Next.

Verify the installation parameters on the Installation Summary page and press Next to proceed with the installation.

Now proceed with the installation of WebLogic using the exact same installation package and the steps outlined above on the other machine.

Ensure that in page Choose Middleware Home Directory the same Middleware Home is specified. In this case we will accept the default Oracle/Middleware under the weblogic user's home directory /home/weblogic.

2. Create the WebLogic Domain in one physical machine

Now, having completed the WebLogic installations on both machines, we can proceed with the creation of the WebLogic Domain. We will do this only on one of the two machines.

Start the Fusion Middleware Configuration Wizard utility - it is called config.sh and is located in the wlserver_10.3/common/bin directory under the Middleware Home (/home/weblogic/Oracle/Middleware for our installation) and on the Welcome page select Create a new WebLogic domain.

At this point accept the default products on the Select Domain Source page and click Next. On the Specify Domain Name and Location page specify the name and location of the domain. For this example we have called it clusteredDomain. On the Configure Administrator User Name and Password page specify the weblogic administrator password. Accept the appropriate startup mode and JDK on the Configure Server Start Mode and JDK page and click Next. On the Select Optional Configuration page select Administrator Server only. We will create the Managed Servers, Machines and Clusters later on using the Administration Console.

On the Configure the Administration Server page you can accept the defaults for the administration server  Name and Listen port and click Next. Verify your selections on the Configuration Summary page and when ready click on Create to proceed with the creation of the domain.

The Configuration Wizard proceeds with the creation of the domain under the path specified, which defaults to user_projects/domains under the Middleware Home. Once the domain is created successfully, we can start it by running the startWebLogic.sh script in the user_projects/domains/clusteredDomain. Verify that the domain is started successfully by observing the logs in the terminal window.

3. Configure the domain by adding Managed Servers and Machines in one physical machine
With the domain running we can proceed with the creation of the Managed Servers and Machines using the WebLogic Administration Console. So, start your browser and go to the Administration Console by specifying its address as: http://machine1:7001/console ,where machine1 is the ip address or host name of the machine that you used in step 2 above to create the Administration Server. This should display the Administration Console login form.

Enter the server administration name and password you specified in step 2 when creating the domain - username defaults to weblogic - and click Log In to login.

The first thing that we need to do now is to create the managed servers. Using the Domain Structure tree navigate to Environment and then Servers. On the Summary of Servers screen Configuration tab click on the New button to create a new Managed Server.

On the Create a New Server screen specify the Server Name, Server Listen Address and Server Listen Port. For the Server Listen Address specify the address of the first physical machine. You can click Next to verify your input and finally Finish to proceed with the creation of the Managed Server.

The Administration Console proceeds with the creation of the server. When done displays a success message in the Messages area and the server is shown in the Summary of Servers table in a SHUTDOWN state.

Repeat the steps above to create the Managed Server sunning on the other physical machine. Make sure that you specify the Server Listen Address of the other physical machine.

Now that the Managed Servers have been created we will proceed with the creation of the WebLogic Machines - one for each physical machine - and assign the Managed Servers to them.

Using the Domain Structure tree navigate to Environment and then Machines. On the Summary of Machines table click on the New button to create a new WebLogic Machine.

On the Create a New Machine page specify the name of the first machine and click OK to create the Machine.

A success message in the Messages should indicate the successful creation of the Machine and the Machine will be available in the Summary of Machines table.

Now, click on the newly created Machine and then on the Node Manager tab under the Configuration tab. The node manager will be used on each physical machine to start the Managed Servers. On this tab we need to properly configure the node manager parameters. Make sure that the Type is set to Plain and that the Listen Address corresponds to the ip address of the physical machine where this Machine's node manager is running. You can leave the port to the default 5556. Don't forget to click on the Save button to save your settings.

Verify that your settings were set successful by observing the Messages area.

Now you will need to assign the appropriate Managed Server to the Machine. You do this by clicking on the Servers tab under the Machine Configuration tab and then the Add button.

In the Add a Server to Machine page ensure that the appropriate Managed Server - the one that corresponds to this Machine - is selected in the Select a server choice list. Click Next or Finish to proceed with the server assignment.

The Managed Server should be assigned successfully and should be displayed in the Servers table for this Machine.

Now repeat the steps above to create a WebLogic Machine for the second physical machine. Make sure that you configure the Machine correctly and assign the appropriate Managed Server to it.

Now with the WebLogic Machines created and configured we can start the node manager on the machine. Remember that we are still working on the same machine where we run the Configuration Wizard in the first place. It can be started by running the startNodeManager.sh script in the wlserver_10.3/server/bin directory under the Middleware Home. Before starting it, go over the start-up parameters specified in the nodemanager.properties configuration file in the wlserver_10.3/common/nodemanager directory and verify that it is not configured for SSL by ensuring that the SecureListener parameter is set to false. Once started the terminal window should indicate that the node manager is  running and listening to the port configured.

To verify that the node manager is reachable by the WebLogic Machine, return to the Machines page in the Administrator Console and on the Summary of Machines table click on the first Machine - Machine1 in this case the one we are currently configuring. Click on the Monitoring tab and observe in the Node Manager Status tab that the node manager Status is Reachable.

4. Copy the domain file structure to the other physical machine and enroll it to the WebLogic Domain

WebLogic comes with a pack utility that allows you to pack a domain and move it from one place to another. We will instead use plain old Linux tar and gzip to pack our WebLogic domain directory file structure and move it from one physical machine - the one that we have done steps 2 and 3 so far - to the other. Before doing so, we will shutdown the domain and node manager if they are still running on the first machine. Once the packing is done, we will re-start them.

To shutdown the domain, simply press Ctrl-C within the console running the domain. Alternatively, you could shutdown the  domain by shutting down the Administration Server using the Administration Console. Once the  domain is shutdown, go to the Middleware Home directory and issue the following tar command to pack the domain directory file structure and contents, i.e. the user_projects directory: tar cvf user_projects.tar user_projects

When the tar command is done, zip the archive by typing: gzip user_projects.tar. The file that needs to be moved and unpacked on the other machine is called user_projects.tar.gz

The compressed archive of the user_projects domain file structure should be moved to the Middleware Home on the other physical machine - remember that the Middleware Home directories on both physical machines should be exactly the same - and uncompress by typing gunzip user_projects.tar.gz

Finally extract the archive by typing tar xvf user_projects.tar. This will create the user_projects directory structure on the second physical machine.

Now you need to re-start the domain - on the first (original) machine - and use the  WebLogic Scripting Tool - on the second machine - to enroll the domain structure - the one we just extracted - to the WebLogic Domain. So, after making sure that the domain is started and is in RUNNING state, issue the following commands on the second machine - the one we are now setting up - to enroll it to the WebLogic Domain:

Start the WebLogic Scripting Tool by changing to the wlserver_10.3/common/bin directory and typing wlst. On the wlst prompt issue the following command to connect to the WebLogic Domain running on the other machine: connect('weblogic','weblogic1','t3://').

Verify that the connection was successful and then issue the following command to enroll this machine to the WebLogic Domain:

nmEnroll('/home/weblogic/Oracle/Middleware/user_projects/domains/clusteredDomain', '/home/weblogic/Oracle/Middleware/wlserver_10.3/common/nodemanager')

Verify that the machine was successfully enrolled into the Domain.

5. Ensure that both WebLogic Machines are operational

Now that the second machine was successfully enrolled into the WebLogic Domain, we can start the node manager on the second machine and verify that it is Reachable by the WebLogic Machine. We have already verified that the node manager on the first machine is Reachable back in step 3.

You can start the node manager by running the startNodeManager.sh script in the wlserver_10.3/server/bin directory. Again, ensure that the node manager is not started in SSL mode by setting the SecureListener parameter to false in the nodemanager.properties configuration file in wlserver_10.3/common/nodemanager. Note that nodemanager.properties might not exist until you first run startNodeManager.sh.

Now, with the node managers running on both physical machines return to the Administration Console and verify that the node managers are Reachable for both Machines.

6. Create and configure the WebLogic Cluster

To create a WebLogic Cluster use the Administration Console and via the Domain Structure tree navigate to Environment and then Clusters. On the Summary of Clusters table click the New button to create a new WebLogic Cluster.

On the Create a New Cluster page enter the Name of the Cluster, select Unicast for the Messaging Mode and click OK.

Ensure that the WebLogic Cluster is created successfully by observing the messages on the top Messages area and the new Cluster appearing in the Clusters table. Now, click on the newly created Cluster to go to the Settings for Cluster page. On the Configuration General tab on the Cluster Address field enter the IP addresses of the physical machines separated by comma(s) supplying 7003 as the cluster listen port, e.g., in this example. Click Save to save the changes.

While still on the Settings for Cluster page, go to the Configuration Servers tab and click Add on the Servers table to assign the Managed Servers to the Cluster.

On the Add a Server to Cluster page select the Managed Servers - one at a time - to add to the Cluster using the Select a server drop down and click Finish.

Repeat this step for both Managed Servers. In the end both Managed Servers should be shown on the Servers table for the Cluster.

7. Ensure that the Cluster is operational

Now that the Cluster is setup we can start the Managed Servers and do some monitoring on the Cluster to ensure its proper operation. You can start the Managed Servers either by running the startManagedWebLogic.sh script in the user_projects/domains/clusteredDomain/bin directory - for each physical machine - or from within the Administration Console. We will use the Administration Console to do so, which will at the same time validate our setup. So, go to the Summary of Servers page - via the Domain Structure tree by clicking on Environment and then Servers - and click on the Control tab. Click on the  checkboxes to select both Managed Servers - notice that they are both in SHUTDOWN state - and click the Start button to start them.

The Administration Console will delegate the start-up process to the node managers running on both machines. Each node manager will start the Managed Server assigned to the Machine controlled by the node manager. The process might take a minute or so to complete and once done both Managed Servers should be displayed on the Servers table with a RUNNING state. The Status of Last Action should be TASK COMPLETED. You will need to periodically refresh the page to see the final start-up status.

Now with both Managed Servers running, let's take a look at our Cluster status. Return to the Summary of Clusters page - via Environment and then Clusters on the navigation tree - and click on the Cluster in the Clusters table. Then click on the Monitoring tab and observe the Cluster Server status on the Summary tab.

Click on the Health and Failover tabs to see other monitoring information and status.

We have completed Oracle Weblogic clustering on different hosts, I hope this will help you in Clustering.

Enjoy working in middleware and any queries, suggestions and any mistakes, kindly let me know via comments,

Ajinkya Vichare


  1. Thanks a lot :) for the excellent HOWTO.....

  2. @tavorserenase Hope it helps you....

    ALL the best.... Enjoyy working with Middleware


  3. really it is good for MW guys.

  4. @Anonymous..

    Thanks for reading my blogs and appreciation..

    Enjoy working with Middleware.


  5. Thank you so much....
    keep working with middle-ware technology... :)

  6. Good reference, thanks for sharing

  7. This comment has been removed by the author.

  8. Hi,

    its a good reference. got a clear picture. However i got some problem here. I'm setting up note just 2 machine but 4 machine with 1 admin server. I manage to get the all 4 nodemanager reachable from the admin server but failed to start the managed server using admin. Do you have any idea?

    1. Hi Thanks for referring my blog..

      kindly let me know how you are starting your managed servers.


    2. Start the managed server from Admin Server. Tried restarting the managed server manually on the remote host but cannot because it ask me for the credentials.

    3. Hi Fadliezzati,

      You can start your managed Server from the same host with help of Admin Server.

      You can start your managed server like :

      startManagedServer.sh MNG_ServerName http://adminServer_host:admin_port

      I hope it will help you in starting your managed Server.

      Note : Kindly do nmConnect from all the host.

    4. nmConnect or nmEnroll?

    5. To setup more than 1 remote managed server in same domain, simply do the trick above on all of the hosts? No other extra step need to be done?


    6. we need to Node manager connect and Enroll for all weblogic domains.

      to setup more than 1 remote managed server in same domain, no need to do any extra step for the same.

      You just need to add in cluster.


  9. When machine 1 is down which means admin console is down and machine 2 is up, whether applications will is running up or down?

    1. Hi,

      There will not be any issue if your Machine 1 is down, your application will start as it is.

      But if you want to start your managed Server then you need to have admin server UP.


  10. I tried the same setup in Windows machine and done with all the steps and I am not able to start the managed server in the second machine.

    Server 1 -, where I have the admin server running on port 7001 and managed server on port 7003
    Server 2 - - Managed Server running on port 7003

    The Node Managers are in Reachable status on both the machines.

    <[STANDBY] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)'> <> <> <> <1332838725999>

    <> <> <> <1332838725999>

    Any help will be of great support.

  11. <[STANDBY] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)'> <> <> <> <1332838725999>

    <> <> <> <1332838725999>

    Any help will be of great support.

    Pasting again for the rest of the stacktrace

  12. Starting "async" replication service with remote cluster address ","
    ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)' BEA-090087 Server failed to bind to the configured Admin port. The port may already be used by another process

    BEA-002606 Unable to create a server socket for listening on channel "Default". The address might be incorrect or another process is using port 7003: java.net.BindException: Address already in use: JVM_Bind

  13. Hi ,

    I couldn't wait to send my appreciations for this wornderful article.

  14. Ajinkya,
    This is really a wonderful article and I am referring this as a bible in setting up my environment.
    My requirements are also same, I have 2 physical machines (Windows 7) and have same weblogic versions (10.3.5) installed. First machine will have admin and one managed server and another machine will host another managed server and all belong to the single domain.

    After following all the steps as per your blog, I could able to start admin and managed server successfully without any issue. But when I try to start another managed server (of different machine) it was not getting started successfully. I am experiencing below error

    <[STANDBY] ExecuteThread: '1' for queue: 'weblogic.kernel.Default (self-tuning)'> <> <> <> <1363675381109>

    I did lot of search and research but unable to find a solution. Can you please help.

    1. Hi Shrikant,

      Greetings @@@@

      Thank you so much for reviewing my blog and congratulations for your cluster implementation.

      If you can then drop me managed server log on middlewaresolution1@gmail.com

      Will update you with solution ASAP.


    2. Ajinkya,
      Thanks for your quick reply. I have sent the log file.

  15. Hi,

    Thank you for your blog post, that has given a ray of hope.

    I have 2 separate installations of weblogic server on two separate hosts.
    Each has its own Admin Server, and the Middleware home has exactly the same structure.
    Each physical server already has weblogic server cluster setup (Containing 2 managed servers each for SOA and BAM).

    Will I be able to deploy to the two separate servers at deployment time?
    If a physical server goes down, how will the failover work out?

    I am in a great fix.
    Kindly assist.

  16. Please post the step by step procedure for configuring the cluster on a single machine

  17. Hi Ajinkya,

    We have cluster members hosted on two different machines.
    We faced an issue couple of days back where one of the host itself wend down(hosting 2 managed servers which are the part of the cluster and the Admin Server) and the application became unavailable; though we had the other host running perfectly fine(hosting other two members of the cluster)

    And when the first host came up and we started all the weblogic services there; we noticed that the other host also started responding or serving application requests(checked with IP and Port combination).

    Means to say that we didn't touch the other host during the host 1 downtime, but it was not serving the request but when the host one came up with all the weblogic services up and running; the other node also started working.

    Please help me out resolving this issue.

    Thanks in advance.

    Mudit Nigam