Enterprise Manager Appears To Hang While Displaying the List of SOA Composites on Dashboard

Hi Everyone,

Best Wishes..@@@@

From last two days was working on issue where EM performs very poor or it appears hang while displaying SOA composites application and services.

Scenario :

We have installed of Oracle SOA suite 11g where we are running a large volume of BPEL processes over time. Initially it was working fine but  after few days When we tried to access the dashboard page displayed by Enterprise Manager for one or more composites, the regions that display list views of composite's instances take a long time to render and during this time Enterprise Manager remains hung and unresponsive for other tasks.

At the time when the performance of Enterprise Manager is impacted, you will likely have one or more composites that have a reasonably large number of composite instances defined  and the performance impact may be most noticeable the first time a dashboard page is accessed following a restart of the AdminServer.

In severe cases you may be experiencing very long delays when using Enterprise Manager to accessing the dashboard pages of these composites.

We have found cause for the issue as below:

As everybody knows SOA has it’s own metadata store, at the database layer the SQL query that Enterprise Manager executes to obtain the data displayed on the dashboard is taking far longer than expected to execute, as a result of a less than optimal execution plan having been determined by the Cost Based Optimizer component of the database.

The execution plan identified by the Cost Based Optimizer is, in turn, driven by information about the various tables and column indexes that is maintained by database statistics.

To reduce processing overhead while creating statistics, the statistics for large tables are not collected from all rows of the table but using a sampling technique on a smaller subset (by default, 20%).

As a result of this sampling based approach, over time the statistics gathered against the sample may not be representative for the larger data set as a whole. When such a discrepancy occurs the unrepresentative statistics can lead the Cost Based Optimizer to identify a bad execution plan that is less efficient on data access and takes significantly longer to complete. In such cases, this causes the source of the long running SQL which so significantly impacts the performance of the list views on Enterprise Manager dashboard pages.

Solution :


Step 1. Login to the SOA 11g EM Console

 
Step 2. Right Click on SOA-INFRA and go to SOA ADMINISTRATION->COMMON PROPERTIES

 
Step 3. Enable the Disable fetching of instance and fault count metrics as shown below.

 
Step 4. Set the Duration Details to 24 or 48 hours as shown below.

Step 5. Now log back to EM console , check EM performance its faster.

It Works for me hope it will work for you as well....

Again thanks to Oracle Metalink for such wonderful ID : 1351032.1.

For Any queries you can write me on middlewaresolution1@gmail.com.

Enjoy working With Middleware Technologies.

Regards,

Ajinkya Vichare

ajinkya-vichare.blogspot.com



 

Different ways to take thread dumps in WebLogic Server

Dear All, 

Best Wishes… @@@@ Hope Everyone doing well..

Today we will talk about different ways to take thread dumps in Weblogic server environment.

Thread dumps are essential diagnosis information used to analyze and troubleshoot performance related issues such as server hangs, deadlocks, slow running, idle or stuck applications, slow database interactions etc...

WebLogic Server (WLS) and Java offer several ways to generate thread dumps, they are detailed below. It is always recommended to obtain the thread dumps by using operating system (OS) commands rather than by using Java classes or the Administration Console, because if the console is hanging, users won't be able to connect to it to issue thread dumps.

Option 1) Use operating system commands to get the thread dumps when WLS starts up from a command-line script:

On Windows OS, thread dumps can be created by

<ctrl>+<break> -- the thread dumps are generated in the server stdout

On POSIX-compliant platforms (e.g. Solaris and Linux), first identify the process ID (pid) using

 ps -ef | grep java, then run

kill -3 <pid> 2>&1

Signal 3 is equivalent to SIGQUIT. Note that in Solaris, the thread dump is generated in the current shell, but in Linux, the thread dump is generated in the shell which started the java process specified by the pid.

Option 2) Using beasvc (up to WLS 10.3.5 included):

beasvc -dump -svcname:<service_name>
[Note: service_name is the Windows service that is running the server instance (e.g. mydomain_myserver)]

Option 3) Using wlsve (from 10.3.6/12.1.1):

wlsve -dump -svcname:<service_name>

Option 4) Using weblogic.WLST:

setDomain.cmd or setDomain.sh depending on the OS
 

java weblogic.WLST 

connect("<username>","<password>","t3://<url>:<port>")
threadDump()

[Note : The thread dump will be generated in Thread_Dump_AdminServer.txt.]

Option 5) From a command line or shell, a thread dump can be generated via the following command (deprecated from WLS 9.0):
 

setDomain.cmd or setDomain.sh depending on the OS

java weblogic.Admin <url>:<port> -username <username> -password <password> THREAD_DUMP

[Note : The thread dump will be generated in the defined server stdout.]

Option 6) From the WLS Administration Console, a thread dump can be created by navigating to

Server -> <server_name> -> Monitoring -> Dump threads stack.

This method could lead to truncated or incomplete thread dumps.

Option 7) From the JRockit command line:

jrcmd <pid> print_threads

All are the methods or ways to generate thread dump in weblogic. We have several tools to analyse those tools.

Best WebLogic (JVM ) Thread Dump analyzer are:

• Samurai
• Java TDA
• Thread Dump Viewer
• Eclipse lock analyzer plugin

Hope it will help you in your day to day activities. Feel free to write me on middlewaresolution1@gmail.com.

Enjoy Working with middleware technologies.

Regards,
Ajinkya Vichare
Ajinkya-vichare@blogspot.com

Tuning the default WorkManager on WebLogic Server

Hi Everyone,

Best Wishes...@@@ Hope you all are doing well ..... @@@

Today one of my friend came across scenario where he needs to tune weblogic default work manager. so today i am sharing few steps for tuning default work managers.

In my last post we have got brief idea of Work managers, now in this post we will talk about tuning of default work managers.

In WLS 8, we had the concept of Execute Queues, replaced by work-managers, starting with WLS 9. Well, it's got self-tuning, allows a large number of SLA combinations.

So lets Play with the default work-manager :)

The default work-manager, as its name tells, is the work-manager defined by default.Thus, all applications deployed on WLS will use it.

But sometimes, when your application is already in production, it's obvious you can't take your EAR / WAR, update the deployment descriptor(s) and redeploy it.

The default work-manager belongs to a thread-pool, as initial thread-pool comes with only five threads, that's not much.

If your application has to face a large number of hits, you may want to start with more than that.Well, that's quite easy. You've got two ways to do so.

Option 1) Modifying the config.xml

Just add the following line(s) in your server definition :

    <server>

    <name>AdminServer</name>

    <self-tuning-thread-pool-size-min>100</self-tuning-thread-pool-size-min>

    <self-tuning-thread-pool-size-max>200</self-tuning-thread-pool-size-max>

    [...]

    </server>

Option 2) Adding some JVM parameters

Personally, that's my favorite option since I don't like when people go in a generated config file

and mess up with tags. If you're sure of what you're doing, that's ok, but it's safer the following way :

add the following option in your command line : -Dweblogic.threadpool.MinPoolSize=100

Personally, I decided to apply the change by adding the JVM parameter.

To do so, I just updated the startWeblogic (in Domain_home) script with the USER_MEM_ARGS line.

@REM WARNING: This file is created by the Configuration Wizard.
@REM Any changes to this script may be lost when adding extensions to this configuration.

SETLOCAL

set DOMAIN_HOME=F:\Oracle\Middleware\user_projects\domains\base_domain
set USER_MEM_ARGS=-Dweblogic.threadpool.MinPoolSize=100
call "%DOMAIN_HOME%\bin\startWebLogic.cmd" %*

ENDLOCAL

(Note : I'll recommend use Option 2 )


Reboot WLS and see the option has been taken into account  :) 

Wasn't so hard, was it ? :) 

feel free to write me on middlewaresolution1@gmail.com.

Enjoy Working with Middleware technologies.

Regards,

Ajinkya Vichare

ajinkya-vichare.blogspot.com

Introduction to Work Managers

Hi Everyone,

Best Wishes..  Hope all are doing well..!!

Today lets talk about work managers in oracle Weblogic server.

In earlier version there is concept called execute threads, BEA and IBM has come up with concept called work managers, we will check both concept one by one.

Execute Queues:

The main purpose of having the Execute Queues is to handle the request internal and external related to weblogic server. Here, external requests could be the requests related to:-

1)  Applications deployed inside the weblogic server

2)  Redirection requests from weblogic server to the database, etc

Internal requests could be either the communication between the managed servers or between the managed server and admin server, etc.

These requests are mainly handled using the threads. Hence, we can conclude that the Execute Queues are mainly used for thread management. This management is achieved by using the using the different queues like weblogic.kernel.default, weblogic.admin. HTTP, weblogic.admin.RMI, etc.

Work Managers:

 With the release of the new versions of weblogic server and new advancement the execute queues or the thread management in weblogic server also had the enhancements. From Weblogic Server 9x we had a replacement done to Execute Queues with the introduction of Work Managers.

With introduction of Work Managers, weblogic server helps to prioritize work and allocate the threads based on an execution model. The is taken care by the administrator-defined parameters after observing the actual run-time performance, throughput, monitoring, etc. Defining of this behaviour or rules is done at the different levels differently for differently for different applications, group of applications, etc as per the requirement. We can define one or more of the following Work Manager components in our deployment descriptor or config files:

1) Fair Share Request Class:

2) Response Time Request Class:

3) Min Threads Constraint:

4) Max Threads Constraint:

5) Capacity Constraint

6) Context Request Class

Work Managers can be configured either at the domain level, application level, module level in one of the following configuration files:

a) config.xml — Work Managers specified in config.xml can be assigned to any application, or application component, in the domain. You can use the Administration Console to define a Work Manager.

b) weblogic-application.xml — Work Managers specified at the application level can be assigned to that application, or any component of that application.

c) weblogic-ejb-jar.xml or weblogic.xml — Work Managers specified at the component-level can be assigned to that component.

d) weblogic.xml — Work Managers specified for a Web Application

The scope of the work managers could be any of the following three:

1) The default Work Manager

2) Global Work Managers

3) Application-scoped Work Managers

Default Work Manager:

These are used to handle thread management and perform self-tuning.This Work Manager is used by an application when no other Work Managers are specified in the application’s deployment descriptors.

In many situations, the default Work Manager may be sufficient for most application requirements. WebLogic Server’s thread-handling algorithms assign each application its own fair share by default. Applications are given equal priority for threads and are prevented from monopolizing them.

Global Work Managers:

These have the scope of the domain. Global Work Managers are created in the WebLogic Administration Console using the following steps:

Step a) Login to WebLogic Admin Console and click on the left Panel “Work Manager”.

Step b) Then click on “Lock and Edit” button and click on “New” button from right panel

Step c) Select “Work Manager” radio button from Select Work Manager Definition type then click on Next.

Step d) Give the work Manager Name and click Next.

Step e) Now Target it to any Server or Cluster.

Step f) Now click on “Finish” and then “Activate Changes”

These are then defined in config.xml.

An application uses a globally defined Work Manager as a template. Each application creates its own instance which handles the work associated with that application and separates that work from other applications. This separation is used to handle traffic directed to two applications which are using the same dispatch policy. Handling each application’s work separately, allows an application to be shut down without affecting the thread management of another application. Although each application implments its own Work Manager instance, the underlying components are shared.

Application-scoped Work Managers:

 In addition to globally-scoped Work Managers, you can also create Work Managers that are available only to a specific application or module. Work Managers can be specified in the following descriptors:

a) weblogic-application.xml

b) weblogic-ejb-jar.xml

c) weblogic.xml

If you do not explicitly assign a Work Manager to an application, it uses the default Work Manager.

you can also check following URL for more details

http://docs.oracle.com/cd/E11035_01/wls100/config_wls/self_tuned.html

Feel free to write me on middlewaresolution1@gmail.com.

Enjoy working with Middleware technologies..

Regards,

Ajinkya Vichare 

ajinkya-vichare.blogspot.com

 

how to setup JConsole for Oracle Oracle WebLogic Server

Hi Everyone,

We all know JConsole is part of JDK, it can be found in $JAVA_HOME/bin directory, and this utility can be used as JMX Console for Managing WebLogic Server, Where you can see the WebLogic Server configuration and invoke various MBean operations, it helps to view and edit the MBean configuration and invoke various JMX Operations on the serverRuntime, serverConnfig and domainRuntime

Here in this post we will see how to setup JConsole for WebLogic Server 10g.

As JConsole is free and distributed with JDK, there is no need of downloading it seperately

We will follow simple steps to setup the JConsole for WebLogic Server

JConsole works in two modes :

1. Local Process Mode -> Checks for Local Java Process to Manage them through console
2. Remote Mode ->We Can provide JMX URL to Manage the Remote Java Process

We Normally Use Remote mode as we need to connect the JConsole to MBean Server of WebLogic Application Server

Step 1. Start WebLogic Admin Server

 

Step  2. Login to Admin Console and check the domain configuration

Fig 2. Check Domain's Advanced Configuration as The MBean Servers are enabled or not

Step 3. Check for the Parameters "Compatibility MBean Server Enabled" and "Management EJB Enabled" Status ( They Should be Enabled - By Default They are enabled)

 

Step 4. Check The Configuration at Admin Server Side in Ports Tab - > IIOP

Fig 4. IIOP Should be enabled in Admin Server IIOP Port Configuration

Fig 5. Provide some user name and password (preferable weblogic/weblogic), as Anonymous users are not allowed to connect to the JMX MBean Server (Advanced Options in IIOP)

Step 5. Save and Activate Changes - This Needs Restart of the Admin Server.

Notice that the logs at the startWebLogic.cmd command window shows as below

The non-dynamic attribute DefaultIIOPPasswordEncrypted on weblogic.management.configuration.ServerMBeanImpl@453b029e([ProdDomain]/Servers[AdminServer]) has been changed. This may require redeploying or rebooting configured entities

change has been made which affects the server AdminServer. This server must berebooted in order to consume this change

Step 6. Restart the Admin Server

Step 7. Start JConsole - Execute following Steps and Verify the output

F:\>echo %WL_HOME%
F:\Oracle\Middleware\wlserver_12.1

F:\>echo %JAVA_HOME%
F:\Oracle\Middleware\jdk160_29\bin

 
F:\>cd %WL_HOME%

F:\Oracle\Middleware\wlserver_12.1>cd server

F:\Oracle\Middleware\wlserver_12.1\server>cd bin

F:\Oracle\Middleware\wlserver_12.1\server\bin>setWLSEnv.cmd


Step 8. Execute the following command to invoke "jconsole remote"

jconsole -J-Djava.class.path=%JAVA_HOME%\lib\jconsole.jar;%WL_HOME%\server\lib\wljmxclient.jar -J-Djmx.remote.protocol.provider.pkgs=weblogic.management.remote

Step 9. We will get to see the console as below.

 

Fig 9 . JConsole Login Screen

Step 10. Please provide Service JMX URL for Admin Server and Specify the MBean Tree to enter as below and their particular JConsole Space.


1. service:jmx:rmi:///jndi/iiop://localhost:7011/weblogic.management.mbeanservers.domainruntime


username/password = weblogic/weblogic

 3. service:jmx:rmi:///jndi/iiop://localhost:7011/weblogic.management.mbeanservers.edit
username/password = same as above

Fig 12. JConsole Edit Session in Edit Tree.

This is it..!!

Please let me know if you have any feed back suggestion here


Thanks for Visiting My Blog - Please Share your feed back and ideas.

Enjoy working with middleware....

Regards,

Ajinkya Vichare

ajinkya-vichare.blogspot.com