Enterprise Manager Appears To Hang While Displaying the List of SOA Composites on Dashboard

Hi Everyone,

Best Wishes..@@@@

From last two days was working on issue where EM performs very poor or it appears hang while displaying SOA composites application and services.

Scenario :

We have installed of Oracle SOA suite 11g where we are running a large volume of BPEL processes over time. Initially it was working fine but  after few days When we tried to access the dashboard page displayed by Enterprise Manager for one or more composites, the regions that display list views of composite's instances take a long time to render and during this time Enterprise Manager remains hung and unresponsive for other tasks.

At the time when the performance of Enterprise Manager is impacted, you will likely have one or more composites that have a reasonably large number of composite instances defined  and the performance impact may be most noticeable the first time a dashboard page is accessed following a restart of the AdminServer.

In severe cases you may be experiencing very long delays when using Enterprise Manager to accessing the dashboard pages of these composites.

We have found cause for the issue as below:

As everybody knows SOA has it’s own metadata store, at the database layer the SQL query that Enterprise Manager executes to obtain the data displayed on the dashboard is taking far longer than expected to execute, as a result of a less than optimal execution plan having been determined by the Cost Based Optimizer component of the database.

The execution plan identified by the Cost Based Optimizer is, in turn, driven by information about the various tables and column indexes that is maintained by database statistics.

To reduce processing overhead while creating statistics, the statistics for large tables are not collected from all rows of the table but using a sampling technique on a smaller subset (by default, 20%).

As a result of this sampling based approach, over time the statistics gathered against the sample may not be representative for the larger data set as a whole. When such a discrepancy occurs the unrepresentative statistics can lead the Cost Based Optimizer to identify a bad execution plan that is less efficient on data access and takes significantly longer to complete. In such cases, this causes the source of the long running SQL which so significantly impacts the performance of the list views on Enterprise Manager dashboard pages.

Solution :


Step 1. Login to the SOA 11g EM Console

 
Step 2. Right Click on SOA-INFRA and go to SOA ADMINISTRATION->COMMON PROPERTIES

 
Step 3. Enable the Disable fetching of instance and fault count metrics as shown below.

 
Step 4. Set the Duration Details to 24 or 48 hours as shown below.

Step 5. Now log back to EM console , check EM performance its faster.

It Works for me hope it will work for you as well....

Again thanks to Oracle Metalink for such wonderful ID : 1351032.1.

For Any queries you can write me on middlewaresolution1@gmail.com.

Enjoy working With Middleware Technologies.

Regards,

Ajinkya Vichare

ajinkya-vichare.blogspot.com



 

Different ways to take thread dumps in WebLogic Server

Dear All, 

Best Wishes… @@@@ Hope Everyone doing well..

Today we will talk about different ways to take thread dumps in Weblogic server environment.

Thread dumps are essential diagnosis information used to analyze and troubleshoot performance related issues such as server hangs, deadlocks, slow running, idle or stuck applications, slow database interactions etc...

WebLogic Server (WLS) and Java offer several ways to generate thread dumps, they are detailed below. It is always recommended to obtain the thread dumps by using operating system (OS) commands rather than by using Java classes or the Administration Console, because if the console is hanging, users won't be able to connect to it to issue thread dumps.

Option 1) Use operating system commands to get the thread dumps when WLS starts up from a command-line script:

On Windows OS, thread dumps can be created by

<ctrl>+<break> -- the thread dumps are generated in the server stdout

On POSIX-compliant platforms (e.g. Solaris and Linux), first identify the process ID (pid) using

 ps -ef | grep java, then run

kill -3 <pid> 2>&1

Signal 3 is equivalent to SIGQUIT. Note that in Solaris, the thread dump is generated in the current shell, but in Linux, the thread dump is generated in the shell which started the java process specified by the pid.

Option 2) Using beasvc (up to WLS 10.3.5 included):

beasvc -dump -svcname:<service_name>
[Note: service_name is the Windows service that is running the server instance (e.g. mydomain_myserver)]

Option 3) Using wlsve (from 10.3.6/12.1.1):

wlsve -dump -svcname:<service_name>

Option 4) Using weblogic.WLST:

setDomain.cmd or setDomain.sh depending on the OS
 

java weblogic.WLST 

connect("<username>","<password>","t3://<url>:<port>")
threadDump()

[Note : The thread dump will be generated in Thread_Dump_AdminServer.txt.]

Option 5) From a command line or shell, a thread dump can be generated via the following command (deprecated from WLS 9.0):
 

setDomain.cmd or setDomain.sh depending on the OS

java weblogic.Admin <url>:<port> -username <username> -password <password> THREAD_DUMP

[Note : The thread dump will be generated in the defined server stdout.]

Option 6) From the WLS Administration Console, a thread dump can be created by navigating to

Server -> <server_name> -> Monitoring -> Dump threads stack.

This method could lead to truncated or incomplete thread dumps.

Option 7) From the JRockit command line:

jrcmd <pid> print_threads

All are the methods or ways to generate thread dump in weblogic. We have several tools to analyse those tools.

Best WebLogic (JVM ) Thread Dump analyzer are:

• Samurai
• Java TDA
• Thread Dump Viewer
• Eclipse lock analyzer plugin

Hope it will help you in your day to day activities. Feel free to write me on middlewaresolution1@gmail.com.

Enjoy Working with middleware technologies.

Regards,
Ajinkya Vichare
Ajinkya-vichare@blogspot.com