Dear All,
Y'day i came across one strange issue it might well known issue but i faced first time.
Whenever am trying to access Oracle Weblogic Serer Administration console Some hanging has been reported.
I tried many things to solve this issue, but finally i took thread dump and have found below that some of the thread getting blocked.
"[ACTIVE] ExecuteThread: '15' for queue: 'weblogic.kernel.Default (self-tuning)'"
id=62 idx=0xf0 tid=23248 prio=5 alive, in native, blocked, daemon
-- Blocked trying to get lock: weblogic/servlet/internal/WebAppServletContext@0x4332e7b0[fat lock]
at jrockit/vm/Threads.waitForUnblockSignal()V(Native Method)
at jrockit/vm/Locks.fatLockBlockOrSpin(Locks.java:1675)[optimized
at jrockit/vm/Locks.lockFat(Locks.java:1776)[optimized]
at jrockit/vm/Locks.monitorEnterSecondStageHard(Locks.java:1312)[optimized]
at jrockit/vm/Locks.monitorEnterSecondStage(Locks.java:1259)[optimized]
at jrockit/vm/Locks.monitorEnter(Locks.java:2456)[optimized]
at weblogic/servlet/internal/ServletStubImpl.onAddToMapException(ServletStubImpl.java:361)
at weblogic/servlet/internal/ServletStubImpl.execute(ServletStubImpl.java:318)
And the object WebAppServletContext is waiting for a bootstrap method.
"[ACTIVE] ExecuteThread: '12' for queue: 'weblogic.kernel.Default (self-tuning)'" id=59 idx=0xe4 tid=23245 prio=5 alive,
in native, waiting, daemon
-- Waiting for notification on: java/lang/Object@0x44116170[fat lock]
at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
at java/lang/Object.wait(J)V(Native Method)[optimized]
at weblogic/rjvm/ConnectionManager.bootstrap(ConnectionManager.java:401)
^-- Lock released while waiting: java/lang/Object@0x44116170[fat lock]
at weblogic/rjvm/RJVMImpl.ensureConnectionEstablished(RJVMImpl.java:303
This is because server is calling a bootstrap method and threads are getting blocked, waiting for another server's heartbeat response until the incoming response is notified to the blocked thread.
After doing so many things and searching on Google and Oracle Metalink, got an solution which works for me.
Solution
There is a timeout set for bootstrap methods by the bootstrap Wait Period, which is the heartbeat period length of servers. By default, the bootstrap wait period is 60 seconds.
To decrease the delay in the admin console, decrease this wait period, for example, to 30 seconds.
From the admin console, navigate to
Server name -> Configuration -> Tuning -> Advanced -> Period Length
and set the desired value in seconds.
I hope it will help you in such situation.
Enjoy Working with Middleware technologies.
Regards,
Ajinkya Vichare
ajinkya-vichare.blogspot.com
hi.
ReplyDeleteam facing the error with user has the insufficient permission to access the ejb? this error found in stderr log file. let me know the solution for this..
Thank you,
Hi,
ReplyDeletePlease let me know following details :
- Operating system :
- Weblogic Version :
Regards,
Ajinkya
Thanks for the helpful info. Ajinkya !! You have a good day !!
ReplyDeleteHi Ajinkya, I have similar problem. WL console hangs and does not respond. If I kill the java process and restart the admin server, the console comes back on but again after some time I cannot open the console. I cannot see any error in the log files, also the log does not get updated upto the current time. I think it is updated upto the time when the server hangs. Can you please help me to fix this. I know to take thread dump but not sure how to use. I use kill -3 PID, the thread dump is created in AdminServer.out file. How do I analyse and find out what is causing the console to hang. How many thread dumps should be taken and after how much time gap. Which tool to be used to analyse thread dump. I am using WL 10.3.5 on Solaris 10. Many thanks, Harish.
ReplyDeleteHi Harish,
DeleteBest Wishes..@@@@
We need to understand few things to solve this issue :
1) physical resources of your server / Machine
2) Review logs and Analysis of thread dump
You can take thread dump at time of your hang issue, take 5-6 thread dump at 30 Seconds interval. You can analyze those thread dump with the help of thread analyzer or JCA tool.
Any further help you can E-mail me all above mentioned details on middlewaresolution1@gmail.com.
Regards,
Ajinkya Vichare
ajinkya-vichare.blogspot.com
I think I am having the same issue as Harish,the frustrating part is there is no log to point me to any direction. my guess is my machines resources are exhausted , any way to find this one out?
ReplyDeletemy email - patil.deepac@gmail.com
Dear Deepac Patil,
DeleteWhat is your system configuration and weblogic version?
what component of fusion middleware you are using?
Would like to help you if you still have the same issue.
You can write me more on middlewaresolution1@gmail.com
Regards,
Ajinkya