Thursday, September 29, 2011

Coherence*Web with WebLogic Server Simple Example and Troubleshooting Overview

This blog was originally posted on http://blogs.oracle.com/jamesbayer when I worked for Oracle.
This quick troubleshooting guide covers a basic setup of a simple sample application using Coherence*Web instead of traditional WebLogic Server session replication. I used Coherence 3.6.1.x and WebLogic Server 10.3.4 for when I originally created this example, but the steps should be similar for any Coherence 3.5+ and WebLogic Server release capable of using the Coherence*Web SPI, which include some older releases of WLS, but consider 10.3.1+ as the most well-worn-path.  I apologize for the formatting, which is a bit sloppy compared to my normal blog posts.  This has been in a word doc on my desktop for a long time waiting for when I had time to post it properly.  Given two requests for this in the last week I figured a sloppy blog post was better than not sharing it broadly.
For simplicity, we are assuming 2 WebLogic Server managed server instances in a WLS Cluster are running Coherence*Web as storage disabled nodes. The WebLogic Servers will have a simple test web application located here deployed that stores session data on a Coherence cluster with 2 storage enabled cache servers. There are a sequence of steps that help validate that session failover is working properly. Some of these steps are from product documentation, and I've supplemented with my own comments in red. Note that this example will not use Active Cache, which is a way to manage Coherence configuration and Cache Server lifecycle from WebLogic Server. I recommend users understand this example first before adding on Active Cache. More detail on Active Cache including a video is covered in this blog entry http://blogs.oracle.com/jamesbayer/entry/weblogic_server_1034_screencas
2.2.6.2 Configuring EAR-Scoped Cluster Nodes
To use Coherence*Web for session management in EAR-scoped cluster nodes, follow these steps:
  1. Use the WebLogic Server Administration Console to deploy the coherence.jar and coherence-web-spi.war files as shared libraries to all of the target servers where the application will be deployed. See "Install a Java EE Library" in Oracle Fusion Middleware Oracle WebLogic Server Administration Console Help for more information.
I always recommend using a shared library to include the coherence.jar on the classpath of your application instead of the approach where it is embedded directly inside your application (say in APP-INF/lib or WEB-INF/lib). This way if you patch or upgrade Coherence, the most you need to do is redeploy the coherence.jar shared library and your application. If you use the approach where your application directly embeds coherence.jar inside of it, then you need a new build of your application which updates the embedded coherence.jar, which is typically an undesirable process.
Note: The WLS 10.3.1 Administrative Console will not let you deploy coherence.jar without an error, but you can use the WLST command like:
deploy('coherence','/scratch/jbayer/coherence/lib/coherence.jar', libraryModule='true')
This console deployment issue does not affect WLS 10.3.2+. However, you may note what appears to be an error when deploying coherence.jar, but if you read it closely "Assuming this is a library deployment." you will see that it is not an error, it is a warning and you can proceed with the deployment..




Assuming you are using an EAR, create a library reference to coherence in your sample application. Assuming you have an EAR file with a WAR module inside, then weblogic-application.xml looks like this:
<weblogic-application ...>
...
<library-ref>
<library-name>coherence</library-name>
</library-ref>
...
</weblogic-application>
The above library-ref element will merge the coherence.jar into the APP-INF/lib of the App during the deployment process, which will make Coherence classes EAR scoped.
Quick aside, If you are using a stand-alone WAR file that is not encapsulated inside of an EAR, then instead of using a library reference, you can use an Optional Package approach to refer to the library, which involves creating an entry in the META-INF/manifest.mf of the WAR file like this:
Manifest-Version: 1.0
Class-Path:
Extension-List: coherencejar
coherencejar-Extension-Name: coherence
Note that there must be a newline at the end of the last line with text otherwise it will not work correctly. Unlike the library reference mechanism, the library is not merged into an archive on disk at deployment time, rather it utilizes a Java Optional Package approach to put the coherence.jar on the classpath.
For the WAR modules, have the weblogic.xml use a library reference for the coherence-web-spi.war like this:
<wls:weblogic-web-app ...

<wls:library-ref>

<wls:library-name>coherence-web-spi</wls:library-name>

</wls:library-ref>
</wls:weblogic-web-app>
This will merge the contents of the coherence-web-spi.war with the application's web module.
Now we also have to configure the tangosol-coherence-override.xml file if you want to override default configuration settings. One such setting I like to override is to use Well-Known-Addresses (WKA), which is a unicast approach instead of the default multicast networking that many Network Administrators do not permit. The tangosol-coherence-override.xml file approach will be easier than using -D options on the command line because in WLS there may be multiple web applications using different Coherence configurations, and if you use -D options on the Java command line, then it is the same configuration for all Coherence instances in that JVM without some other tedious intervention. I used the latest release of Oracle Enterprise Pack for Eclipse (OEPE) to create the below configuration file with the Coherence tooling, which will use these two WKA addresses and auto-increment the port the more servers you start (default). I put one copy in the EAR files APP-INF/classes directory and one in $COHERENCE_HOME. You can replace the hostname of adc2101131.us.oracle.com with appropriate hostnames for your systems. Note that the newer versions of Coherence use XSD instead of DTD.
<?xml version="1.0"?>
<!DOCTYPE coherence SYSTEM "coherence-override.dtd">
<coherence>
<cluster-config>
<unicast-listener>
<well-known-addresses>
<socket-address id="2">
<address>adc2101131.us.oracle.com</address>
<port>8090</port>
</socket-address>
<socket-address id="1">
<address>adc2101131.us.oracle.com</address>
<port>8088</port>
</socket-address>
</well-known-addresses>
<address system-property="tangosol.coherence.localhost">adc2101131.us.oracle.com</address>
<port system-property="tangosol.coherence.localport">8088</port>
<port-auto-adjust system-property="tangosol.coherence.localport.adjust">true</port-auto-adjust>
</unicast-listener>
</cluster-config>
</coherence>
Then before you deploy the application to Weblogic Server, you need to setup and start the cache servers. I simply clone $COHERENCE_HOME/bin/cache-server.sh to $COHERENCE_HOME/bin/sessions-cache-server.sh I'll have bolded and highlighted the edits I made....
#!/bin/sh
# This will start a cache server
# specify the Coherence installation directory
COHERENCE_HOME=.
# specify the JVM heap size
MEMORY=512m
if [ ! -f ${COHERENCE_HOME}/bin/session-cache-server.sh ]; then
echo "session-cache-server.sh: must be run from the Coherence installation directory."
exit
fi
if [ -f $JAVA_HOME/bin/java ]; then
JAVAEXEC=$JAVA_HOME/bin/java
else
JAVAEXEC=java
fi
JAVA_OPTS="-Xms$MEMORY -Xmx$MEMORY -Dtangosol.coherence.session.localstorage=true -Dtangosol.coherence.cacheconfig=WEB-INF/classes/session-cache-config.xml -Dtangosol.coherence.override=/scratch/jbayer/coherence/tangosol-coherence-override.xml"
$JAVAEXEC -server -showversion $JAVA_OPTS -cp "$COHERENCE_HOME/lib/coherence.jar:$COHERENCE_HOME/lib/coherence-web-spi.war" com.tangosol.net.DefaultCacheServer $1
Then start two instances of the session-cache-server.sh from the $COHERENCE_HOME directory like this:
bash-3.2$ bin/session-cache-server.sh
Make sure they join the cluster together and see each other by viewing the member list in the output. Here is the output from the first server. Note the highlighted areas in red which show that:
  • · the override file we intend is being used
  • · the session-cache-config.xml from the coherence-web-spi.war is being used
  • · that the protocol specified in the override is being used, in this case WKA mans unicast instead of multicast
  • · the distributed sessions service is storage enabled
bash-3.2$ bin/session-cache-server.sh
java version "1.6.0_21"
Java(TM) SE Runtime Environment (build 1.6.0_21-b51)
Java HotSpot(TM) Server VM (build 17.0-b17, mixed mode)
2011-02-15 23:26:56.374/0.580 Oracle Coherence 3.6.1.0 <Info> (thread=main, member=n/a): Loaded operational configuration from "jar:file:/scratch/jbayer/coherence/lib/coherence.jar!/tangosol-coherence.xml"
2011-02-15 23:26:56.379/0.585 Oracle Coherence 3.6.1.0 <Info> (thread=main, member=n/a): Loaded operational overrides from "file:/scratch/jbayer/coherence/tangosol-coherence-override.xml"
2011-02-15 23:26:56.386/0.592 Oracle Coherence 3.6.1.0 <D5> (thread=main, member=n/a): Optional configuration override "/custom-mbeans.xml" is not specified
Oracle Coherence Version 3.6.1.0 Build 19636
Grid Edition: Development mode
Copyright (c) 2000, 2010, Oracle and/or its affiliates. All rights reserved.
2011-02-15 23:26:56.760/0.966 Oracle Coherence GE 3.6.1.0 <Info> (thread=main, member=n/a): Loaded cache configuration from "jar:file:/scratch/jbayer/coherence/lib/coherence-web-spi.war!/WEB-INF/classes/session-cache-config.xml"
2011-02-15 23:26:57.257/1.463 Oracle Coherence GE 3.6.1.0 <D4> (thread=main, member=n/a): TCMP bound to /10.229.188.238:8088 using SystemSocketProvider
2011-02-15 23:27:27.613/31.819 Oracle Coherence GE 3.6.1.0 <Info> (thread=Cluster, member=n/a): Created a new cluster with Member(Id=1, Timestamp=2011-02-15 23:26:57.261, Address=10.229.188.238:8088, MachineId=21486, Location=machine:jbayer-vm,process:30802, Role=CoherenceServer, Edition=Grid Edition, Mode=Development, CpuCount=2, SocketCount=2) UID=0x0AE5BCEE0000012E2D5D5EED53EE1F98
2011-02-15 23:27:27.616/31.822 Oracle Coherence GE 3.6.1.0 <Info> (thread=main, member=n/a): Started cluster Name=n/a
WellKnownAddressList(Size=2,
WKA{Address=10.229.188.238, Port=8090}
WKA{Address=10.229.188.238, Port=8088}
)
MasterMemberSet
(
ThisMember=Member(Id=1, Timestamp=2011-02-15 23:26:57.261, Address=10.229.188.238:8088, MachineId=21486, Location=machine:jbayer-vm,process:30802, Role=CoherenceServer)
OldestMember=Member(Id=1, Timestamp=2011-02-15 23:26:57.261, Address=10.229.188.238:8088, MachineId=21486, Location=machine:jbayer-vm,process:30802, Role=CoherenceServer)
ActualMemberSet=MemberSet(Size=1, BitSetCount=2
Member(Id=1, Timestamp=2011-02-15 23:26:57.261, Address=10.229.188.238:8088, MachineId=21486, Location=machine:jbayer-vm,process:30802, Role=CoherenceServer)
)
RecycleMillis=1200000
RecycleSet=MemberSet(Size=0, BitSetCount=0
)
)
TcpRing{Connections=[]}
IpMonitor{AddressListSize=0}
2011-02-15 23:27:27.665/31.871 Oracle Coherence GE 3.6.1.0 <D5> (thread=Invocation:Management, member=1): Service Management joined the cluster with senior service member 1
2011-02-15 23:27:27.828/32.034 Oracle Coherence GE 3.6.1.0 <D5> (thread=ReplicatedCache:ReplicatedSessionsMisc, member=1): Service ReplicatedSessionsMisc joined the cluster with senior service member 1
2011-02-15 23:27:28.074/32.280 Oracle Coherence GE 3.6.1.0 <D5> (thread=DistributedCache:DistributedSessions, member=1): Service DistributedSessions joined the cluster with senior service member 1
2011-02-15 23:27:28.105/32.311 Oracle Coherence GE 3.6.1.0 <Info> (thread=main, member=1):
Services
(
ClusterService{Name=Cluster, State=(SERVICE_STARTED, STATE_JOINED), Id=0, Version=3.6, OldestMemberId=1}
InvocationService{Name=Management, State=(SERVICE_STARTED), Id=1, Version=3.1, OldestMemberId=1}
ReplicatedCache{Name=ReplicatedSessionsMisc, State=(SERVICE_STARTED), Id=2, Version=3.0, OldestMemberId=1}
PartitionedCache{Name=DistributedSessions, State=(SERVICE_STARTED), LocalStorage=enabled, PartitionCount=257, BackupCount=1, AssignedPartitions=257, BackupPartitions=0}
)
Started DefaultCacheServer...
Here is the output from the second cache server. Note that it shows both servers being part of the cluster now. I'll highlight a few things to look for. Notably, the same configuration values are used as the previous cluster and that this time the member id is 2 and at the end this node asks the member 1 for some partitions.
bash-3.2$ bin/session-cache-server.sh
java version "1.6.0_21"
Java(TM) SE Runtime Environment (build 1.6.0_21-b51)
Java HotSpot(TM) Server VM (build 17.0-b17, mixed mode)
2011-02-15 23:34:51.856/0.251 Oracle Coherence 3.6.1.0 <Info> (thread=main, member=n/a): Loaded operational configuration from "jar:file:/scratch/jbayer/coherence/lib/coherence.jar!/tangosol-coherence.xml"
2011-02-15 23:34:51.861/0.256 Oracle Coherence 3.6.1.0 <Info> (thread=main, member=n/a): Loaded operational overrides from "file:/scratch/jbayer/coherence/tangosol-coherence-override.xml"
2011-02-15 23:34:51.866/0.261 Oracle Coherence 3.6.1.0 <D5> (thread=main, member=n/a): Optional configuration override "/custom-mbeans.xml" is not specified
Oracle Coherence Version 3.6.1.0 Build 19636
Grid Edition: Development mode
Copyright (c) 2000, 2010, Oracle and/or its affiliates. All rights reserved.
2011-02-15 23:34:52.147/0.542 Oracle Coherence GE 3.6.1.0 <Info> (thread=main, member=n/a): Loaded cache configuration from "jar:file:/scratch/jbayer/coherence/lib/coherence-web-spi.war!/WEB-INF/classes/session-cache-config.xml"
2011-02-15 23:34:52.500/0.895 Oracle Coherence GE 3.6.1.0 <D4> (thread=main, member=n/a): TCMP bound to /10.229.188.238:8090 using SystemSocketProvider
2011-02-15 23:35:16.251/24.646 Oracle Coherence GE 3.6.1.0 <Info> (thread=Cluster, member=n/a): This Member(Id=2, Timestamp=2011-02-15 23:35:16.04, Address=10.229.188.238:8090, MachineId=21486, Location=machine:jbayer-vm,process:31080, Role=CoherenceServer, Edition=Grid Edition, Mode=Development, CpuCount=2, SocketCount=2) joined cluster with senior Member(Id=1, Timestamp=2011-02-15 23:34:45.487, Address=10.229.188.238:8088, MachineId=21486, Location=machine:jbayer-vm,process:31060, Role=CoherenceServer, Edition=Grid Edition, Mode=Development, CpuCount=2, SocketCount=2)
2011-02-15 23:35:16.270/24.665 Oracle Coherence GE 3.6.1.0 <D5> (thread=Cluster, member=n/a): Member 1 joined Service Cluster with senior member 1
2011-02-15 23:35:16.270/24.665 Oracle Coherence GE 3.6.1.0 <D5> (thread=Cluster, member=n/a): Member 1 joined Service Management with senior member 1
2011-02-15 23:35:16.270/24.665 Oracle Coherence GE 3.6.1.0 <D5> (thread=Cluster, member=n/a): Member 1 joined Service ReplicatedSessionsMisc with senior member 1
2011-02-15 23:35:16.270/24.665 Oracle Coherence GE 3.6.1.0 <D5> (thread=Cluster, member=n/a): Member 1 joined Service DistributedSessions with senior member 1
2011-02-15 23:35:16.273/24.668 Oracle Coherence GE 3.6.1.0 <Info> (thread=main, member=n/a): Started cluster Name=n/a
WellKnownAddressList(Size=2,
WKA{Address=10.229.188.238, Port=8088}
WKA{Address=10.229.188.238, Port=8090}
)
MasterMemberSet
(
ThisMember=Member(Id=2, Timestamp=2011-02-15 23:35:16.04, Address=10.229.188.238:8090, MachineId=21486, Location=machine:jbayer-vm,process:31080, Role=CoherenceServer)
OldestMember=Member(Id=1, Timestamp=2011-02-15 23:34:45.487, Address=10.229.188.238:8088, MachineId=21486, Location=machine:jbayer-vm,process:31060, Role=CoherenceServer)
ActualMemberSet=MemberSet(Size=2, BitSetCount=2
Member(Id=1, Timestamp=2011-02-15 23:34:45.487, Address=10.229.188.238:8088, MachineId=21486, Location=machine:jbayer-vm,process:31060, Role=CoherenceServer)
Member(Id=2, Timestamp=2011-02-15 23:35:16.04, Address=10.229.188.238:8090, MachineId=21486, Location=machine:jbayer-vm,process:31080, Role=CoherenceServer)
)
RecycleMillis=1200000
RecycleSet=MemberSet(Size=0, BitSetCount=0
)
)
TcpRing{Connections=[1]}
IpMonitor{AddressListSize=0}
2011-02-15 23:35:16.321/24.716 Oracle Coherence GE 3.6.1.0 <D5> (thread=Invocation:Management, member=2): Service Management joined the cluster with senior service member 1
2011-02-15 23:35:16.429/24.825 Oracle Coherence GE 3.6.1.0 <D5> (thread=ReplicatedCache:ReplicatedSessionsMisc, member=2): Service ReplicatedSessionsMisc joined the cluster with senior service member 1
2011-02-15 23:35:16.618/25.013 Oracle Coherence GE 3.6.1.0 <D5> (thread=DistributedCache:DistributedSessions, member=2): Service DistributedSessions joined the cluster with senior service member 1
2011-02-15 23:35:16.648/25.043 Oracle Coherence GE 3.6.1.0 <Info> (thread=main, member=2):
Services
(
ClusterService{Name=Cluster, State=(SERVICE_STARTED, STATE_JOINED), Id=0, Version=3.6, OldestMemberId=1}
InvocationService{Name=Management, State=(SERVICE_STARTED), Id=1, Version=3.1, OldestMemberId=1}
ReplicatedCache{Name=ReplicatedSessionsMisc, State=(SERVICE_STARTED), Id=2, Version=3.0, OldestMemberId=1}
PartitionedCache{Name=DistributedSessions, State=(SERVICE_STARTED), LocalStorage=enabled, PartitionCount=257, BackupCount=1, AssignedPartitions=0, BackupPartitions=0}
)
Started DefaultCacheServer...
2011-02-15 23:35:16.657/25.052 Oracle Coherence GE 3.6.1.0 <D4> (thread=DistributedCache:DistributedSessions, member=2): Asking member 1 for 128 primary partitions
If the cluster has formed well, then we are ready to deploy the application to WebLogic Server. Deploy CWebTestApp.ear (assuming you have the correct values for APP-INF/classes/tangosol-coherence-override.xml inside of it and see if it activates and joins the cluster. Here is output from one of the two managed servers in the WLS cluster. Your output may vary to a degree.
<Feb 16, 2011 12:12:02 AM PST> <Notice> <WebLogicServer> <BEA-000360> <Server started in RUNNING mode>
<Feb 16, 2011 12:13:09 AM PST> <Warning> <Munger> <BEA-2156203> <A version attribute was not found in element application in the deployment descriptor in /scratch/jbayer/wls1034/user_projects/domains/base_domain/servers/ManagedServer_2/stage/CWebTestApp/CWebTestApp.ear/META-INF/application.xml. A version attribute is required, but this version of the Weblogic Server will assume that the JEE5 is used. Future versions of the Weblogic Server will reject descriptors that do not specify the JEE version.>
<Feb 16, 2011 12:13:09 AM PST> <Warning> <Munger> <BEA-2156203> <A version attribute was not found in element application in the deployment descriptor in /scratch/jbayer/wls1034/user_projects/domains/base_domain/servers/ManagedServer_2/tmp/_WL_user/CWebTestApp/sapbbg/META-INF/application.xml. A version attribute is required, but this version of the Weblogic Server will assume that the JEE5 is used. Future versions of the Weblogic Server will reject descriptors that do not specify the JEE version.>
2011-02-16 00:13:10.175/115.554 Oracle Coherence 3.6.1.0 <Info> (thread=[STANDBY] ExecuteThread: '5' for queue: 'weblogic.kernel.Default (self-tuning)', member=n/a): Loaded operational configuration from "zip:/scratch/jbayer/wls1034/user_projects/domains/base_domain/servers/ManagedServer_2/stage/coherence/3.6.1.0@3.6.1.0/coherence.jar!/tangosol-coherence.xml"
2011-02-16 00:13:10.190/115.568 Oracle Coherence 3.6.1.0 <Info> (thread=[STANDBY] ExecuteThread: '5' for queue: 'weblogic.kernel.Default (self-tuning)', member=n/a): Loaded operational overrides from "zip:/scratch/jbayer/wls1034/user_projects/domains/base_domain/servers/ManagedServer_2/stage/coherence/3.6.1.0@3.6.1.0/coherence.jar!/tangosol-coherence-override-dev.xml"
2011-02-16 00:13:10.191/115.569 Oracle Coherence 3.6.1.0 <Info> (thread=[STANDBY] ExecuteThread: '5' for queue: 'weblogic.kernel.Default (self-tuning)', member=n/a): Loaded operational overrides from "file:/scratch/jbayer/wls1034/user_projects/domains/base_domain/servers/ManagedServer_2/tmp/_WL_user/CWebTestApp/sapbbg/APP-INF/classes/tangosol-coherence-override.xml"
2011-02-16 00:13:10.197/115.575 Oracle Coherence 3.6.1.0 <D5> (thread=[STANDBY] ExecuteThread: '5' for queue: 'weblogic.kernel.Default (self-tuning)', member=n/a): Optional configuration override "/custom-mbeans.xml" is not specified
Oracle Coherence Version 3.6.1.0 Build 19636
Grid Edition: Development mode
Copyright (c) 2000, 2010, Oracle and/or its affiliates. All rights reserved.
2011-02-16 00:13:10.391/115.769 Oracle Coherence GE 3.6.1.0 <Info> (thread=[STANDBY] ExecuteThread: '5' for queue: 'weblogic.kernel.Default (self-tuning)', member=n/a): Loaded cache configuration from "file:/scratch/jbayer/wls1034/user_projects/domains/base_domain/servers/ManagedServer_2/tmp/_WL_user/coherence-web-spi/4yxuqd/WEB-INF/classes/session-cache-config.xml"
<Feb 16, 2011 12:13:10 AM PST> <Warning> <WorkManager> <BEA-002919> <Unable to find a WorkManager with name wm/CoherenceWorkManager. Dispatch policy wm/CoherenceWorkManager will map to the default WorkManager for the application CWebTestApp>
2011-02-16 00:13:10.681/116.059 Oracle Coherence GE 3.6.1.0 <D5> (thread=[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)', member=n/a):
Clustered Session Cache Name=session-storage
Local Session Cache Name=local-session-storage
Local Session Attribute Cache Name=local-attribute-storage
Death Certificate Cache Name=session-death-certificates
SessionDistributionController Class Name=
AttributeScopeController Class Name=com.tangosol.coherence.servlet.AbstractHttpSessionCollection$ApplicationScopeController
Maximum Session Inactive Seconds=3600
Session ID Character Length=52
Session Locking Enforced=false
Member Session Locking Enforced=false
Application Session Locking Enforced=false
Thread Session Locking Enforced=false
Session Get Lock Timeout=300
Suspect Attribute Detection=false
Strict "Servlet Specification" Exception Handling=true
Sticky Session Ownership=false
Sticky Session Ownership Service Name=SessionOwnership
Assume Session Locality for Reaping=false
Parallel Session Reaping=true
Allow Local Attributes=false
Use Default Session ID Decoding=true
Use Default Session ID Encoding=false
Session Expiry Filter Factory=
2011-02-16 00:13:11.256/116.634 Oracle Coherence GE 3.6.1.0 <Warning> (thread=[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)', member=n/a): PreferredUnicastUdpSocket failed to set receive buffer size to 1428 packets (1.99MB); actual size is 25%, 357 packets (512KB). Consult your OS documentation regarding increasing the maximum socket buffer size. Proceeding with the actual value may cause sub-optimal performance.
2011-02-16 00:13:11.257/116.635 Oracle Coherence GE 3.6.1.0 <D4> (thread=[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)', member=n/a): TCMP bound to /10.229.188.238:8094 using SystemSocketProvider
2011-02-16 00:13:11.595/116.973 Oracle Coherence GE 3.6.1.0 <Info> (thread=Cluster, member=n/a): Failed to satisfy the variance: allowed=16, actual=45
2011-02-16 00:13:11.596/116.974 Oracle Coherence GE 3.6.1.0 <Info> (thread=Cluster, member=n/a): Increasing allowable variance to 19
2011-02-16 00:13:11.944/117.322 Oracle Coherence GE 3.6.1.0 <Info> (thread=Cluster, member=n/a): This Member(Id=10, Timestamp=2011-02-16 00:13:11.72, Address=10.229.188.238:8094, MachineId=21486, Location=machine:jbayer-vm,process:32763, Role=WeblogicServer, Edition=Grid Edition, Mode=Development, CpuCount=2, SocketCount=2) joined cluster with senior Member(Id=1, Timestamp=2011-02-15 23:34:45.487, Address=10.229.188.238:8088, MachineId=21486, Location=machine:jbayer-vm,process:31060, Role=CoherenceServer, Edition=Grid Edition, Mode=Development, CpuCount=2, SocketCount=2)
2011-02-16 00:13:11.997/117.375 Oracle Coherence GE 3.6.1.0 <D5> (thread=Cluster, member=n/a): Member(Id=2, Timestamp=2011-02-15 23:35:16.04, Address=10.229.188.238:8090, MachineId=21486, Location=machine:jbayer-vm,process:31080, Role=CoherenceServer) joined Cluster with senior member 1
2011-02-16 00:13:11.999/117.377 Oracle Coherence GE 3.6.1.0 <D5> (thread=Cluster, member=n/a): Member(Id=9, Timestamp=2011-02-16 00:13:11.687, Address=10.229.188.238:8092, MachineId=21486, Location=machine:jbayer-vm,process:32676, Role=WeblogicServer) joined Cluster with senior member 1
2011-02-16 00:13:12.044/117.422 Oracle Coherence GE 3.6.1.0 <D5> (thread=Cluster, member=n/a): Member 1 joined Service Cluster with senior member 1
2011-02-16 00:13:12.045/117.424 Oracle Coherence GE 3.6.1.0 <D5> (thread=Cluster, member=n/a): Member 1 joined Service Management with senior member 1
2011-02-16 00:13:12.046/117.424 Oracle Coherence GE 3.6.1.0 <D5> (thread=Cluster, member=n/a): Member 1 joined Service ReplicatedSessionsMisc with senior member 1
2011-02-16 00:13:12.046/117.424 Oracle Coherence GE 3.6.1.0 <D5> (thread=Cluster, member=n/a): Member 1 joined Service DistributedSessions with senior member 1
2011-02-16 00:13:12.046/117.424 Oracle Coherence GE 3.6.1.0 <D5> (thread=Cluster, member=n/a): Member 2 joined Service Cluster with senior member 1
2011-02-16 00:13:12.046/117.424 Oracle Coherence GE 3.6.1.0 <D5> (thread=Cluster, member=n/a): Member 2 joined Service Management with senior member 1
2011-02-16 00:13:12.046/117.424 Oracle Coherence GE 3.6.1.0 <D5> (thread=Cluster, member=n/a): Member 2 joined Service ReplicatedSessionsMisc with senior member 1
2011-02-16 00:13:12.046/117.424 Oracle Coherence GE 3.6.1.0 <D5> (thread=Cluster, member=n/a): Member 2 joined Service DistributedSessions with senior member 1
2011-02-16 00:13:12.087/117.465 Oracle Coherence GE 3.6.1.0 <Info> (thread=[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)', member=n/a): Started cluster Name=n/a
WellKnownAddressList(Size=2,
WKA{Address=10.229.188.238, Port=8088}
WKA{Address=10.229.188.238, Port=8090}
)
MasterMemberSet
(
ThisMember=Member(Id=10, Timestamp=2011-02-16 00:13:11.72, Address=10.229.188.238:8094, MachineId=21486, Location=machine:jbayer-vm,process:32763, Role=WeblogicServer)
OldestMember=Member(Id=1, Timestamp=2011-02-15 23:34:45.487, Address=10.229.188.238:8088, MachineId=21486, Location=machine:jbayer-vm,process:31060, Role=CoherenceServer)
ActualMemberSet=MemberSet(Size=4, BitSetCount=2
Member(Id=1, Timestamp=2011-02-15 23:34:45.487, Address=10.229.188.238:8088, MachineId=21486, Location=machine:jbayer-vm,process:31060, Role=CoherenceServer)
Member(Id=2, Timestamp=2011-02-15 23:35:16.04, Address=10.229.188.238:8090, MachineId=21486, Location=machine:jbayer-vm,process:31080, Role=CoherenceServer)
Member(Id=9, Timestamp=2011-02-16 00:13:11.687, Address=10.229.188.238:8092, MachineId=21486, Location=machine:jbayer-vm,process:32676, Role=WeblogicServer)
Member(Id=10, Timestamp=2011-02-16 00:13:11.72, Address=10.229.188.238:8094, MachineId=21486, Location=machine:jbayer-vm,process:32763, Role=WeblogicServer)
)
RecycleMillis=1200000
RecycleSet=MemberSet(Size=0, BitSetCount=0
)
)
TcpRing{Connections=[9]}
IpMonitor{AddressListSize=0}
2011-02-16 00:13:12.199/117.577 Oracle Coherence GE 3.6.1.0 <D5> (thread=Invocation:Management, member=10): Service Management joined the cluster with senior service member 1
2011-02-16 00:13:12.234/117.612 Oracle Coherence GE 3.6.1.0 <D5> (thread=Cluster, member=10): Member 9 joined Service Management with senior member 1
2011-02-16 00:13:12.478/117.856 Oracle Coherence GE 3.6.1.0 <D5> (thread=DistributedCache:DistributedSessions, member=10): Service DistributedSessions joined the cluster with senior service member 1
2011-02-16 00:13:12.560/117.938 Oracle Coherence GE 3.6.1.0 <D5> (thread=Cluster, member=10): Member 9 joined Service DistributedSessions with senior member 1
2011-02-16 00:13:12.618/117.996 Oracle Coherence GE 3.6.1.0 <Info> (thread=[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)', member=10): Configured session model "SplitHttpSessionCollection":
Clustered Session Cache Name=session-storage
Local Session Cache Name=local-session-storage
Local Session Attribute Cache Name=local-attribute-storage
Death Certificate Cache Name=session-death-certificates
SessionDistributionController Class Name=
AttributeScopeController Class Name=com.tangosol.coherence.servlet.AbstractHttpSessionCollection$ApplicationScopeController
Maximum Session Inactive Seconds=3600
Session ID Character Length=52
Session Locking Enforced=false
Member Session Locking Enforced=false
Application Session Locking Enforced=false
Thread Session Locking Enforced=false
Session Get Lock Timeout=300
Suspect Attribute Detection=false
Strict "Servlet Specification" Exception Handling=true
Sticky Session Ownership=false
Sticky Session Ownership Service Name=SessionOwnership
Assume Session Locality for Reaping=false
Parallel Session Reaping=true
Allow Local Attributes=false
Use Default Session ID Decoding=true
Use Default Session ID Encoding=false
Session Expiry Filter Factory=
2011-02-16 00:13:12.723/118.101 Oracle Coherence GE 3.6.1.0 <Info> (thread=[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)', member=10): Registering HttpSessionManager(ServetContext=ServletContextWrapper (2.5)
SessionHelper!=null
MajorVersion=2
MinorVersion=5
Clustered=false
ServletContext (wrapped)=weblogic.servlet.internal.session.CoherenceWebServletContextWrapper@1bc1ecee
AttributeMap=null
Oracle Coherence 3.6.1.0) using object name "type=HttpSessionManager,nodeId=10,appId=CWebTestAppcounter"





If so, now try invoking the index.jsp of the counter context path. If both managed servers on are the same hostname, then the same cookie will be sent by the browser (ports differences are ignored by browsers when sending cookies), and you should see the counter increment by 1 when you have the same session. If your WLS servers happen to have different IP addresses or hostnames, then you will have to use a load balancer. More on that later.

To see if Coherence is really being used for session state, kill all the WLS JVM(s) (not the Coherence session cache servers!). Hit refresh in your browser. Both servers should be down. Keep the browser sessions open.





Now restart the WLS managed servers. Since the browser is open, the session reference is still in your browser cookie. When WLS servers are started again and you can refresh the page, the count should pick up where you left off and not start over.





Now you can try the load balancer such as Apache or the simple one included with coherence. In this case I'm going to have the simple load balancer on port 8080 and refer to the managed servers on ports 8001/8002 respectively.
java -cp coherence.jar com.tangosol.net.LoadBalancer adc2101131:8080 adc2101131:8001 adc2101131:8002
Now I should be able to hit the load balancer directly on port 8080. Using the same browser session, I see that the cookie is still valid.





Plus I see that the output from the simple load balancer shows me going to both servers, therefore we observe that the simple Coherence load balancer is not sticky. Below is an example of that output:
routing to 10.229.188.238:8002
routing to 10.229.188.238:8001
Now you should be able to stop one WLS managed server and refresh and still get a valid result. If you refresh several times the LoadBalancer output should have something like this in the System.Out showing that it failed to connect to the server you stopped.
*** could not connect to 10.229.188.238:8002; java.net.ConnectException: Connection refused
Then stop the other WLS managed server. Refresh using the proxy address and there should be a failure to connect from the browser. Then restart the managed server you stopped first and the session should recover.
This marks the end of the basic Coherence*Web with WebLogic Server example. For more information, I recommend reading the Coherence*Web documentation and turning on the JMX reporting features of Coherence and using a tool like JConsole to explore the MBeans for Coherence*Web.

Monday, September 26, 2011

JRockit Flight Recorder Plugin for WebLogic Tab Pack behind a Proxy

This blog was originally posted on http://blogs.oracle.com/jamesbayer when I worked for Oracle.

One of my favorite demos to show is JRockit Flight Recorder integration with WebLogic Server and the WebLogic Server Diagnostic Framework.  It's a great way to see what's going on inside applications and the JVM without agents using out-of-the-box defaults with WebLogic Server and JRockit.  I've made video's and blog posts about it previously and we'll be happy to give you a demo of it at Oracle Open World at the JavaOne booth or the WebLogic Server booth at the Demogrounds.  JRockit Mission Control, which is used to view the JVM recordings called JFRs for JRockit Flight Recording, has some plugins that it downloads from an Eclipse Update site to get the WebLogic Server specific view.  If you're behind a firewall and need to use a proxy, there is a simple way to launch JRMC from a java command line that let's you specify the proxy host, port and other settings you might use like authentication.  So from the JRockit bin directory, this is the command that you can use to launch Mission Control and make sure you use a proxy for network connections so you can use the Update Site with the proxy you specify.  You can substitute your JROCKIT_HOME for your actual path and your actual proxy settings instead of the ones we use at Oracle.
JROCKIT_HOME/bin/java -Dhttp.proxyPort=80 -Dhttp.proxyHost=www-proxy.us.oracle.com -jar JROCKIT_HOME/missioncontrol/mc.jar
 Once you launch JRMC with the proxy enabled (or without the proxy settings if you do not need a proxy to access the Oracle Eclipse Update Site), you can then use the link to install the experimental plugin from the Help menu as shown below.  The one you want for sure is the WebLogic Tab Pack.