OverviewClustering is essential if your CAS instance is to be "highly available," or HA in manager-speak. Since CAS is a stateful application, there must be a way for each CAS instance to know about what the other CAS instance has done. It would be nice to just use one CAS instance (and one instance on the appropriate hardware can probably easily handle your login needs), but if that instance fails, you do not want all of your users to have to log in again. As mentioned above, CAS is a stateful application, and stateful in more than one way. CAS keeps track of users in the application's session, and it keeps track of the services the user visits and the tickets used to visit those services. Although the service and proxy tickets are only stored in memory for a brief amount of time, if you are load balancing and clustering CAS, each instance of CAS must immediately know about those tickets. If they do not, CAS simply will not work (most of the time). You may think that LB sticky sessions will save you, but they won't! Sticky sessions are good for sending the user (via a web browser) back to the same CAS instance, but it does not solve the problem that applications also use CAS, and the LB may have already determined that a particular application should be using another CAS instance (via sticky sessions)! So, there are several things that need to be be done for clustering to work:
Since CAS is a Java application (and based on Spring at that), there are many ways to do clustering. Furthermore, there is no easy "on/off" switch for clustering, hence this document. The CAS clustering described here takes advantage of the Spring aspects of CAS, and implements the clustering purely via XML configuration! (Of course, we do use Java classes that have already been written by the CAS team.) AssumptionsThis HOW TO makes the following assumptions:
ClusteringGuaranteeing Ticket Uniqueness*If you are using CAS 3.2.x, feel free to skip this step. It is already part of your implementation. Since all the tickets need to be unique across JVMs, we will configure this part first, and it is the easiest part to do, too. The first problem you need to solve is what unique identifier to use. I choose the hostname of the server from which CAS is being served. Because this is Java and we do everything via XML configuration and not Java code, we will solve this problem using the applicationContext.xml file and one other file external to CAS. Let's address the one other file external to CAS first. You need to create a .properties file - maybe something like host.properties - and put it on the filesystem where the user who is running Tomcat can access it. I have mine here: /apps/local/share/etc/host.properties In this file, put one property: host.name: <hostname of your server> Replace <hostname of your server> with the hostname of your server. Now that you have this, you need to edit applicationContext.xml. This file is located in the following location in the CAS distribution: CAS 3.0.x cas-distribution/webapp/WEB-INF/applicationContext.xml CAS 3.1.x cas-distribution/cas-server-webapp/src/main/webapp/WEB-INF/applicationContext.xml Now you need to reference the host.properties file in the applicationContext.xml file. Add the following to the top of the file (after the <beans> tag): <bean id="placeholderConfig" class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer"> <property name="locations"> <list> <value>file:/apps/local/share/etc/host.properties</value> </list> </property> </bean> This directive tells CAS to load the host.properties file into memory. And once it has done this, you can access the values of the properties inside using a $. So, we want to use the value of host.name in the ticketGrantingTicketUniqueIdGenerator bean: <bean
id="ticketGrantingTicketUniqueIdGenerator"
class="org.jasig.cas.util.DefaultUniqueTicketIdGenerator">
<constructor-arg
index="0"
value="${host.name}" />
</bean>
Note the use of host.name. This comes from our host.properties file. This will create a CAS TGT that looks something like this: TGT-2-Lj1aIVkEqGDCSLaXwXVQlIcYQcyyqcI0tuR-<hostname of your server>
We are technically done for the user-level (TGT) ticket uniqueness, but since we will be clustering the service ticket registry, let's do that part while we have this file open. Also, let's set up a unique TicketIdGenerator for the Cas20ProxyHandler as well. Edit the serviceTicketUniqueIdGenerator bean to look like this: <bean
id="serviceTicketUniqueIdGenerator"
class="org.jasig.cas.util.DefaultUniqueTicketIdGenerator">
<constructor-arg
index="0"
value="${host.name}" />
</bean>
Now the TicketIdGenerator for the Cas20ProxyHandler is not so easy as we need to actually define a bean for it (we could cheat and use an existing bean, but let's make a new one in case we want to change it independent of the other beans (after-all, that is the Spring way!)), and then reference it in the proxy20Handler bean section. So, under the serviceTicketUniqueIdGenerator bean that you just edited, add this new bean for the proxy tickets: <bean
id="proxy20TicketUniqueIdGenerator"
class="org.jasig.cas.util.DefaultUniqueTicketIdGenerator">
<constructor-arg
index="0"
value="${host.name}" />
</bean>
Now, add a reference to the above as an additional propert called uniqueTicketIdGenerator to the proxy20Handler bean. So, the whole bean should look like this: <bean
id="proxy20Handler"
class="org.jasig.cas.ticket.proxy.support.Cas20ProxyHandler">
<property
name="httpClient"
ref="httpClient" />
<property
name="uniqueTicketIdGenerator"
ref="proxy20TicketUniqueIdGenerator" />
</bean>
Ok, we are now done with guaranteeing ticket uniqueness. Now onto session replication! Tomcat Session ReplicationSince CAS stores the login information in the application session 2 we need to setup session replication between our Tomcat instances. The first thing you need to do is tell CAS (the application) that it is distributable 3. So, in the CAS web.xml file you need to add the <distributable/> tag. The web.xml file is located here: CAS 3.0.x cas-distribution/webapp/WEB-INF/web.xml CAS 3.1.x & CAS 3.2.x cas-distribution/cas-server-webapp/src/main/webapp/WEB-INF/web.xml In this file, I put the distributable tag right below the context-param section: ... <context-param> <param-name>contextConfigLocation</param-name> <param-value> /WEB-INF/applicationContext.xml, /WEB-INF/deployerConfigContext.xml </param-value> </context-param> <!-- Set the application as distributable: http://tomcat.apache.org/tomcat-5.0-doc/cluster-howto.html --> <distributable /> ... Now you need to tell Tomcat to replicate the session information using muticast. This is done in the Tomcat server.xml file. You need to add the cluster information inside of the Engine tag (Skip this if you are using JBoss): <Engine name="Standalone" defaultHost="localhost"> <Host name="localhost" deployOnStartup="true" autoDeploy="false" appBase="webapps"> <!-- When configuring for clustering, you also add in a valve to catch all the requests coming in, at the end of the request, the session may or may not be replicated. A session is replicated if and only if all the conditions are met: 1. useDirtyFlag is true or setAttribute or removeAttribute has been called AND 2. a session exists (has been created) 3. the request is not trapped by the "filter" attribute The filter attribute is to filter out requests that could not modify the session, hence we don't replicate the session after the end of this request. The filter is negative, ie, anything you put in the filter, you mean to filter out, ie, no replication will be done on requests that match one of the filters. The filter attribute is delimited by ;, so you can't escape out ; even if you wanted to. filter=".*\.gif;.*\.js;" means that we will not replicate the session after requests with the URI ending with .gif and .js are intercepted. --> <Cluster className="org.apache.catalina.cluster.tcp.SimpleTcpCluster" managerClassName="org.apache.catalina.cluster.session.DeltaManager" expireSessionsOnShutdown="false" useDirtyFlag="true"> <Membership className="org.apache.catalina.cluster.mcast.McastService" mcastAddr="239.255.0.1" mcastPort="45564" mcastFrequency="500" mcastDropTime="3000" mcastTTL="1"/> <Receiver className="org.apache.catalina.cluster.tcp.ReplicationListener" tcpListenAddress="auto" tcpListenPort="4001" tcpSelectorTimeout="100" tcpThreadCount="6"/> <Sender className="org.apache.catalina.cluster.tcp.ReplicationTransmitter" replicationMode="synchronous"/> <Valve className="org.apache.catalina.cluster.tcp.ReplicationValve" filter=".*\.gif;.*\.js;.*\.jpg;.*\.htm;.*\.html;.*\.txt;"/> <ClusterListener className="org.apache.catalina.cluster.session.ClusterSessionListener"/> </Cluster> </Host> </Engine> This is the config for Tomcat 6.0 <!--
When configuring for clustering, you also add in a valve to catch all the requests
coming in, at the end of the request, the session may or may not be replicated.
A session is replicated if and only if all the conditions are met:
1. useDirtyFlag is true or setAttribute or removeAttribute has been called AND
2. a session exists (has been created)
3. the request is not trapped by the "filter" attribute
The filter attribute is to filter out requests that could not modify the session,
hence we don't replicate the session after the end of this request.
The filter is negative, ie, anything you put in the filter, you mean to filter out,
ie, no replication will be done on requests that match one of the filters.
The filter attribute is delimited by ;, so you can't escape out ; even if you wanted to.
filter=".*\.gif;.*\.js;" means that we will not replicate the session after requests with the URI
ending with .gif and .js are intercepted.
-->
<Cluster className="org.apache.catalina.ha.tcp.SimpleTcpCluster">
<Manager className="org.apache.catalina.ha.session.DeltaManager"
expireSessionsOnShutdown="false"
notifyListenersOnReplication="true"/>
<Channel className="org.apache.catalina.tribes.group.GroupChannel">
<Membership
className="org.apache.catalina.tribes.membership.McastService"
address="239.255.0.1"
port="45564"
frequency="500"
dropTime="3000"
mcastTTL="1"/>
<Receiver
className="org.apache.catalina.tribes.transport.nio.NioReceiver"
address="auto"
port="4000"
autoBind="0"
selectorTimeout="100"
maxThreads="6"/>
<Sender className="org.apache.catalina.tribes.transport.ReplicationTransmitter">
<Transport className="org.apache.catalina.tribes.transport.nio.PooledParallelSender"/>
</Sender>
<Interceptor className="org.apache.catalina.tribes.group.interceptors.TcpFailureDetector"/>
<Interceptor className="org.apache.catalina.tribes.group.interceptors.MessageDispatch15Interceptor"/>
</Channel>
<Valve className="org.apache.catalina.ha.tcp.ReplicationValve"
filter=".*\.gif;.*\.js;.*\.jpg;.*\.htm;.*\.html;.*\.txt;"/>
<ClusterListener className="org.apache.catalina.ha.session.ClusterSessionListener"/>
</Cluster>
Note 1: Again, please check with your network administrator before turning this on. I have set mcastTTL to 1 because my network admin told me " If you want to force it to stay within your subnet, my understanding is that you can do so by using a TTL of 1." If you want to do clustering outside of a single subnet, you will probably have to change this value, or remove the mcastTTL attribute and value altogether. Note 2: You will see a lot of references to the jvmRoute attribute of the Engine tag, but you only need to specify that if you are clustering more than one Tomcat on one host. In that case, you will have to specify the jvmRoute that corresponds to the Apache worker you have specified for that Tomcat instance. 4 Note 3: If your Tomcat cluster doesn't work (Tomcat instance not seeing other member), perhaps you must change auto in tcpListenAddress="auto" by IP address of server. Note 4: If your Tomcat cluster still doesn't work ensure that the TCP and UDP ports on the servers are not being blocked by a host-based firewall, that your network interface has multicast enabled, and that it has the appropriate routes for multicast. Note 5: If you see a large stacktrace in the cas.log file that ends with a root cause of: "java.net.BindException: Cannot assign requested address", it's likely due to the JVM trying to use IPv6 sockets while your system is using IPv4. Set the JVM to prefer IPv4 by setting the Java system property -Djava.net.preferIPv4Stack=true. You can set the CATALINA_OPTS environment variable so Tomcat will pick it up automatically with: export CATALINA_OPTS=-Djava.net.preferIPv4Stack=true Now start up your two (or more) Tomcat instances (on separate hosts!) and you should see something like the following in the catalina.out log: May 22, 2007 4:25:54 PM org.apache.catalina.cluster.tcp.SimpleTcpCluster memberAdded
INFO: Replication member added:org.apache.catalina.cluster.mcast.McastMember
[tcp://128.32.143.78:4001,catalina,128.32.143.78,4001, alive=5]
Conversly, in the catalina.out log on my other server, I see: May 22, 2007 4:27:13 PM org.apache.catalina.cluster.tcp.SimpleTcpCluster memberAdded
INFO: Replication member added:org.apache.catalina.cluster.mcast.McastMember
[tcp://128.32.143.79:4001,catalina,128.32.143.79,4001, alive=5]
Excellent, you now have clustering of the user's login information for CAS. Test it out by logging into CAS, then stopping Tomcat on the server you logged in at, and then hit the login page again, and CAS should show you the "you are already logged in page." Ticket Cache ReplicationNow you we need to setup the ticket cache replication using the org.jasig.cas.ticket.registry.JBossCacheTicketRegistry class. We implement this by editing the applicationContext.xml config file again. Open up applicationContext.xml and change the "Ticket Registry" section from this: <bean
id="ticketRegistry"
class="org.jasig.cas.ticket.registry.DefaultTicketRegistry" />
CAS 3.2.x: You need to edit the file WEB-INF/spring-configuration/ticketRegistry.xml: <bean
id="ticketRegistry"
class="org.jasig.cas.ticket.registry.DefaultTicketRegistry" />
to this: <bean id="ticketRegistry" class="org.jasig.cas.ticket.registry.JBossCacheTicketRegistry"> <property name="cache" ref="cache" /> </bean> <bean id="cache" class="org.jasig.cas.util.JBossCacheFactoryBean"> <property name="configLocation" value="classpath:jbossTicketCacheReplicationConfig.xml" /> </bean> Note 1: No space between classpath: and jbossTicketCacheReplicationConfig.xml, otherwise you have a not found exception. In the cache bean above, there is a property with a value of classpath:jbossTicketCacheReplicationConfig.xml so now we have to find and do something with this file. jbossCache.xml started out life as jbossTestCache.xml. Since I do not like to put things into production with the word "test" in them, I changed the name (and a few things inside the file). This file is located at: CAS 3.0.x cas-distribution/core/src/test/resources/jbossTestCache.xml CAS 3.1.x & CAS 3.2.x cas-distribution/cas-server-integration-jboss/src/test/resources/jbossTestCache.xml Open this file up and get ready for some editing. I discovered that the default file did not work in my installation, as was noted by some others on the CAS mailing list. Scott Battaglia sent an edited version to the list. 5 You have to comment-out the following lines: <!-- <depends>jboss:service=TransactionManager</depends> -->
and: <!-- <attribute name="TransactionManagerLookupClass"> org.jboss.cache.DummyTransactionManagerLookup</attribute> --> Next, you have to edit the mcast_addr In the ClusterConfig section, set the mcast_addr to the value appropriate for your network, and if your hosts are on the same subnet, set ip_ttl to 1. You may also need to set the bind_addr property to the IP address you want this host to listen for TreeCache updates. This is especially true if you are using bonding and/or IPV6 on your system : <UDP mcast_addr="239.255.0.2" mcast_port="48866" ip_ttl="1" ip_mcast="true" bind_addr="192.168.10.10" mcast_send_buf_size="150000" mcast_recv_buf_size="80000" ucast_send_buf_size="150000" ucast_recv_buf_size="80000" loopback="false"/> Now that you have edited this file, you have to get it onto your CLASSPATH 6 I have decided to put it directly into my Tomcat directory: $CATALINA_BASE/share/classes/jbossTicketCacheReplicationConfig.xml For JBOSS, this is a good location: $JBOSS_HOME/server/all/conf/jbossTicketCacheReplicationConfig.xml If you know of a better way to get it on your CLASSPATH by putting it somewhere in the localPlugins directory, please let me know. Now, the hard part: Rounding up the 10 jars needed to make JBossCache work! JBossCache for CAS requires the following jars concurrent.jar jboss-cache-jdk50.jar jboss-common.jar jboss-j2ee.jar jboss-jmx.jar jboss-minimal.jar jboss-serialization.jar jboss-system.jar jgroups.jar trove.jar CAS 3.0.x You can get all of these jar files in the JBossCache distribution. 7 Once you have these jars, put them in your localPlugins/lib directory: cas-distribution/localPlugins/lib CAS 3.1.x Using Maven 2, it is not as hard as CAS 3.0.x branch. Add the following dependency to the pom.xml file located at the folder cas-server-webapp and it will include the JBoss cache stuff in cas.war Remarks: The dependency is needed if you are NOT using JBoss Application Server. ... <dependency> <groupId>org.jasig.cas</groupId> <artifactId>cas-server-integration-jboss</artifactId> <version>3.1</version> <scope>runtime</scope> </dependency> ... CAS 3.2.x on JBOSS (or probably any CAS implementation on JBoss) You need to exclude some jars from the deployment otherwise they will conflict with JBOSS. <dependency>
<groupId>org.jasig.cas</groupId>
<artifactId>cas-server-integration-jboss</artifactId>
<version>3.2.1-RC1</version>
<scope>runtime</scope>
<exclusions>
<exclusion>
<groupId>concurrent</groupId>
<artifactId>concurrent</artifactId>
</exclusion>
<exclusion>
<groupId>jboss</groupId>
<artifactId>jboss-serialization</artifactId>
</exclusion>
<exclusion>
<groupId>jboss</groupId>
<artifactId>jboss-jmx</artifactId>
</exclusion>
<exclusion>
<groupId>jboss</groupId>
<artifactId>jboss-common</artifactId>
</exclusion>
<exclusion>
<groupId>jboss</groupId>
<artifactId>jboss-j2ee</artifactId>
</exclusion>
<exclusion>
<groupId>jboss</groupId>
<artifactId>jboss-minimal</artifactId>
</exclusion>
<exclusion>
<groupId>jboss</groupId>
<artifactId>jboss-system</artifactId>
</exclusion>
</exclusions>
</dependency>
Ok, now let's test this thing! Build cas.war and redeploy to your two (or more) Tomcat instances and you should see the JBossCache info in the catalina.out log: 2007-05-23 16:59:34,486 INFO [org.jasig.cas.util.JBossCacheFactoryBean] - <Starting TreeCache service.> ------------------------------------------------------- GMS: address is 128.32.143.78:51052 ------------------------------------------------------- In the catalina.out log on my other server, I see: 2007-05-23 17:01:22,113 INFO [org.jasig.cas.util.JBossCacheFactoryBean] - <Starting TreeCache service.> ------------------------------------------------------- GMS: address is 128.32.143.79:56023 ------------------------------------------------------- If you see this, and no Java exceptions, you are doing well! If you see Java exceptions, they are probably related to Tomcat not being able to find the jbossTicketCacheReplicationConfig.xml file in its CLASSPATH or it can't fine some class related to the JBossCache, i.e., one of the jars is missing. Ensuring Ticket Granting Ticket Cookie VisibilityThe last step before you can test out whether CAS is set up to be clustered correctly is to ensure that the ticket granting ticket (TGT) cookie set in the users' browsers is visible by all of the nodes in the CAS cluster. Using your favorite text editor (shameless plug for vim), open the cas-servlet.xml file and look for the warnCookieGenerator and ticketGrantingTicketCookieGenerator beans. Both of these beans need to have the cookieDomain property set to the domain where the TGT cookie should be visible to. Edit the bean declarations based on the following example (substitute your domain as necessary):
<bean id="warnCookieGenerator" class="org.springframework.web.util.CookieGenerator"> <property name="cookieDomain" value="example.com" /> <property name="cookieSecure" value="true" /> <property name="cookieMaxAge" value="-1" /> <property name="cookieName" value="CASPRIVACY" /> <property name="cookiePath" value="/cas" /> </bean> <bean id="ticketGrantingTicketCookieGenerator" class="org.springframework.web.util.CookieGenerator"> <property name="cookieDomain" value="example.com" /> <property name="cookieSecure" value="true" /> <property name="cookieMaxAge" value="-1" /> <property name="cookieName" value="CASTGC" /> <property name="cookiePath" value="/cas" /> </bean> CAS 3.2.x WEB-INF/spring-configuration/warnCookieGenerator.xml <bean id="warnCookieGenerator" class="org.jasig.cas.web.support.CookieRetrievingCookieGenerator" p:cookieSecure="true" p:cookieMaxAge="-1" p:cookieName="CASPRIVACY" p:cookiePath="/cas" p:cookieDomain="example.com"/> WEB-INF/spring-configuration/ticketGrantingCookieGenerator.xml <bean id="ticketGrantingTicketCookieGenerator" class="org.jasig.cas.web.support.CookieRetrievingCookieGenerator" p:cookieSecure="true" p:cookieMaxAge="-1" p:cookieName="CASTGC" p:cookiePath="/cas" p:cookieDomain="example.com"/> Ok, now test it out (hopefully your load balancers will cooperate). JBOSS You will need to deploy as a .war file into JBoss's farm at: ${JBOSS_HOME}/server/all
After you have started your cluster servers, insure you have a cluster by checking the JBoss DefaultPartition. The CurrentView should show all the ip's of your cluster. If not, you will need to research why your cluster is not finding the other nodes.
ReferencesThe following references are used in this document:
|
