|
Sort Order:
After further research I have concluded that the randomness can be explained by the way in which we restart our Weblogic instance. The hack works perfectly acceptable 100% of the time for a start restart of Weblogic. Therefore, this needs including in the next release of Spring (but perhaps in a cleaner form!). The randomness seems to be something to do with a restart of WL that involves a full rebuild of the configuration between the stop and start. While this is only a theory, the JMS client drivers may be caching something from the configuration (such as a salt) and using this cached value to encrypt the user credentials. Hence, after the configuration is rebuilt, the client drivers can not connect, because Weblogic server can not correctly decrypt the credentials. But we can live with this as it's clearly a fault in the JMS drivers. We actually had the underlying issue reported before (see We may provide such a WebLogic-specific ConnectionFactory decorator ourselves at some point. I'll reconsider this for a later Spring 2.5.x release. Juergen Actually, given how this WebLogic JNDI arrangement works, you could also try defining your ConnectionFactory lookup as JndiObjectFactoryBean with proxyInterface="javax.jms.ConnectionFactory" and cache="false"... (or the equivalent <jee:jndi-lookup jndi-name="..." proxy-interface="javax.jms.ConnectionFactory" cache="false"/>). This would perform a fresh JNDI lookup for every createConnection call, which shouldn't be too bad in terms of overhead - and naturally reinitializes the JNDI InitialContext for every such call... Juergen Hello, I've removed my hack and defined the JndiObjectFactoryBean as discussed above and the listener does connect correctly with user credentials. But interestingly, this appears in the logs: 12-05 04:13:47,434 jms.messageListenerContainer.events.ms1-1779 INFO DefaultMessageListenerContainer.handleListenerSetupFailure:748 - Setup of JMS message listener invoker failed - trying to recover: weblogic.jms.common.JMSSecurityException: Access denied to resource: type=<jms>, application=Reporting, destinationType=queue, resource=EventQueue, action=receive This happens on every connection attempt, but it shouldn't be failing then working. I've read the linked bug and I do understand WL's JMS implementation. While I think it's really quite poor, the username & password in the initial context is bound to the thread, so the only safe way to connect is to create the initial context before each connection attempt. This is why a cached copy, accessed by different threads, fails to work. I am still chasing another problem and while I'm 99% convinced it's not Spring (but WL's JMS), if I can see a way to improve Spring then I'll update this ticket. In the meantime, what is the official Spring recommended route to fixing this problem? I think a non-cached JndiObjectFactoryBean is the safest, although the above debug has to be addressed. This should be documented somewhere in Spring given WL JMS is common and others will come across this problem. John This looks like you cannot use a shared JMS Connection at all in such a scenario then... I assume you're setting up your DefaultMessageListenerContainer without JTA transactions (i.e. without "transactionManager" property), in which case it will be caching a shared Connection by default... Try setting DMLC's "cacheLevelName" property to "CACHE_NONE"; this might make that setup failure log disappear. Juergen Hiya, I still see this: 15-05 10:50:29,656 jms.messageListenerContainer.events.ms2-271 INFO DefaultMessageListenerContainer.handleListenerSetupFailure:748 - Setup of JMS message listener invoker failed - trying to recover: weblogic.jms.common.JMSSecurityException: Access denied to resource: type=<jms>, application=Reporting, destinationType=queue, resource=EventQueue, action=receive When the DFML is setup as follows: <bean id="jms.messageListenerContainer.events.ms2" John In fact, it's quite a serious problem because the listener never seems to pick up messages again - it simply prints out that debug, announces that it's recovered the connection and does no more. So setting CACHE_NONE makes things somewhat worse! I'm concerned that this solution isn't particularly efficient. We don't want to be opening connections to WLS every time the listener refreshes - we simply want a new Initial context created every time a new connection is created. Therefore, despite CACHE_NONE not working as described above, I think caching of the connection/session does have to happen inside the DFML. So if I forget about CACHE_NONE and assume DFML should be caching the connection/session, how do I get rid of the "invoker failed" error? I' m currently debugging the source but it's fairly complicated... FWIW, I would prefer caching to remain active there as well, in particular for non-XA scenarios. CACHE_NONE is usually used in conjunction with connection pools. Reopening connections all the time is certainly not desirable for the general case. However, reobtaining connections for each receive attempt is necessary in case of XA-aware pools in order to get proper XA enlistment. So where do those invoker failures come from? It seems that for some reason the invoker threads fail when using the shared Connection... That should have affected your original workaround as well, since you were just setting the InitialContext for the thread that recovered the shared Connection. Did you see any such invoker failure logging at that point? We need to track down when exactly the security credentials are accepted and when not (leading to JMSSecurityExceptions)... Juergen I didn't see the failure in my original workaround: public class WeblogicMessageListenerContainer extends DefaultMessageListenerContainer public void setJndiTemplate(JndiTemplate jndiTemplate) { this.jndiTemplate = jndiTemplate; } protected void wlFix() catch (NamingException e) { logger.warn(e.getMessage()); }} protected Connection createSharedConnection() } I simply recreated the InitialContext (in the current thread) and let Spring use the InitialContext from the JndiObjectFactoryBean - the important point is that my hack associated the credentials with the current thread. Ironically, I think my workaround actually worked, but we have been trying to diagnose other WLS errors (which I believe are unrelated to Spring) and I wanted to remove it. So with the cache mode set to CONSUMER (i.e. I haven't set it), the error is thrown when this is called: return session.createConsumer(destination, getMessageSelector()); in AbstractPollingMessageListenerContainer.createConsumer. The exception suggests a different thread created the connection, but I've only got one thread running. I've written a test program to connect, create a session, create a message consumer, close it and create another, and that works correctly. So I don't think the exception is being thrown because the consumer is being creaetd twice from the same session (I put nothing past WLS!). I shall carry on debugging... Are there multiple threads sharing the same connection? You need to tell me more about how this works. All I currently have is a set of evidence to suggest that multiple threads are sharing the same connection. I've followed this issue for few days as I face the similar problem. I came to the same workaround but it didn't work for me as I use cluster address to my weblogic cluster like t3://192.168.42.58:8001,192.168.42.58:9001 If I use single node weblogic instance, it works just fine. When I use two nodes failover still works but only when I shutdown first node and then start it again. However when I try to shutdown the first node and migrate JMS server to the second node my Spring client doesn't failover. I get this exception: May 18, 2008 9:14:37 PM org.springframework.jms.listener.DefaultMessageListenerContainer handleListenerSetupFailure My simple (no Spring) single-thread client works fine and fails over as I expect. Here comes my configuration: <bean id="queueMDP" class="spring.QueueMDP" /> <bean id="jndiTemplate" class="org.springframework.jndi.JndiTemplate"> <bean id="queueConnectionFactory" class="org.springframework.jndi.JndiObjectFactoryBean"> <bean id="queue1" class="org.springframework.jndi.JndiObjectFactoryBean"> <bean id="queue1Listener" class="org.springframework.jms.listener.DefaultMessageListenerContainer"> I found out that my Spring-based client connects to only one node and I can't set it to use both nodes as defined in cluster address. Maybe there's something cached in queue1 (I think so because of "Destination not found" exception) bean but I don't know how to get over it. The destination failover problem is a common issue on WebLogic. Our recommendation is to not use JndiObjectFactoryBean for the lookup of Destination objects there but rather to specify the "destinationName" property as value "jms.queue1" (in your case) on DefaultMessageListenerContainer itself. For the latter to resolve destinations against JNDI, configuring a JndiDestinationResolver and pass it into DMLC's "destinationResolver" property as a bean reference: <bean id="destinationResolver" class="org.springframework.jms.support.destination.JndiDestinationResolver"> <bean id="queue1Listener" class="org.springframework.jms.listener.DefaultMessageListenerContainer"> This should work fine with the default cache level as well, actually, so I'm not sure CACHE_NONE is really needed in your scenario. Juergen Thanks, Juergen. It works great now even with cached consumer. However I still have to use the same initial context hack as mentioned by John Baker earlier in this issue to get over JMSSecurityException: Access denied. So far I am quite happy with the solution because it fails over correctly and initial context is retrieved only when creating connection. Nonetheless I would appreciate to use clean pure Spring no-hack-needed design. DMLC's shared Connection is indeed used by multiple threads - it has to, since the very point of message listener containers is to manage asynchronous consumers. However, it's unclear to me what exactly works and what doesn't now: Does the "createSharedConnection" overriding hack solve the user credentials problem completely, or is there some reproducible issue remaining in that case? Does it also work if you factor out that "hack" code into a ConnectionFactory decorator - i.e. leave DMLC as-is but link it to a custom ConnectonFactory decorator bean that has the JNDI initialization code in its "createConnection" implementation? public class JndiInitializingConnectionFactory extends DelegatingConnectionFactory { public Connection createConnection()) { wlFix(); return super.createConnection(); } protected void wlFix() { <bean id="wlfixConnectionFactory" class="yourpackage.JndiInitializingConnectionFactory"> And your DefaultMessageListenerContainer bean definition's "connectionFactory" property in turn linking to that "wlfixConnectionFactory" bean. Finally, does it also work if you pass in the same credentials not into a JNDI InitialContext but rather into "ConnectionFactory.createConnection(user, pw)"? That could for example be done in such a custom ConnectionFactory decorator as well, adapting a plain "createConnection()" call onto a "createConnection(user, pw)" call on the target ConnectionFactory. Juergen > Does the "createSharedConnection" overriding hack solve the user credentials problem completely, or is there some This works, but I'd prefer not to extend DMLC. I think we should be able to address this within Spring. There should be a documented approach for using DMLC with Weblogic (given the 'associate user credentials with current thread' bug - I call it a bug as it's not worthy of being described as a feature!). I'll get back to you on the other suggestions. Actually, it's going to work, isn't it? I think you now understand the problem, but the question is how best do we solve it throuh spring without a nasty hack! I guess the factored-out ConnectionFactory decorator should work fine then, with no need to extend DMLC. I'm afraid there is no solution with less configuration effort, since DMLC itself doesn't deal with JNDI at all... Juergen So we need to incorporate the factored out connection factory into Spring and document it properly. Weblogic is a popular JMS platform and while BEA are entirely at fault for this bizarre design, some documentation within Spring would be most helpful. Yes... FYI, we have a related problem in Juergen Can I throw something else into the pot? Check out this exception coming from WLS: ay 20, 2008 5:39:25 AM EDT> <Warning> <RMI> <BEA-080003> <RuntimeException thrown by rmi server: weblogic.jndi.internal.AdminRoleBasedDispatchServerRef@9, implementation: 'weblogic.jndi.internal.RootNamingNode@1f59bbd', oid: '9', implementationClassName: 'weblogic.jndi.internal.RootNamingNode' We are seeing this over and over again and our client is the one we're discussing above. Do you have any ideas why WLS is doing this? We're seeing no obvious failure on the client side... Hmm... that could be a related issue when doing RMI authorization. This may happen anywhere when operating on JNDI-obtained WebLogic handles... All a consequence of that odd design choice in WLS, I guess. Juergen I'm having a read of this: http://edocs.beasys.com/wls/docs81/jndi/jndi.html It may prove useful. As a matter of interest, is Spring calling jndiContext.close(); when a connection is closed? The docs say: "You can use the same properties on either a client or a server. If you define the properties on a server-side object, a local Context is used. If you define the properties on a client or another WebLogic Server, the Context delegates to a remote Context running on the WebLogic Server specified by the Context.PROVIDER_URL property. A remote object bound to the server will not be serviced by peerGone, and will not be reachable if the client should fail." Can you tell me more about the RMI problem? I've got some sample code running without Spring that I'm trying to use to nail this problem. Juergen, On the related bug you've written "As for the JndiCallback solution: It should actually be sufficient to override "doInvoke" only there, since that encompasses "create" and "remove" already (at least in Spring 2.x). So executing "doInvoke" within one all-encompassing JndiCallback should result in equivalent behavior. Is this assumption correct? " Could you write a couple lines of sample code and I'll integrate/play with it to see if it may help solve my problem? J The idea is essentially simply to do a "Context ctx = new InitialContext(...)" first, then perform your invocations on the target object, then close the Context object afterwards. WebLogic apparently expects all access to obtained objects to happen within an open InitialContext, at least for access that requires authorization... Juergen That makes sense and ties in with what we suspect is "dangling proxies" (or whatever they'd be called) on the server. Where would you fit this in though? And would it involve this: Listing 2-3 JNDI Context and Threads Wrapper Code import java.security.PrivilegedAction; public class client I.e. do we need: Security.runAs(new Subject(), Which of course doesn't sound very efficient! I've tried extending the JndiTemplate and it did not solve the problem: public class JndiTemplate extends org.springframework.jndi.JndiTemplate @Override catch (NamingException e) { logger.warn(e); } finally catch (NamingException ex) { logger.debug("Could not close JNDI InitialContext", ex); } } However I've discovered something a little odd. The exception in WLS appears every five minutes to the second! Is there any kind of refresh connection code firing every five minutes? WLS is mad. I've disconnected the client but the exceptions are still appearing in the server logs, every five minutes pretty much to the second! The client has been disconnected for 20 minutes now and it's still printing out the exception. We reckon there's some server side stub still running I've introduced an "exposeAccessContext" flag on JndiObjectFactoryBean and <jee:jndi-lookup>, for WebLogic resource factories that have authorization requirements on their connection retrieval methods. This should work fine for a JMS ConnectionFactory obtained that way, satisfying WebLogic's strange JNDI context requirements... with reasonably little configuration (just add expose-access-context="true" to your <jee:jndi-lookup> tag). The same flag is also available on Spring's EJB accessors now, for access to WebLogic EJBs with authorization requirements. I have no clue what those connection refresh messages are doing there every fine minutes, even for a disconnected client... I'm not aware of anything Spring might do wrong here. If you nevertheless find something we could do about it from Spring's side, let me know! Juergen Juergen, I believe some of the strange WLS server side stack traces are due to multiple Weblogic servers connecting to one server resulting in some kind of cross domain problem. It's (sort of) documented in various locations when one Googles for the stack trace. Could you expand on your fix for this problem? How does one configure the JndiObjectFactoryBean to work around this problem? John It's a simple "exposeAccessContext" flag on JndiObjectFactoryBean: <bean id="..." class="org.springframework.jndi.JndiObjectFactoryBean"> The equivalent is: <jee:jndi-lookup ... expose-access-context="true"/> Juergen I want to make sure I understand correctly, the issue with Spring and WebLogic's need for credentials on the InitialContext has been resolved by the addition of "exposeAccessConext" on JndiObjectFactoryBean? Rather than use a Decorator and init the InitialContext with every connection I was hoping to use a Spring provided resolution. The Decorator works with secure JMS but I removed it and tried the code shown in the last example by Juergen and I am now getting the following: Cannot resolve reference to bean 'queueConnectionFactory' while setting bean prope Here is my config: <bean id="jndiTemplate" class="org.springframework.jndi.JndiTemplate"> <bean id="queueConnectionFactory" class="org.springframework.jndi.JndiObjectFactoryBean"> <bean id="testJmsEventSender" class="troy.jms.EvenSender"> Also, how can <jee:jndi-lookup> be used, I didn't think that lookup accepted the properties required to accept the credentials? Hello, I've had reason to review this problem today and I'm not sure the exposeAccessContext flag actually works. This is what happens to a working configuration when I switch it on: <06-Aug-2009 16:31:11 o'clock BST> <Warning> <Deployer> <BEA-149078> <Stack trace for message 149004 All I did was enable it in the JndiObjectFactoryBean. I'd been previously using my hacked connection factory to create the InitialContext each time a call was made (to a connection factory method), and I thought it may be interesting to switch it off and try the exposeAccessContext flag. I've looked at the code and am not really sure how it's supposed to work either, given the only way to work around this Weblogic "design feature" is to create the InitialContext in the thread that makes a call to the connection factory. John Perhaps a longer stack trace would help: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'jms.core.connectionFactory.y' defined in class path resource [jms-core-context.xml]: Invocation of init method failed; nested exception is java.lang.IllegalAccessError: tried to access class weblogic/jms/client/Reconnectable from class weblogic/jms/client/$Proxy195 (And can we unfix this issue given it's not fixed! Hi, By reading the thread, it seems that people believe that this issue is caused by the fact that weblogic jndi security context is attached to the thread which opens the initial context, in other words, it is thread local. However I have experienced the same issue when trying to send a message to a queue, which I believe that everything happens in the same thread. What I did is (not using any spring code here at all): 1. Created a queue on weblogic server and secure it with certain groups. I was wondering if Spring tries to close the jndi context some where (such as in JndiObjectFactoryBean) after creating the initialContext or after a successful lookup()? I wonder if this explains why the jms client failed in the same thread that did open the initialContext. Thanks, Larry After extensive testing I think this issue is fixed with WebLogic 10.3 and the usage of the following properties (please see also I've created a small demo during the tests which I've attached to this issue. I've also tried to use a WebLogic 9.2 appl2 server, but here I still the following error after restart of appl2. We have worked around that issue by using a plain old Java message putter instead of Spring. |
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Improvement
Resolved
Major
DefaultMessageListenerContainer failover to work with Weblogic JMS and security credentials
.
Hello,
Annoyingly, my fix sometimes works and sometimes does not - I can't work out why! Look at this stack trace and you'll see it's passing through the com.db.websso object (my extension of the default message listener that recreates the InitialContext) yet it still fails to reconnect...
18-04 05:36:03,233 jms.messageListenerContainer.events.ms1-19 INFO WeblogicMessageListenerContainer.refreshConnectionUntilSu
ccessful:793 - Could not refresh JMS Connection - retrying in 5000 ms
java.lang.SecurityException: [Security:090398]Invalid Subject: principals=[eventfetcher]
at weblogic.rjvm.ResponseImpl.unmarshalReturn(ResponseImpl.java:195)
at weblogic.rmi.cluster.ClusterableRemoteRef.invoke(ClusterableRemoteRef.java:338)
at weblogic.rmi.cluster.ClusterableRemoteRef.invoke(ClusterableRemoteRef.java:252)
at weblogic.jndi.internal.ServerNamingNode_921_WLStub.lookup(Unknown Source)
at weblogic.jndi.internal.WLContextImpl.lookup(WLContextImpl.java:374)
at weblogic.jndi.internal.WLContextImpl.lookup(WLContextImpl.java:362)
at weblogic.rmi.cluster.BasicReplicaHandler.refreshReplicaList(BasicReplicaHandler.java:506)
at weblogic.rmi.cluster.BasicReplicaHandler.failOver(BasicReplicaHandler.java:206)
at weblogic.rmi.cluster.ClusterableRemoteRef.invoke(ClusterableRemoteRef.java:257)
at weblogic.jms.frontend.FEConnectionFactoryImpl_922_WLStub.connectionCreateRequest(Unknown Source)
at weblogic.jms.client.JMSConnectionFactory.setupJMSConnection(JMSConnectionFactory.java:238)
at weblogic.jms.client.JMSConnectionFactory.createConnectionInternal(JMSConnectionFactory.java:299)
at weblogic.jms.client.JMSConnectionFactory.createConnection(JMSConnectionFactory.java:205)
at org.springframework.jms.support.JmsAccessor.createConnection(JmsAccessor.java:184)
at org.springframework.jms.listener.AbstractJmsListeningContainer.createSharedConnection(AbstractJmsListeningContaine
r.java:401)
at com.db.websso.jms.WeblogicMessageListenerContainer.createSharedConnection(WeblogicMessageListenerContainer.java:41
)
at org.springframework.jms.listener.AbstractJmsListeningContainer.refreshSharedConnection(AbstractJmsListeningContain
er.java:386)