PersistentMap NotSerializeableException

If you have a java.util.(Hash)Map object, by itself, which contains a map from String to String, you would think it was a Serializeable Map, and nothing more. And it is, until Hibernate gets involved.

When you persist the map, Hibernate add some extra information to it, so that when you retrieve the persisted object, it is no longer a HashMap at all, but an org.hibernate.collection.PersistentMap, which implements the java.util.Map interface. This class claims to be Serializable, and it usually is.

However, as part of all that extra information, Hibernate adds a reference to the object associated with the map. Since that parent object is a hibernate mapped object, I erroneously assumed it would never persist unmapped fields. But I really did not realize it would store the whole object as part of the map that the object belongs to.

In my case, the Object was also additionally holding a reference to a third object, which in turn contained a reference to the map, not just any map, but the PersistentMap, which also held the Request object, and so on, an infinitely recursive reference.

But the real lesson here is not the infinite recursion, but that by going through hibernate to persist a collection, you end up actually storing a hibernate-specific object that contains a lot more data than the java collection itself, so be careful, what goes in is not what comes out: there are potentially storage and performance issues that may result.

CLASSPATH

The jwsc ant task internally invokes Javadoc (probably to look at the JSR 181 tags). This spawns the following error:

An error has occurred while invoking javadoc to inspect your source
files.  This may be due to the fact that $JAVA_HOME/lib/tools.jar does
not seem to be in your system classloader.  One common case in which
this happens is when using the 'ant' tool, which uses a special
context classloader to load classes from tools.jar.

This situation elicits what is believed to a javadoc bug in the initial
release of JDK 1.5.  Javadoc attempts to use its own context classloader
tools.jar but ignores one that may have already been set, which leads
to some classes being loaded into two different classloaders.  The
telltale sign of this problem is a javadoc error message saying that
'languageVersion() must return LanguageVersion - you might see this\nmessage in your process' output.

This will hopefully be fixed in a later release of JDK 1.5; if a new
version of 1.5 has become available, you might be able to solve this\nby simply upgrading to the latest JDK.

Alternatively, you can work around it by simply including
$JAVA_HOME/lib/tools.jar in the java -classpath\nparameter.  If you are running ant, you will need to modify the standard\nant script to include tools.jar in the -classpath.

invoking javadoc is done by com.bea.util.jam.internal.javadoc.JavadocRunner, which uses its own classloader:
int result = Main.execute("JAM", spewWriter, spewWriter, spewWriter, getClass().getName(), args);

Sure enough, explicitly adding $JAVA_HOME/lib/tools.jar to $CLASSPATH *does* fix the problem. However, $JAVA_HOME had to be C:/path/lib/tools.jar, not C:\path\lib\tools.jar nor /cygwin/c/path/lib/tools.jar!

non singleton bean initialization

Bit me.
Sympton: javax.management.InstanceAlreadyExistsException
Cause: one singleton bean a was calling ApplicationContext.getBean(), while another web.xml defined non-singleton bean b was statically calling ApplicationContext.getBean(a)

Doh!

Moral: never load beans statically, put in initialization or class constructor

more WebLogic nuances: expanding archives at runtime is slow

When you deploy in exploded format, WebLogic copies the exploded directory structure into its temporary working directory upon deployment.

If the exploded format contains .jars, those jars are just copied, not expanded.

At runtime, if you reference a class that the classloader finds in the jar, it unpacks the archive at runtime, then loads the class.

"If you are unpacking an archive file that contains other module archive files (for example, an Enterprise Application or Web Service that includes JAR or WAR files) and you want to perform partial updates of those modules, you must expand the embedded archive files as well. Make sure that you unpack each module into a subdirectory having the same name as the archive file. For example, unpack a module named myejb.jar into a /myejb.jar subdirectory of the exploded Enterprise Application directory."

BEA link that hints at issue

Do not know whether this affects deployment in archive format of a war that contains a jar, have not tried.

Thread [[STUCK] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'] (Suspended)
Inflater.inflateBytes(byte[], int, int) line: not available [native method]
Inflater.inflate(byte[], int, int) line: 219
ZipFile$2(InflaterInputStream).read(byte[], int, int) line: 128
DataInputStream.readFully(byte[], int, int) line: 176
JarFile.getBytes(ZipEntry) line: 357
JarFile.initializeVerifier() line: 314
JarFile.getInputStream(ZipEntry) line: 383
GenericClassLoader$JarSource(ZipSource).getInputStream() line: 38
GenericClassLoader$JarSource(ZipSource).getBytes() line: 69
ChangeAwareClassLoader(GenericClassLoader).defineClass(String, Source) line: 250
ChangeAwareClassLoader(GenericClassLoader).findLocalClass(String) line: 227
ChangeAwareClassLoader(GenericClassLoader).findClass(String) line: 195
ChangeAwareClassLoader.findClass(String) line: 54
ChangeAwareClassLoader(ClassLoader).loadClass(String, boolean) line: 306
ChangeAwareClassLoader(ClassLoader).loadClass(String) line: 251
ChangeAwareClassLoader(GenericClassLoader).loadClass(String) line: 130
ChangeAwareClassLoader.loadClass(String) line: 35
ChangeAwareClassLoader(ClassLoader).loadClassInternal(String) line: 319
b.() line: not available
a.() line: not available
bg.Z() line: not available
bg.y(e, boolean) line: not available
bg.a(boolean) line: not available
bg(ag).a(String, bm, boolean) line: not available
bg(ag).if(String, bm, boolean) line: not available
bg.if(String, bm, boolean) line: not available
bg.y(String, bm, boolean, f3, Locale) line: not available
a.a(String, Locale) line: not available
g.if(FetchDataDocumentRequest, Locale) line: not available
JRCCommunicationAdapter.request(int, int, String, ISecurityContext, IXMLSerializable) line: not available
y.a(ISecurityContext, String, int, int, IXMLSerializable, k) line: not available
r.a(int, int, IXMLSerializable, k) line: not available
ReportClientDocument.if(int, int, IXMLSerializable) line: not available
ReportClientDocument.if(PropertyBag, int) line: not available
ReportClientDocument.a(PropertyBag, int) line: not available
ReportClientDocument(ClientDocument).open(Object, int) line: not available
ReportClientDocument.open(Object, int) line: not available
GRV.sR(HttpSession, HttpServletRequest, HttpServletResponse, ServletContext, OutputStream) line: 71
RMSBImpl.sR(HttpSession, HttpServletRequest, HttpServletResponse, ServletContext) line: 255
rRR.jsp line: 3
__rrr(JspBase).service(ServletRequest, ServletResponse) line: 34
StubSecurityHelper$ServletServiceAction.run() line: 225
StubSecurityHelper.invokeServlet(ServletRequest, HttpServletRequest, ServletRequestImpl, ServletResponse, HttpServletResponse, Servlet) line: 127
JavelinxJSPStub(ServletStubImpl).execute(ServletRequest, ServletResponse, FilterChainImpl) line: 272
ServletStubImpl.onAddToMapException(Throwable, ServletRequestImpl, ServletRequest, ServletResponse, FilterChainImpl) line: 380
ServletStubImpl.execute(ServletRequest, ServletResponse, FilterChainImpl) line: 298
ServletStubImpl.execute(ServletRequest, ServletResponse) line: 165
RequestDispatcherImpl.invokeServlet(boolean, ServletRequest, ServletResponse, ServletResponseImpl, int) line: 493
RequestDispatcherImpl.forward(ServletRequest, ServletResponse) line: 245
TilesRequestProcessor(RequestProcessor).doForward(String, HttpServletRequest, HttpServletResponse) line: 1085
TilesRequestProcessor(TilesRequestProcessor).doForward(String, HttpServletRequest, HttpServletResponse) line: 263
TilesRequestProcessor.doForward(String, HttpServletRequest, HttpServletResponse) line: 74
TilesRequestProcessor(RequestProcessor).processForwardConfig(HttpServletRequest, HttpServletResponse, ForwardConfig) line: 398
TilesRequestProcessor(TilesRequestProcessor).processForwardConfig(HttpServletRequest, HttpServletResponse, ForwardConfig) line: 318
TilesRequestProcessor(RequestProcessor).process(HttpServletRequest, HttpServletResponse) line: 241
TilesRequestProcessor.process(HttpServletRequest, HttpServletResponse) line: 89
ActionServlet.process(HttpServletRequest, HttpServletResponse) line: 1196
ActionServlet.doPost(HttpServletRequest, HttpServletResponse) line: 432
ActionServlet(HttpServlet).service(HttpServletRequest, HttpServletResponse) line: 763
ActionServlet(HttpServlet).service(ServletRequest, ServletResponse) line: 856
StubSecurityHelper$ServletServiceAction.run() line: 225
StubSecurityHelper.invokeServlet(ServletRequest, HttpServletRequest, ServletRequestImpl, ServletResponse, HttpServletResponse, Servlet) line: 127
ServletStubImpl.execute(ServletRequest, ServletResponse, FilterChainImpl) line: 272
TailFilter.doFilter(ServletRequest, ServletResponse, FilterChain) line: 26
FilterChainImpl.doFilter(ServletRequest, ServletResponse) line: 42
SetCharacterEncodingFilter.doFilter(ServletRequest, ServletResponse, FilterChain) line: 32
FilterChainImpl.doFilter(ServletRequest, ServletResponse) line: 42
WebAppServletContext$ServletInvocationAction.run() line: 3151
AuthenticatedSubject.doAs(AbstractSubject, PrivilegedAction) line: 321
SecurityManager.runAs(AuthenticatedSubject, AuthenticatedSubject, PrivilegedAction) line: 121
WebAppServletContext.securedExecute(HttpServletRequest, HttpServletResponse, boolean) line: 1973
WebAppServletContext.execute(ServletRequestImpl, ServletResponseImpl) line: 1880
ServletRequestImpl.run() line: 1310
ExecuteThread.execute(Runnable) line: 207
ExecuteThread.run() line: 179

Internet Explorer bug is a feature

Let's see. I am generating a web page for a user to see. It expires immediately, so I apply the HTTP "Cache-Control" setting to "no-cache".

What does Internet Explorer do with that? Refuses to display it. After all, if it can't cache it, it tries to be "security conscious" and refuses to put it on the filesystem, and IE can't display any content unless its rendered to the filesystem.

IE bug 316431

Use Cache-Control: must-revalidate, max-age=0
to get around this non-compliant behaviour.

hellish Hibernate blobs under WebLogic with Oracle

There were bugs in the Oracle driver that caused problems reading blobs under Hibernate. Basically the byte array can be written to Oracle, but what's read back from Oracle is always 86 bytes which is the Oracle Blob locator (pointer), not the blob.

Now, this has long since been fixed by Oracle (thin on or before 10.1.0.4) and Hibernate (on or before 3.05), yet the problem continues to exist under WebLogic 9.1. Stack trace looks something like BEA thinks they have a better way of handling storing blobs than Oracle (see CR112225). In any event, one workaround is:
-Dhibernate.jdbc.use_streams_for_binary=true
<property name="content" type="binary" column="CONTENT" />

(note, in 3.05 at least, this property is not read from the hibernate configuration file, regardless of what the hibernate documentation actually says).

But perhaps would be better to actually follow the CR112225 advice and use
<serialize-byte-array-to-oracle-blob>

The WebLogic description of CR11225 is correct as to the cause, but is misleading and inaccurate regarding the extent or consequence: their example of using weblogic.jdbc.common.OracleBloc is merely an example, not the cause, and the problem was seen with blobs created and stored under WebLogic 9.1, not "some other vendor".

From http://e-docs.bea.com/wls/docs81/pdf/notes.pdf:

WebLogic Server 8.1 Release Notes

Resolved Problems for Service Pack 2
7-22

CR112225

For BLOBs, in the generated code, WebLogic Server calls
ObjectOutputStream.writeObject and
ObjectInputStream.readObject to serialize/deserialize the object before
writing/reading it to the database. These calls add extra header information. The
writeObject method writes the class of the object, the signature of the class, the values
of the non-transient and non-static fields of the class, and all of its supertypes are written.
(This is the reason for the extra header seen in the database.) These calls do not cause a
problem when customers are using only WebLogic Server to set and get BLOBs, because it
uses readObject to convert the bytes into the appropriate object, which needs that extra
header information. However, if the BLOB has been inserted directly into the database by
some other vendor or programmer using:
OutputStream os = ((weblogic.jdbc.common.OracleBlob)
lob).getBinaryOutputStream(); os.write(this.tiffImage); //
byte[] tiffImage
then problems may occur because WebLogic Server uses readObject and the header
information is missing. For the data inserted using WebLogic Server , the other programs that
would read the bytes directly get the extra header information and fail.
<serialize-byte-array-to-oracle-blob> has been added to control the
persistence behavior. This element is used to specify whether a cmp-field of type
byte[] mapped to an OracleBlob should be serialized. The tag has been added to a new
compatibility stanza in the weblogic-cmp-rdbms descriptor.
Note that, in versions prior to SP2, the default behavior was to serialize a cmp-field of
type byte[] mapped to an OracleBlob. Now, the byte[] is written directly to the
OutputStream obtained from the BLOB. To revert to the old behavior, set the value of this
tag to true.

Fixed Release: 8.1 SP2

ORACLE: Io exception: Got minus one from a read call

Using Oracle Shared Server Processes: http://www.mid.main.vsu.ru/docs/oracle/network.816/a76933/mtsa.gif


Incoming requests hit the TNS listener, and are held in a queue to obtain a shared server process. A broken pipe occurred during the handoff to a shared server process. We still do not know which pipe broke, nor why the Oracle and other web forums report this as a possibility on HP and Linux hosts, whereas on Solaris and Windows, we observed a more appropriate Oracle ORA-12519 �TNS:no appropriate service handler found� instead.

Alternative is to use dedicated server processes:
http://www.mid.main.vsu.ru/docs/oracle/network.816/a76933/net81036.gif

Although there were 300 shared server processes configured, the error always occurred on the 283rd. If one used dedicated server processes, you could configure your deployments exactly match the number of server processes to the number of database pool connections, rather than attempting to guess what the algorithm is when breakdown occurs. 300 Oracle server processes just was not enough for the load experienced.

Error seen in Oracle log (DB Server side):
03-AUG-2006 21:29:36 * (CONNECT_DATA=(SID=sid)(CID=(PROGRAM=)(HOST=__jdbc__)(USER=))) * (ADDRESS=(PROTOCOL=tcp)(HOST=10.10.2.2)(PORT=48460)) * establish * sid * 12518
TNS-12518: TNS:listener could not hand off client connection
TNS-12547: TNS:lost contact
TNS-12560: TNS:protocol adapter error
TNS-00517: Lost contact
Linux Error: 32: Broken pipe

Stack trace seen on server (DB Connection client side):
2006 Aug 03 21:28:57 [ACTIVE] ExecuteThread: '63' for queue: 'weblogic.kernel.Default (self-tuning)' ERROR: 1656935/:: DB Connection unavailable.
java.sql.SQLException: Pool connect failed : weblogic.common.ResourceException: 8:weblogic.common.ResourceException: Could not create pool connection. The DBMS driver exception was: Io exception: Got minus one from a read call
at weblogic.jdbc.common.internal.JDBCUtil.wrapAndThrowResourceException(JDBCUtil.java:241)
at weblogic.jdbc.pool.Driver.connect(Driver.java:161)
at weblogic.jdbc.jts.Driver.getNonTxConnection(Driver.java:647)
at weblogic.jdbc.jts.Driver.connect(Driver.java:137)
at weblogic.jdbc.common.internal.RmiDataSource.getConnection(RmiDataSource.java:359)
at dbpool.DBConnectionCache.getConnection(DBConnectionCache.java:101)

Stack trace as seen by Weblogic (DB Connection client side):
java.sql.SQLException: Io exception: Got minus one from a read call
at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:125)
at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:162)
at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:274)
at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:328)
at oracle.jdbc.driver.PhysicalConnection.(PhysicalConnection.java:348)
at oracle.jdbc.driver.T4CConnection.(T4CConnection.java:151)
at oracle.jdbc.driver.T4CDriverExtension.getConnection(T4CDriverExtension.java:32)
at oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:563)
at weblogic.jdbc.common.internal.ConnectionEnvFactory.makeConnection(ConnectionEnvFactory.java:301)
at weblogic.jdbc.common.internal.ConnectionEnvFactory.createResource(ConnectionEnvFactory.java:205)
at weblogic.common.resourcepool.ResourcePoolImpl.makeResources(ResourcePoolImpl.java:1049)
at weblogic.common.resourcepool.ResourcePoolImpl.makeResources(ResourcePoolImpl.java:977)
at weblogic.common.resourcepool.ResourcePoolImpl.reserveResourceInternal(ResourcePoolImpl.java:372)
at weblogic.common.resourcepool.ResourcePoolImpl.reserveResource(ResourcePoolImpl.java:295)
at weblogic.common.resourcepool.ResourcePoolImpl.reserveResource(ResourcePoolImpl.java:285)
at weblogic.jdbc.common.internal.ConnectionPool.reserve(ConnectionPool.java:455)
at weblogic.jdbc.common.internal.ConnectionPool.reserve(ConnectionPool.java:346)
at weblogic.jdbc.common.internal.ConnectionPoolManager.reserve(ConnectionPoolManager.java:83)
at weblogic.jdbc.common.internal.ConnectionPoolManager.reserve(ConnectionPoolManager.java:96)
at weblogic.jdbc.pool.Driver.connect(Driver.java:150)
at weblogic.jdbc.jts.Driver.getNonTxConnection(Driver.java:647)
at weblogic.jdbc.jts.Driver.connect(Driver.java:137)
at weblogic.jdbc.common.internal.RmiDataSource.getConnection(RmiDataSource.java:359)
at dbpool.DBConnectionCache.getConnection(DBConnectionCache.java:101)