Thursday, August 31, 2006

hellish Hibernate blobs under WebLogic with Oracle

There were bugs in the Oracle driver that caused problems reading blobs under Hibernate. Basically the byte array can be written to Oracle, but what's read back from Oracle is always 86 bytes which is the Oracle Blob locator (pointer), not the blob.

Now, this has long since been fixed by Oracle (thin on or before 10.1.0.4) and Hibernate (on or before 3.05), yet the problem continues to exist under WebLogic 9.1. Stack trace looks something like BEA thinks they have a better way of handling storing blobs than Oracle (see CR112225). In any event, one workaround is:
-Dhibernate.jdbc.use_streams_for_binary=true
<property name="content" type="binary" column="CONTENT" />


(note, in 3.05 at least, this property is not read from the hibernate configuration file, regardless of what the hibernate documentation actually says).

But perhaps would be better to actually follow the CR112225 advice and use
<serialize-byte-array-to-oracle-blob>

The WebLogic description of CR11225 is correct as to the cause, but is misleading and inaccurate regarding the extent or consequence: their example of using weblogic.jdbc.common.OracleBloc is merely an example, not the cause, and the problem was seen with blobs created and stored under WebLogic 9.1, not "some other vendor".

From http://e-docs.bea.com/wls/docs81/pdf/notes.pdf:


WebLogic Server 8.1 Release Notes

Resolved Problems for Service Pack 2
7-22

CR112225

For BLOBs, in the generated code, WebLogic Server calls
ObjectOutputStream.writeObject and
ObjectInputStream.readObject to serialize/deserialize the object before
writing/reading it to the database. These calls add extra header information. The
writeObject method writes the class of the object, the signature of the class, the values
of the non-transient and non-static fields of the class, and all of its supertypes are written.
(This is the reason for the extra header seen in the database.) These calls do not cause a
problem when customers are using only WebLogic Server to set and get BLOBs, because it
uses readObject to convert the bytes into the appropriate object, which needs that extra
header information. However, if the BLOB has been inserted directly into the database by
some other vendor or programmer using:
OutputStream os = ((weblogic.jdbc.common.OracleBlob)
lob).getBinaryOutputStream(); os.write(this.tiffImage); //
byte[] tiffImage
then problems may occur because WebLogic Server uses readObject and the header
information is missing. For the data inserted using WebLogic Server , the other programs that
would read the bytes directly get the extra header information and fail.
<serialize-byte-array-to-oracle-blob> has been added to control the
persistence behavior. This element is used to specify whether a cmp-field of type
byte[] mapped to an OracleBlob should be serialized. The tag has been added to a new
compatibility stanza in the weblogic-cmp-rdbms descriptor.
Note that, in versions prior to SP2, the default behavior was to serialize a cmp-field of
type byte[] mapped to an OracleBlob. Now, the byte[] is written directly to the
OutputStream obtained from the BLOB. To revert to the old behavior, set the value of this
tag to true.

Fixed Release: 8.1 SP2

Wednesday, August 30, 2006

ORACLE: Io exception: Got minus one from a read call

Using Oracle Shared Server Processes: http://www.mid.main.vsu.ru/docs/oracle/network.816/a76933/mtsa.gif


Incoming requests hit the TNS listener, and are held in a queue to obtain a shared server process. A broken pipe occurred during the handoff to a shared server process. We still do not know which pipe broke, nor why the Oracle and other web forums report this as a possibility on HP and Linux hosts, whereas on Solaris and Windows, we observed a more appropriate Oracle ORA-12519 �TNS:no appropriate service handler found� instead.

Alternative is to use dedicated server processes:
http://www.mid.main.vsu.ru/docs/oracle/network.816/a76933/net81036.gif

Although there were 300 shared server processes configured, the error always occurred on the 283rd. If one used dedicated server processes, you could configure your deployments exactly match the number of server processes to the number of database pool connections, rather than attempting to guess what the algorithm is when breakdown occurs. 300 Oracle server processes just was not enough for the load experienced.

Error seen in Oracle log (DB Server side):
03-AUG-2006 21:29:36 * (CONNECT_DATA=(SID=sid)(CID=(PROGRAM=)(HOST=__jdbc__)(USER=))) * (ADDRESS=(PROTOCOL=tcp)(HOST=10.10.2.2)(PORT=48460)) * establish * sid * 12518
TNS-12518: TNS:listener could not hand off client connection
TNS-12547: TNS:lost contact
TNS-12560: TNS:protocol adapter error
TNS-00517: Lost contact
Linux Error: 32: Broken pipe

Stack trace seen on server (DB Connection client side):
2006 Aug 03 21:28:57 [ACTIVE] ExecuteThread: '63' for queue: 'weblogic.kernel.Default (self-tuning)' ERROR: 1656935/:: DB Connection unavailable.
java.sql.SQLException: Pool connect failed : weblogic.common.ResourceException: 8:weblogic.common.ResourceException: Could not create pool connection. The DBMS driver exception was: Io exception: Got minus one from a read call
at weblogic.jdbc.common.internal.JDBCUtil.wrapAndThrowResourceException(JDBCUtil.java:241)
at weblogic.jdbc.pool.Driver.connect(Driver.java:161)
at weblogic.jdbc.jts.Driver.getNonTxConnection(Driver.java:647)
at weblogic.jdbc.jts.Driver.connect(Driver.java:137)
at weblogic.jdbc.common.internal.RmiDataSource.getConnection(RmiDataSource.java:359)
at dbpool.DBConnectionCache.getConnection(DBConnectionCache.java:101)

Stack trace as seen by Weblogic (DB Connection client side):
java.sql.SQLException: Io exception: Got minus one from a read call
at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:125)
at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:162)
at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:274)
at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:328)
at oracle.jdbc.driver.PhysicalConnection.(PhysicalConnection.java:348)
at oracle.jdbc.driver.T4CConnection.(T4CConnection.java:151)
at oracle.jdbc.driver.T4CDriverExtension.getConnection(T4CDriverExtension.java:32)
at oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:563)
at weblogic.jdbc.common.internal.ConnectionEnvFactory.makeConnection(ConnectionEnvFactory.java:301)
at weblogic.jdbc.common.internal.ConnectionEnvFactory.createResource(ConnectionEnvFactory.java:205)
at weblogic.common.resourcepool.ResourcePoolImpl.makeResources(ResourcePoolImpl.java:1049)
at weblogic.common.resourcepool.ResourcePoolImpl.makeResources(ResourcePoolImpl.java:977)
at weblogic.common.resourcepool.ResourcePoolImpl.reserveResourceInternal(ResourcePoolImpl.java:372)
at weblogic.common.resourcepool.ResourcePoolImpl.reserveResource(ResourcePoolImpl.java:295)
at weblogic.common.resourcepool.ResourcePoolImpl.reserveResource(ResourcePoolImpl.java:285)
at weblogic.jdbc.common.internal.ConnectionPool.reserve(ConnectionPool.java:455)
at weblogic.jdbc.common.internal.ConnectionPool.reserve(ConnectionPool.java:346)
at weblogic.jdbc.common.internal.ConnectionPoolManager.reserve(ConnectionPoolManager.java:83)
at weblogic.jdbc.common.internal.ConnectionPoolManager.reserve(ConnectionPoolManager.java:96)
at weblogic.jdbc.pool.Driver.connect(Driver.java:150)
at weblogic.jdbc.jts.Driver.getNonTxConnection(Driver.java:647)
at weblogic.jdbc.jts.Driver.connect(Driver.java:137)
at weblogic.jdbc.common.internal.RmiDataSource.getConnection(RmiDataSource.java:359)
at dbpool.DBConnectionCache.getConnection(DBConnectionCache.java:101)

sending/receiving files with JAX-RPC & Axis 1.3

Use javax.activication.DataHandler
because it can stream: reduces memory usage (don't have to hold file in core), and is faster (can start to send/receive before whole object is created/read).

MIME TYPE: versus MIME Type
MIME is our standard for encoding DataHandler streams. The actual over-the-wire XML is a multipart MIME message, but unlike one might have thought, the SOAP response is one attachment and the attachment is another attachment, rather than the file being an attachment to the SOAP message.

In any event, Axis 1.3 doesn't handle it, invents its own DataHandler type instead. No good for non-Axis clients.

The Transferable interface, spot the difference:
1.new DataHandler(new String("foo"), "text/csv"))
new DataHandler(new String("foo"), "text/plain"))

This is silly. The client code chooses the object to decode the data handler based on its MIME type, not the type of the object that it was created with. These are both Strings and could be de-serialized in exactly the same way. But noooo, a built-in handler exists for text/plain (String), but not for text/csv.

I ended up using
new DataHandler(new org.apache.commons.mail.ByteArrayDataSource(os.toByteArray(), "text/csv"));

even though this doesn't really take advantage of the streaming capability or reduce the memory requirements. Should really make an inputstream out of the outputstream, and stream the stream.

WebLogic listens on one interface

Setting the listen-address attribute it a Bad Idea. Hosts are multi-homed, and if you want to restrict access, don't stick your application on a machine that has an interface to the network you don't want it accessed from.

The WebLogic documents describe very accurately the considerations (except their last point about Windows NT machines is a bit whacked):

Listen Address Considerations

The following table describes important considerations related to setting the listen address value.

If the Listen Address is set to . . .

Then, the following is true . . .

IP address or DNS name

  • Processes must specify the IP address or DNS name to connect to the server instance.

  • Clients that specify localhost will fail to connect.

  • You must update existing processes that use localhost to connect to the server instance.

Note: To resolve a DNS name to an IP address, WebLogic Server must be able to contact an appropriate DNS server or obtain the IP address mapping locally. Therefore, if you specify a DNS name for the listen address, you must either leave a port open long enough for the WebLogic Server instance to connect to a DNS server and cache its mapping or you must specify the IP address mapping in a local file. If you specify an IP address for the listen address and then a client request specifies a DNS name, WebLogic Server will attempt to resolve the DNS name, but if it cannot access DNS name mapping, the request will fail.

localhost

  • Processes must specify localhost to connect to the server instance.

  • Only processes that reside on the machine that hosts the server instance (local processes) will be able to connect to the server instance.

  • Remote (non-local) processes will not be able to connect to the server instance.

Undefined or Blank ("")

  • Processes can specify the IP address, DNS name, or localhost to connect to the server instance.

  • Processes that specify localhost must reside on the machine that hosts the server instance.

  • If the server instance must be accessible as localhost (for instance, if you have administrative scripts that connect to localhost), and must also be accessible by remote processes, leave the listen address blank.

Note: For WebLogic servers running on multi-homed Windows NT machines, you should not leave the listen address value undefined or blank. (Multi-homed machines are configured with multiple IP addresses.) Otherwise, the WebLogic Server reserves and listens on its port for each of the machine IP addresses. This precludes other servers from using the same port on the machine.


I should really stop regurgitating information that is readily available on the web. The original point of this blog was to document findings that were not documented elsewhere.

Back to basics...

datetime becomes date only

This one surfaced again today:

java code says:
object.setEventTime(new Date());

java.util.Date "Allocates a Date object and initializes it so that it represents the time at which it was allocated, measured to the nearest millisecond."

calls to
object.getEventTime();
give a date that has no time: 2006.08.30 00:00:00

Look at DB, contents are: 2006.08.30, type is an Oracle "Date" which is stored to seconds granularity
http://www-db.stanford.edu/~ullman/fcdb/oracle/or-time.html

Culprit: Hibernate mapping. Use:
<property name="eventTime" column="EVENT_TIME" type="timestamp"/>
instead of
<property name="eventTime" column="EVENT_TIME" type="date">

See
http://www.hibernate.org/hib_docs/v3/api/org/hibernate/type/TimestampType.html

loosely coupled software engineering artefacts

What happens?

Your requirements document says one thing, your design documents say another, the code implements the buggy version, your test cases tests the behaviour QA thinks the product ought to have. If you're lucky enough to have use cases you probably aren't lucky enough to have someone update them when the implementor found a more brilliant way of handling some case, and your documentation refers to functionality present in the last version.

Traceability is a start. Tools that provide the links are better. What tools are those?

Tuesday, August 29, 2006

WLST exceptions

Property “Name” of Server with original name “_sample” is invalid. The property value is duplicated

Actually means that when you read the domain, the existing server in config.xml had not been started with AdminServer running (which creates the server directory for the domain).

Similarly, you must give a JDBC System Resource a name explicitly:

jdbcSR=create(dsName, 'JDBCSystemResource')
jdbcSR.setName(dsName)

WLSTException: 'Error occured while performing deploy : Error deploying the applicationError occured while performing deploy : Unexpected Error.Error occured while performing deploy : Could not find Targets that match mdm-weblogic-8089 in null Use dumpStack() to view the full stacktrace Use dumpStack() to view the full stacktrace'

The “null” is the list of available servers to deploy to. Can look at {domain}\config\config.xml and see that new server did get written, but deploy is not viewing that server as an acceptable target to deploy to.

This is because the AdminServer was (already) running. The offline script updated the config.xml, but the already running AdminServer did not pick up the change, so it did not see any target it could deploy to.

Monday, August 28, 2006

getting thread dumps out of WebLogic 9.1

I've solved far too many technical issues that remain undocumented. This blog is my attempt to log useful solutions to issues I have run across, as they happen. Our first entry is not of my authoring.

Here’s some useful information on getting thread dumps out of WebLogic 9.1 (good luck looking for the thread dumps if you ‘kill –QUIT’ on the JVM pid):

Add the following variables to your environment:

1. Set the following environment variables in your shell:

CLASSPATH=${BEA_HOME}/patch_weblogic910/profiles/default/sys_manifest_classpath/weblogic_patch.jar:${BEA_HOME}/jdk150_04/lib/tools.jar:${BEA_HOME}/weblogic91/server/lib/weblogic_sp.jar:${BEA_HOME}/weblogic91/server/lib/weblogic.jar:${BEA_HOME}/weblogic91/server/lib/webservices.jar

PATH=${BEA_HOME}/weblogic91/server/bin:${BEA_HOME}/jdk150_04/jre/bin:${BEA_HOME}/jdk150_04/bin:${PATH}

2. Use the WebLogic WebLogic Scripting Tool (WLST) and connect to the server of interest:

# java weblogic.WLST

Initializing WebLogic Scripting Tool (WLST) ...

Welcome to WebLogic Server Administration Scripting Shell

Type help() for help on available commands

wls:/offline> connect('weblogic', 'weblogic', 't3://sunmdm7:8095')

Connecting to weblogic server instance running at t3://sunmdm7:8095 as username weblogic ...

Successfully connected to Admin Server 'mdm-8095' that belongs to domain 'mdm_mdm-8095'.

Warning: An insecure protocol was used to connect to the server.

To ensure on-the-wire security, the SSL port or Admin port

should be used instead.

3. Use the WLST command threadDump()

Syntax

threadDump([writeToFile], [fileName], [serverName])

Argument

Definition

writeToFile

Optional. Boolean value specifying whether to save the output to a file. This argument defaults to true, indicating that output is saved to a file.

fileName

Optional. Name of the file to which the output is written. The filename can be absolute or relative to the directory where WLST is running. This argument defaults to Thread_Dump_serverName file, where serverName indicates the name of the server. This argument is valid only if writeToFile is set to true.

serverName

Optional. Server name for which the thread dump is requested. This argument defaults to the server to which WLST is connected.

If you are connected to an Administration Server, you can display a thread dump for the Administration Server and any Managed Server that is running in the domain. If you are connected to a Managed Server, you can only display a thread dump for that Managed Server.

wls:/mdm_mdm-8095/serverConfig> threadDump()

Thread dump for the running server: mdm-8095

"[STANDBY] ExecuteThread: '8' for queue: 'weblogic.kernel.Default (self-tuning)'" waiting for lock weblogic.work.ExecuteThread@af71f0 WAITING

java.lang.Object.wait(Native Method)

java.lang.Object.wait(Object.java:474)

weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:163)

weblogic.work.ExecuteThread.run(ExecuteThread.java:184)

"[STANDBY] ExecuteThread: '7' for queue: 'weblogic.kernel.Default (self-tuning)'" waiting for lock weblogic.work.ExecuteThread@61806f WAITING

java.lang.Object.wait(Native Method)

java.lang.Object.wait(Object.java:474)

weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:163)

weblogic.work.ExecuteThread.run(ExecuteThread.java:184)

"[STANDBY] ExecuteThread: '6' for queue: 'weblogic.kernel.Default (self-tuning)'" waiting for lock weblogic.work.ExecuteThread@1373527 WAITING

java.lang.Object.wait(Native Method)

java.lang.Object.wait(Object.java:474)

weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:163)

weblogic.work.ExecuteThread.run(ExecuteThread.java:184)

"Timer-11" waiting for lock java.util.TaskQueue@182b8fe TIMED_WAITING

java.lang.Object.wait(Native Method)

java.util.TimerThread.mainLoop(Timer.java:509)

java.util.TimerThread.run(Timer.java:462)

"DynamicListenThread[Default]" RUNNABLE native

java.net.PlainSocketImpl.socketAccept(Native Method)

java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)

java.net.ServerSocket.implAccept(ServerSocket.java:450)

java.net.ServerSocket.accept(ServerSocket.java:421)

weblogic.socket.WeblogicServerSocket.accept(WeblogicServerSocket.java:34

. . . .