File Adapter And Partial Stream Reads

I’ve been playing for the last couple of days with BizTalk Server 2006, building a
custom encoder pipeline component. One of the things I’ve been trying to do is finding
a way to do all encoding operations in a streaming fashion, by building a pass-through
stream implementation that only reads and encodes small portions of the message as
they are read by the outbound adapter.

One of the options I’ve been experimenting with was to do partial reads and writes
on an intermediate memory stream: Instead of reading and encoding the entire body
part of the message in a single pass in memory, and then returning that from the encoder
component, I only read from the original stream as much as the adapter asks me for
on the Stream.Read() implementation, encode and write than into the intermediate memory
stream, and then read back from it and return it to the adapter.

I realize it sounds a little convoluted, but it’s the easiest way to do it with
the library I’m using to do the encoding. One of the reasons why this works fairly
well is that the encoding process does some compression, and so, it will be the case
that whatever I read and encode from the original stream will be smaller than what
originally asked for. For example, if someone tried to read 64KB from my custom encoding
stream, I might return just a couple of KB even if there’s further data in the original
stream. Granted, it is not the most efficient implementation, but it ensures I use
little memory during the encoding operations.

Note: This is not a problem in the .NET Stream API; if you’re reading from a stream
you must be prepared to deal with partial reads. A partial read does not mean
that you’ve read the end of the stream nor that a problem was encountered.

Now, I know this works, as I unit tested the encoding stream and component in isolation
(using my PipelineTesting library).
However, when I went to try my custom pipeline component in a real messaging scenario
with the File adapter, it failed, and miserably: The BizTalk host would pretty much
crash (after a huge spike of 100% processor usage) with the following error: “The
parameter is incorrent”. Humm, not much useful.

At this point I took out the debugger and attached to BTSNTSvc.exe to try and repro
the error. I was able to track my custom pipeline component getting called, and see
BizTalk read off my custom stream. At this point I noticed weird things.

The first thing I noticed was that the file adapter (or is it the BizTalk messaging
engine itself?) uses very small buffers to read of the message streams. Indeed, it
only reads it in 4KB chunks. That seems rather small to me, particularly considering
the fact that the FILE adapter is an unmanaged adapter and so each Read() call into
the stream will cause unmanaged<->managed transitions which are costly.
I would’ve expected it to at least use buffers of 64KB, but maybe there’s a good reason
for that.

That by itself was not too much of an issue; my component was perfectly capable of
dealing with that and indeed I had unit tests using both 4KB and 64KB buffers (though
I believe it is the cause of the poor performance and hight CPU usage I noticed).
The real problem was that my stream would almost always do a partial read, and the
adapter seemed unable to cope for that, as it started asking for weird buffer sizes
on consecutive Read() calls.

Let’s see a small table that explains how it called each time (this was on a run with
a 5MB file):

As you can probably guess by now, what seems to be happening is that if the stream
does a partial read, then the next time around the adapter asks for a read of (buffer.Size-bytesRead) length
(or close enough). Eventually, that length reaches zero if the stream hasn’t been
totally consumed, which in this case causes an exception as it is an invalid parameter
value. I’m not sure if this is a bug in the file adapter, or if it’s simply a side-effect
of the way the managed<->unmanaged code interaction happens at the messaging
engine, but I though it was something worth looking at more closely.

I’m planning on working around this by doing a some extra things to try as much as
possible to do full reads and avoid the partial ones (as the ones I’m doing now are
obviously innefficient, but that’s caused partly because the input file is highly
compressible). Hopefully that should make this a non-issue.

Technorati: BizTalk