RE: ISCSI: Urgent Flag requirement violates TCP.

["Y P Cheng" <ycheng@advansys.com>]

Dave Black wrote:
> It is not clear to me that the Urgent feature is "required for
> interoperation or to limit behavior which has potential for
> causing harm". I'm prepared to be convinced otherwise, and would
> like to hear from implementers other than Matt on this subject,
> and specifically comments on his statement that:
>    "... high speed implementations will require framing in order
> 	to prevent a massive amount of buffer resources to 'buffer up' TCP
> 	segments that arrive after a dropped TCP segment."

My apology for taking so long to respond to this request.

I support Matt whole heartedly.  A TCP-Offload-Engine (TOE), a hardware-aid
TCP implementation doing zero-copy TCP, is essential for Gb-plus Ethernet
and Fibre Channel and InfiniBand adapters supporting iSCSI over TCP.  Using
urgent bit to identify the beginning of an iSCSI message enables the TOE to
parse an incoming TCP/IP segment quickly and deals with out-of-order and
duplicated frames efficiently. Most arguments against Matt's position were
based on existing software TCP implementation.  While supporting TCP 100%,
the TOE adapter does require some changes to the TCP implementation at
installation. The change is necessary to enable the zero-copy function.
However, an TOE adapter with its hardware and software will inter-operate
with any existing TCP implementation on any client or server.

An TOE is a multi-function adapter that supports both TCP/IP and iSCSI.  The
NFS implementation with UDP or TCP over IP can be supported by a
scatter/gather DMA list which splits the TCP/IP header from the data payload
such that the data payload are copied directly from and to the NFS cache
buffers.  This is the essence of the zero-copy function. To deal with
out-of-order and duplicated frames, the TOE adapter works with one IP packet
at a time with a score card that tracks all incoming frames.  The maximum IP
packet size is 65K.  Of course, some implementations call for "jumbo"
packets or frames.  Inside each IP packet, the TOE adapter finds the UDP/TCP
segments.

For iSCSI support, the TOE adapter receives its SCSI requests directly from
the iSCSI driver instead from a TCP/IP driver.  As a target, the TOE adapter
passes incoming SCSI commands directly back to waiting application software
like RAID or tape or JBOD storage devices.  For outgoing frames or packets,
the TOE adapter creates TCP segments with IP headers.  It will bundle
several iSCSI PDUs in one TCP/IP packet destined for the same target.  For
incoming frames, the TOE adapter must know the iSCSI message boundary.  This
is why the urgent bit is extremely useful.  Without it, the TOE adapter must
buffer the whole IP packet before it can process an iSCSI header.  While an
TOE adapter can deal with a 65K IP packet with ease, the "jumbo" frame
places a large SRAM requirement on the adapter.  Several incoming packets
from different sources just aggregate the SRAM requirements.  For iSCSI,
jumbo frames are very useful for clients or servers thousands miles away.

Many objections of the "MUST" word were based on out-of-order delivery of
SCSI commands.  For outgoing SCSI commands, the TOE adapter will deliver
them in the same order received from its iSCSI driver.  For incoming SCSI
commands, an TOE adapter operates in the same manner as current SCSI, 1394,
and fibre channel adapters, meaning, no guarantee to in-order command
execution.

For a SCSI adapter lets use an example of command A being followed by
command B.  If a target gets the command A with bus parity check, it would
return check status for command A and proceed to accept B happily, even
there is dependency between command A and B.  Several weeks ago, Bob Snively
and I posted messages stating that this was OK because if B depended on A,
all file system software would hold command B until the completion of
command A.

For a 1394 adapter both command A and B are stored in ORB's.  A target 1394
device will fetch the ORB's.  Again, after encountering error in fetching
the ORB for A, a target device is more than happy to proceed to fetch the
ORB for B.

For a fibre channel adapter, the initiator will send command A and B in two
separate FCP_CMD frames.  If frame A arrives with bad CRC, a target device
simply throws the frame away and proceeds with execution of command B, if it
is arrived with good CRC.

Y.P. Cheng, CTO, ConnectCom Solutions Corp.