RE: iSCSI/iWARP drafts and flow control

[pat_thaler@agilent.com]

Caitlin,

There are multiple problems with the mechanism you suggest. 

For one thing, the credit for additional command window in iSCSI isn't based on command replys. There is an explicit field for MaxCmdSN. Having received and finished processing a command doesn't mean that one has made space for receiving another command. The target might choose to use that space for another session or for something else altogether.

Secondly, having finished a processing a command doesn't mean that other activities have completed. If we did what you suggest there might be times when a response had to be delayed because the command had been finished but the credits for the non-command messages were not yet ready. This could adversely effect performance. Also, commands can involve lengthy actions such as large data transfers. Therefore, there might be times when one is not ready to send a command reply and immediate commands or other non-command related traffic are delayed because of lack of credit.

Third, it is an accounting nightmare for the implementations. Instead of just tracking a simple credit counter and updating it, they have to keep track of how many credits belong to each command.

Forth, the suggestion only takes care of one direction. MaxCmdSN goes from target to initiator, but there are types of PDUs not covered by that mechanisms that go each direction.

Fifth, it may have have a deadlock. If the command window has closed, the only way the target can send more credit is to send a NOP PDU. If the target doesn't have a credit, it won't be able to send the NOP PDU.

It isn't a good idea to complicate implementations excessively to avoid additions to wire protocol. If one adds a flow control mechanism, it should have provision to avoid deadlock.

Pat

-----Original Message-----
From: Caitlin Bestler [mailto:cait@asomi.com]
Sent: Wednesday, July 30, 2003 6:10 PM
To: Mallikarjun C.
Cc: ips@ece.cmu.edu; rddp@ietf.org
Subject: Re: iSCSI/iWARP drafts and flow control



On Wednesday, July 30, 2003, at 07:49 PM, Mallikarjun C. wrote:

>
>> There is an identical flow control issue for
>> RDMA Reads.
>
> Not true.  There's a built-in credit renewal in an RDMA Read.
> The Peer issuing the RDMA Read knows it can reuse the
> Read credit when it receives an RDMA Read Response.
> Send Messages carrying the "fringe" iSCSI PDUs need both
> new wire protocol and cross-layer chit-chat within an end-node
> between iSCSI and iSER - in order to renew credits.

No new wire protocol is required.

Restoration of no-CmdSNs credits could be piggy-backed on top
of the existing CmdSN credit system.

Basically each Cmd reply restores one "Cmd Credit". It could
also restore an implicit number of "NoCmd" credits. Basically
if Cmd X had been proceeded by Y "NoCmd" messages, then the
reply to Cmd X would restore Y "NoCmd" credits.

It is simple credit counting, suitable for implementation in
hardware or software at any number of layers, and requires
no additional wire messages or even fields.