Re: iSCSI/iWARP drafts and flow control

[Caitlin Bestler <cait@asomi.com>]

On Thursday, July 31, 2003, at 12:30 PM, pat_thaler@agilent.com wrote:

> Caitlin,
>
> There are multiple problems with the mechanism you suggest.
>
> For one thing, the credit for additional command window in iSCSI isn't 
> based on command replys. There is an explicit field for MaxCmdSN. 
> Having received and finished processing a command doesn't mean that 
> one has made space for receiving another command. The target might 
> choose to use that space for another session or for something else 
> altogether.
>

There is a restocking mechanism in place. I oversimplified in my 
description of it. The point is that
the ULP already knows how many CmdSNs that it is allowed to send. It 
knows when that count is depleted,
and when it is restored.

> Secondly, having finished a processing a command doesn't mean that 
> other activities have completed. If we did what you suggest there 
> might be times when a response had to be delayed because the command 
> had been finished but the credits for the non-command messages were 
> not yet ready. This could adversely effect performance. Also, commands 
> can involve lengthy actions such as large data transfers. Therefore, 
> there might be times when one is not ready to send a command reply and 
> immediate commands or other non-command related traffic are delayed 
> because of lack of credit.
>
There is no requirement that the asynch processing have completed, just 
that the buffers have been restocked.
If the number of asynch commands is light, as claimed, crediting enough 
to cover the delay of a long command
should not be a problem. The goal is to grant credits, nobody is 
requiring buffers be pre-committed for the
entire lifespan of the credit.

> Third, it is an accounting nightmare for the implementations. Instead 
> of just tracking a simple credit counter and updating it, they have to 
> keep track of how many credits belong to each command.
>
A nightmare? At worst it is a second count.

One implementation could be as simple as setting a flag that will be 
cleared after N cmds.

> Fourth, the suggestion only takes care of one direction. MaxCmdSN goes 
> from target to initiator, but there are types of PDUs not covered by 
> that mechanisms that go each direction.
>
Then how are they being flow controlled?

> Fifth, it may have have a deadlock. If the command window has closed, 
> the only way the target can send more credit is to send a NOP PDU. If 
> the target doesn't have a credit, it won't be able to send the NOP > PDU.

How is this prevented with iSCSI over TCP?

What iWARP is requiring is that the ULP provide flow control for 
untagged buffers,
to replace the flow control that TCP/SCTP provided for transport 
buffers.

This is a *benefit*, it prevents head-of-line blocking of tagged 
transfers. But
in any situation, if the ULP could have a deadlock situation where 
there are
no untagged buffers available at either end then TCP/SCTP could have 
had a
deadlock on transport buffering.

>
> It isn't a good idea to complicate implementations excessively to 
> avoid additions to wire protocol. If one adds a flow control 
> mechanism, it should have provision to avoid deadlock.
>
> Pat

Meanwhile you have not addressed any of the critical problems with the 
lack of
ULP flow control.

Without flow control the sender must *guess* at how many messages can 
be sent.
If they guess wrong the stream is terminated. Stream termination is NOT
"flow control", it is either a *sanction* for failing to conform to the
ULP, or it is the result of a *fault* on the receiving side.

There is, by definition, no reliable mechanism to "guess" how many 
messages
may be sent that does not produce stream terminations. That is 
unacceptable
in something that is supposed to be a reliable protocol.

If you don't want the semantics of a reliable protocol then propose an
adaptation that runs over UDP. Otherwise, you cannot have a reliable
protocol without *some* form of flow control on buffer usage. You cannot
require the Data Sink to provide an *indefinite* amount of buffering.

The arguments you cite seem to prove that iSER cannot be implemented 
over
a standard RNIC. I believe it can be, but that requires documenting the
rules that will *regulate* transmission of *all* untagged messages.

I believe this can be done in a way that meets iSCSI's requirements 
without
violating iWARP's rules. If iSCSI works over TCP, where its messages 
can be
flow controlled by the TCP window, then it can work over a definition of
iSER that complies with iWARPs rules and the definition "reliable 
transport".