Re: iSCSI: Flow Control

[Pierre Labat <pierre_labat@hp.com>]

John Hufferd/San Jose/IBM wrote:

> Pierre,
> I am not sure that I agree.  I do not understand how a credit process to an
> initiator, solves the problem (if it is a problem)

Hoops! i didn't remembered that ExpCmdRN was just an acknowledgement
of the receipt of the command (not the ack of the completion)...
Hence forget about the solution 2). The problem of slow commands
blocking fast one doesn't exist.
The solution in the current draft is perfect for me.

Regards,

Pierre

> with 100s of initiators
> some of which can be dormant for long periods of time and very active at
> others.  I believe that the current command window meets all the real
> problems.  You did put your finger on the key solution, that is the
> management of the Target Buffers, by the target.
>
> .
> .
> .
> John L. Hufferd
>
> Pierre Labat <pierre_labat@hp.com>@ece.cmu.edu on 10/06/2000 09:00:08 AM
>
> Sent by:  owner-ips@ece.cmu.edu
>
> To:   IPS@ece.cmu.edu
> cc:
> Subject:  RE: iSCSI: Flow Control
>
> Hi,
>
> >From the mails on this thread it seems that most
> people agree on the following ?
>
> A) Unsolicited data will be allowed by iSCSI.
> Well, i think that the protocol must be able to handle
> large quantities of unsolicited data to be performant
> for the writes on large latency networks.
>
> B) We need of course a flow control on the data
> because even if we have a flow control on the commands,
> just a few commands with huge unsolicited data could
> overwhelm the target buffers.
>
> C) We need a flow control on the commands to avoid
> the TASK SET FULL condition. If the target hit this condition
> it has to drop command (return task set full) and
> drops data associated. Avoiding the task set full condition
> avoid too the "dead locks".
>
> D) As noted by several, one TCP connection doesn't "block"
> the commands behind the data. The iSCSI adapters (on the
> initiator side) must be build in a way to send first the
> commands/task management requests. As soon as a command/task mgt
> is posted to the adapter, the next iSCSI PDU sent by
> the adapter will be the one encapsulating the command/task mgt
> request. This has to be true for any command (read/write...)
> and task mgt request.
>
> About managing these flow controls:
> ==================================
> The TCP window can handle efficiently the flow control for the data.
> Now we need to flow control the commands. The solutions
> can be:
>
> 1) Use the window described in the draft
>
> Advantage:
> ---------
>  o Flow control the commands
>
> Disavantages:
> -------------
>  o Some said that as the flow control is per session,
>    the target will flow control the whole session
>    (several LUs) when the task set of one LU will
>    become [almost] full. Hence one LU blocks the other
>    ones in the session when they could continue to work.
>
>    This problem can be workaround by the
>    target using a right command blocks allocation policy.
>    (A command block is a buffer containing the command
>     pending (in the task set) in the target).
>
>    Instead of having a policy where the command  blocks
>    are allocated per LU, the target could use a policy
>    where the command blocks are allocated per session.
>    The target maintains a pool of blocks per session not per
>    LU. In this case one LU cannot block another one.
>    That means that inside a session the balance of the commands
>    in the various task sets is driven by the initiator not
>    by the target.
>    It doesn't seems to be wrong doing that, hence this disavantage
>    is NOT a disavantage.
>
>  o If the initiator send a mix of fast and slow commands the slow
>    commands can close the command window even if the target have
>    slots free for new commands.
>
> 2) Replace the window in the iSCSI draft with a command credit.
>    The target advertises (with each response) a credit to the initiator.
>    The command reference number and status reference numbers
>    can be kept for ordering/recovery purpose. Now, the MaxCMdRN
>    becomes a credit unrelated to the CmdRn, that's the only change compared
>    to the draft. It specifies the maximum number of commands in flight over
>    the session.
>
> Advantage
> ---------
>  o flow control commands
>  o get rid of the blocking of fast commands by slow commands.
>
> Disavantage
> -----------
>  o None
>
> 3) Use a (iSCSI) credit per LUN (advertised with the command responses)
>
> Advantage
> ---------
>  o flow control the commands (per LUN)
>
> Disavantage
> -----------
>  o the iSCSI layer on the initiator needs to maintain a state and a flow
> control
>    per LUN
>
> 4) T10 specifies a command credit advertising mechanism (credit returned
>    with the status + some asynchronous advertisements (see Mike mail/IBA))
>
> Advantage
> ---------
>  o flow control the commands (per LUN)
>  o the state of the flow control would be maintained at the SCSI layer,
>    it's ok because the SCSI layer already mantains some states/LUN
>  o will benefit to FC too
>
> Disavantage
> -----------
>  o are we able to persuade T10 to specify that quickly?
>  o need to change the SCSI layer
>
> Conclusion
> ==========
> It seems to me that with 2) and even with only one TCP connection,
> we have a solution very performant in term of throughput, simple,
> and with no dead locks.
> Do you agree?
>
> Regards,
>
> Pierre