Re: iSCSI: Flow Control

[Pierre Labat <pierre_labat@hp.com>]

julian_satran@il.ibm.com wrote:

Julo,

Some answers to yours comments.

Regards,

Pierre

> Pierre,
>
> a) This group never agreed on such a thing. There is a rough agreement
> that the amount should be limited. And again keep in mind that we talking
> about two types of unsolicited data
> - immediate an no-immediate. The only thing we sort of agreed is bad is to
> allow immediate data when all data are required to be solicited by the
> appropriate configuration
> bit.
>
> b) flow control on data is there with both SCSI and TCP taking care of it.
>
> c) it was repeatedly stated that you can't avoid task-set-full as this is
> an individual LU issue.

I disagree, if you have a right buffer management policy on the target
you can avoid it.
If you allocated command block buffers per LU for example 100 per LU,
you are right, you will hit task-set-full:
- the LU1 has its task set full  but the LU2 accessed through the same
   same session has its task set almost empty.
- you don't want to flow control commands for the LU2, so you don't
   close the command window of the session and then you will
   hit task-set-full on the LU1.


But SAM-2 doesn't require you manage the target memory resources
(in this case the command block buffers) in this way.

Let me try to convince you, explaining with some details how could
be such target policies.

The target can manage  a pool of command block buffers per session.
It has a policy to manage the command blocks inside the session
and it has another policy to move unused command blocks from
one session pool to another.
Inside the session the policy is not: same amount of command blocks
per LU. If it is that, you will hit the task-set-full condition to
avoid the underrun of LUs in the session.
The policy could be: each time a new command comes, picks
a command block in the session pool. When you reach a low
threshold in the pool,
you could trigger the inter session policy to try to grab command
blocks from an other session underused. If it fails then you start
flow controlling the commands on the session and avoid a
task-set-full. At the same time you do NOT penalize a LU
with this flow control. The target did the best it could
to satisfy the initiator(s) in the limit of its resources.



> If you are suggesting flow-control per LU and
> per initiator I guess everybody in this space will tell you that this is
> expensive and barely useful.
>
> d) I agree that would be a desiderata but perhaps you can hand as the key
> piece missing - how to do it!

Let me answer to your objections to a) and d) here.
First d)
Let's assume we have a iSCSI adapter of the type SCSI/FC on which
the server posts commands and is only informed of the completion.
Every thing else is managed by the adapter.
In fact this adapter works as a pull machine.
It contemplates a bunch of commands posted (with pointers
on the data buffers associated) and decides what to pull
from host memory after each iSCSI PDU sent on the wire.
Here you can add cleverness in the adapter to help us:
1  after each PDU is sent the adapter enter a loop:

2  for each command posted, encapsulate and
   send the command (in order). Do that for all the commands
   posted to it and not yet sent (stop if the command window is closed)
3 when the commands requests are exhausted
   the target sends data till a new command is posted. At
   this point it restarts in 1

Doing that allows the commands to pass the data.
However the order between the commands is preserved.

Now if you use another kind of adapter (for example
a regular LAN adapter and if you run TCP/IP
on the host, it will not work.
However, the main market will be using adapters
"a la" SCSI, no?

About the objection in a), has commands can pass data
i don't see a problem (as far as unsolicited data has
been authorized (no R2T required)). You can post
a huge WRITE to the adapter let say (1MBytes).
The adapter will slice it in iSCSI PDUs and will be
able to interleave other commands.

It is why it seems to me that the model we have in the draft
combined with adapters as described above is a very good
model in term of performance, even with
just one TCP connection.



>
>
> Julo
>
> Pierre Labat <pierre_labat@hp.com> on 06/10/2000 19:00:08
>
> Please respond to Pierre Labat <pierre_labat@hp.com>
>
> To:   IPS@ece.cmu.edu
> cc:    (bcc: Julian Satran/Haifa/IBM)
> Subject:  RE: iSCSI: Flow Control
>
> Hi,
>
> >From the mails on this thread it seems that most
> people agree on the following ?
>
> A) Unsolicited data will be allowed by iSCSI.
> Well, i think that the protocol must be able to handle
> large quantities of unsolicited data to be performant
> for the writes on large latency networks.
>
> B) We need of course a flow control on the data
> because even if we have a flow control on the commands,
> just a few commands with huge unsolicited data could
> overwhelm the target buffers.
>
> C) We need a flow control on the commands to avoid
> the TASK SET FULL condition. If the target hit this condition
> it has to drop command (return task set full) and
> drops data associated. Avoiding the task set full condition
> avoid too the "dead locks".
>
> D) As noted by several, one TCP connection doesn't "block"
> the commands behind the data. The iSCSI adapters (on the
> initiator side) must be build in a way to send first the
> commands/task management requests. As soon as a command/task mgt
> is posted to the adapter, the next iSCSI PDU sent by
> the adapter will be the one encapsulating the command/task mgt
> request. This has to be true for any command (read/write...)
> and task mgt request.
>