RE: RE: Re: multiple connections

[Robert Snively <rsnively@Brocade.COM>]

Something is missing here.  A window based outbound flow control is
useless to a RAID.  The problem is that the data buffer structure is filled
from two sides.  Information is flowing outbound for write operations.
Information is flowing into the data buffer in the inbound direction
as a result of receiving and executing the small command packets.
As a result, the available buffering for outbound operations may be
zero at any time in a manner scheduled by the target, not the host.
In addition, during the maintenance of a RAID device's internal
redundancy, additional buffer space is used up that is unknown to
both the read and write operations.

The only ways I see to avoid a buffer resource deadlock are:

	1)  Send a transfer ready indication for each outbound transfer.  
		(This is the universal SCSI solution).

	2)  Make the rather ridiculous and unenforceable rule that 
		the sum of all the SCSI buffer areas
		for all initiators be less than the buffer space
		of each logical unit in a RAID.

Note that communication with each initiator about the available target
buffering may also not be sufficient, since any single initiator could
use it all up in an instant.

>  > >  
>  > >  2. WRITES - This is the really bad one in my opinion. For 
>  > >  me, avoiding
>  > >  RTTs in iSCSI would just by itself make iSCSI a superior 
>  > "transport"
>  > >  for SCSI. So assuming RTTs are not being used, the host 
>  > >  would (changing
>  > >  the posting order from READs), first post the write 
>  command to the
>  > >  connection on which the data is to be sent, and then 
>  post the write
>  > >  buffers to the connection which is to be used for sending data.
>  > >  One case is where for whatever reason, the target gets the 
>  > >  data before
>  > >  it gets the command, and has no clue what to do with the data.
>  > >  Let us assume that the target does get the command before 
>  > it gets the
>  > >  data. The target gets a command indicating that data being 
>  > >  written to 
>  > >  whatever lun and whatever location is going to arrive 
>  on some other
>  > >  connection. First the target has taken an extra event from 
>  > >  the adapter.
>  > >  Then, when data arrives on another NIC (and the target 
>  gets another
>  > >  event), the target goes through the list of outstanding WRITE
>  > >  commands to match the command with the data and then go 
>  about the
>  > >  business of processing the data.
>  > >  
>  > >  So in this case the work has increased for the 
>  initiator as well as
>  > >  the target.
>  > 
>  > 
>  > And to top it off, the target is active with a large number of
>  > initiators on behalf of a large number of logical units for a large
>  > number of queued commands, so there is absolutely no guarantee that
>  > any buffer exists for the data that was received, even if the
>  > command had been received first.
>  > 
>  > Every SCSI command execution is managed by the target for 
>  > this reason among
>  > many others.  Thus RTT has been a part of every SCSI protocol 
>  > for write
>  > operations.
>  > 
>  
>  TCP window size should provide a reasonable flow control mechanism,
>  so an additional flow control should not be needed. There will be
>  some statistical determination of the amount of memory needed in
>  a large array vs the window size extended on each connection.
>  
>  I don't know if arrays like to keep commands in seperate memory
>  from data, in which case a command queue depth may have to be 
>  communicated seperately (assuming most of window would typically
>  be used for data)
>  
>