RE: iSCSI: Out of order commands

["John Hufferd" <hufferd@us.ibm.com>]

I understand your point, however, their are several approaches that I and
others have used to mitigate this issue.  We face that problem with Multi
Processing all the time.

One technique that I have seen use that is similar to what you describe is
to allocate a set of buffers to each processor (in this case an HBA), and
when they are used up, and only then, the go back to the Buffer Mother, to
allocate another Buffer Set.  In this way the Processors (HBA) run mostly
independent, but once in awhile get a new supply.  There are other
techniques also, but that one clearly works wells and is very simple.

Therefore, I do not see the Session Window as a problem in the protocol.

.
.
.
John L. Hufferd
Senior Technical Staff Member (STSM)
IBM/SSG San Jose Ca
Main Office (408) 256-0403, Tie: 276-0403,  eFax: (408) 904-4688
Home Office (408) 997-6136, Cell: (408) 499-9702
Internet address: hufferd@us.ibm.com


"Somesh Gupta" <somesh_gupta@silverbacksystems.com> on 11/08/2001 04:57:40
PM

Please respond to <somesh_gupta@silverbacksystems.com>

To:   John Hufferd/San Jose/IBM@IBMUS
cc:   "Robert D. Russell" <rdr@mars.iol.unh.edu>, Julian
      Satran/Haifa/IBM@IBMIL, <ips@ece.cmu.edu>
Subject:  RE: iSCSI: Out of order commands



I think we may be talking about different things here.
Yes, there are memories that are "on the iSCSI HBA"
(in the case where everything is on a single board,
this would be the memories that are controlled by
only the iSCSI ASIC or whatever) - and these memories
are not shared among multiple HBAs. These may be used
for any book-keeping purposes, contexts, eddy buffering
etc etc by the adapter.

Then there is memory pool that is owned by the "system".
Adapters will DMA commands/data/responses directly into
and out of this memory pool which is managed by the "system".
However to prevent adapters from running over each other,
and over the "system" processor, producer/consumer
mechanisms have to be implemented.

This is necessary to maintain order on the system. So
on a long-term basis, the memory in the system is shared
across the HBAs - absolutely yes. But at any given
instance in time, it is partitioned among various
usages.

So all in all I agree with you. The only point of difference
is that it is not possible that a given buffer (part of
a buffer pool) is accessible to multiple adapters at
the same instance in time. It is a necessary condition to
prevent corruption and chaos.

> -----Original Message-----
> From: John Hufferd [mailto:hufferd@us.ibm.com]
> Sent: Thursday, November 08, 2001 4:40 PM
> To: somesh_gupta@silverbacksystems.com
> Cc: Robert D. Russell; Julian Satran; ips@ece.cmu.edu
> Subject: RE: iSCSI: Out of order commands
>
>
>
> I guess I disagree with this design approach.  The buffers on the iSCSI
> HBAs are for Eddy buffers.  Once you have the iSCSI Header should be able
> to place the data directly into a Shared buffer.  If you have a session
> that spans HBAs, that should work just fine.  But the buffers match the
> Session Window, and the Buffers need to be shared across the HBAs, not in
> the HBAs.  Again, the HBAs should just have the Eddy Buffers.
>
> .
> .
> .
> John L. Hufferd
> Senior Technical Staff Member (STSM)
> IBM/SSG San Jose Ca
> Main Office (408) 256-0403, Tie: 276-0403,  eFax: (408) 904-4688
> Home Office (408) 997-6136, Cell: (408) 499-9702
> Internet address: hufferd@us.ibm.com
>
>
> "Somesh Gupta" <somesh_gupta@silverbacksystems.com> on 11/08/2001
04:31:29
> PM
>
> Please respond to <somesh_gupta@silverbacksystems.com>
>
> To:   John Hufferd/San Jose/IBM@IBMUS
> cc:   "Robert D. Russell" <rdr@mars.iol.unh.edu>, Julian
>       Satran/Haifa/IBM@IBMIL, <ips@ece.cmu.edu>
> Subject:  RE: iSCSI: Out of order commands
>
>
>
> John,
>
> You are correct if all the connections are on
> a single adapter. In this case the buffer pool
> can be shared among all the connections on the
> adapter.
>
> However this is not true if the connections are
> on different adapters (the point being discussed).
> Think of there being a big buffer pool in the board
> into which multiple adapters/chips are "plugged" in.
> However, at any given point in time, these buffers
> are allocated (handed over through command queues or
> whatever) to each of the individual adapters. And this
> is where the problem arises.
>
> The assumption that adapters can share a typical host
> command buffer pool on their own without host intervention
> is incorrect.
>
> Somesh
>
> > -----Original Message-----
> > From: John Hufferd [mailto:hufferd@us.ibm.com]
> > Sent: Thursday, November 08, 2001 4:20 PM
> > To: somesh_gupta@silverbacksystems.com
> > Cc: Robert D. Russell; Julian Satran; ips@ece.cmu.edu
> > Subject: RE: iSCSI: Out of order commands
> >
> >
> >
> > Most folks I know have a buffer that applies to all the
> > connections, not to
> > just one buffer per connection.  I fail to see the problem for most
> > implimentations.  The window should apply to the session.
> >
> > .
> > .
> > .
> > John L. Hufferd
> > Senior Technical Staff Member (STSM)
> > IBM/SSG San Jose Ca
> > Main Office (408) 256-0403, Tie: 276-0403,  eFax: (408) 904-4688
> > Home Office (408) 997-6136, Cell: (408) 499-9702
> > Internet address: hufferd@us.ibm.com
> >
> >
> > "Somesh Gupta" <somesh_gupta@silverbacksystems.com>@ece.cmu.edu on
> > 11/08/2001 02:22:30 PM
> >
> > Please respond to <somesh_gupta@silverbacksystems.com>
> >
> > Sent by:  owner-ips@ece.cmu.edu
> >
> >
> > To:   "Robert D. Russell" <rdr@mars.iol.unh.edu>
> > cc:   Julian Satran/Haifa/IBM@IBMIL, <ips@ece.cmu.edu>
> > Subject:  RE: iSCSI: Out of order commands
> >
> >
> >
> > Comments below
> >
> > > -----Original Message-----
> > > From: Robert D. Russell [mailto:rdr@mars.iol.unh.edu]
> > > Sent: Thursday, November 08, 2001 11:46 AM
> > > To: Somesh Gupta
> > > Cc: Julian Satran; ips@ece.cmu.edu
> > > Subject: RE: iSCSI: Out of order commands
> > >
> > >
> > >
> > > Somesh:
> > >
> > > > > It seems to me that if a target can only handle x new commands,
> > > > > then its queueing capacity is x, and in the SCSI Response PDU it
> > > > > should set MaxCmdSN = (ExpCmdSn + x - 1), in accordance with the
> > > > > formula in section 2.2.2.1.  This in turn controls the number of
> > > > > commands the initiator can send, and thus prevents the incoming
> > > > > commands from overfilling the target's queue.
> > > >
> > > >   This is actually a weakness of the multiple connections over
> > > >   multiple adapters model (it is not related to ordering).
> > > >   The target advertises a command window on a session wide basis.
> > > >   At the same time, it has to provide the resource to the adapter
> > > >   to be able to DMA those commands to. Since there is no
restriction
> > > >   on which connection the commands may be received, either the
> > > >   target has to allocate the max resources needed to each adapter
> > > >   (thus committing n times the resources actually required), or
> > > >   has to "lie" (it would not be a complete lie) which could
> > > >   result in running out of resources. One way to fix that would be
> > > >   to have the credit on a per connection basis.
> > >
> > > If I understand you correctly, you are saying that deadlock can
> > > occur even if we enforce ordered delivery by initiators and even
> > > if the advertisements are correct and even if both parties obey
> > > the advertisements.
> >
> >   I would let Julian enlighten us on that, or whether it just
> >   leads to wholesale command drops and retries, or TASK SET FULL
> >   is returned (is TASK SET FULL a valid response in this case
> >   for a LINKED task when it is not the first command?)
> >
> > >
> > > In other words, doesn't this mean that the standard can lead to a
> > > trap in implementing targets and that, in fact, the advertisements
> > > really should be on a per connection basis?
> > > However, what would be the consequences for initiators if
> > > the advertisements were on a per-connection basis?
> >
> >   There is really no issue with adveritisement on a per
> >   connection basis. It leads to a protocol change. However, it
> >   also allows a much more performant flow of commands.
> >
> > >
> > >
> > > Thanks,
> > >
> > > Bob Russell
> > > InterOperability Lab
> > > University of New Hampshire
> > > rdr@iol.unh.edu
> > > 603-862-3774
> > >
> > >
> > >
> >
> >
> >
> >
>
>
>
>