RE: iSCSI: Out of order commands

["Somesh Gupta" <somesh_gupta@silverbacksystems.com>]

John,

> -----Original Message-----
> From: John Hufferd [mailto:hufferd@us.ibm.com]
> Sent: Thursday, November 08, 2001 5:19 PM
> To: somesh_gupta@silverbacksystems.com
> Cc: Robert D. Russell; Julian Satran; ips@ece.cmu.edu
> Subject: RE: iSCSI: Out of order commands
>
>
>
> I understand your point, however, their are several approaches that I and
> others have used to mitigate this issue.  We face that problem with Multi
> Processing all the time.
>
> One technique that I have seen use that is similar to what you describe is
> to allocate a set of buffers to each processor (in this case an HBA), and
> when they are used up, and only then, the go back to the Buffer Mother, to
> allocate another Buffer Set.  In this way the Processors (HBA) run mostly
> independent, but once in awhile get a new supply.  There are other
> techniques also, but that one clearly works wells and is very simple.

  I agree with the mechanism above. So here is where the problem is.
  Let us take a simplification of M adapters and 1 connection per
  adapter. Let us also assume that the N buffers are given to each
  adapter.

  Total number of buffers used - M * N
  Buffers available to each adapter - N

  How large of a command window should be advertised in the case of
  a session wide window.

  If the window is N, then we are under-utilizing the buffers but we
  will never drop commands.

  If we advertise M * N, and the initiator decides to send commands
  in an unbalanced manner (because 1 connection is working better),
  then we have a potential problem where the one adapter uses up
  its allocation - In the meantime the system will provide it some
  more (but not enough),
  and there is an imbalance leading to a situation where commands
  may have to be dropped. (this could also happen if the system
  did not allocate buffers evenly e.g. in a scenario where the
  adapters are not using buffers at the same rate).

  Now we can try to find a balance in the middle, but again there
  is a loss of efficiency on one axis or the other.

>
> Therefore, I do not see the Session Window as a problem in the protocol.
>
> .
> .
> .
> John L. Hufferd
> Senior Technical Staff Member (STSM)
> IBM/SSG San Jose Ca
> Main Office (408) 256-0403, Tie: 276-0403,  eFax: (408) 904-4688
> Home Office (408) 997-6136, Cell: (408) 499-9702
> Internet address: hufferd@us.ibm.com
>
>
> "Somesh Gupta" <somesh_gupta@silverbacksystems.com> on 11/08/2001 04:57:40
> PM
>
> Please respond to <somesh_gupta@silverbacksystems.com>
>
> To:   John Hufferd/San Jose/IBM@IBMUS
> cc:   "Robert D. Russell" <rdr@mars.iol.unh.edu>, Julian
>       Satran/Haifa/IBM@IBMIL, <ips@ece.cmu.edu>
> Subject:  RE: iSCSI: Out of order commands
>
>
>
> I think we may be talking about different things here.
> Yes, there are memories that are "on the iSCSI HBA"
> (in the case where everything is on a single board,
> this would be the memories that are controlled by
> only the iSCSI ASIC or whatever) - and these memories
> are not shared among multiple HBAs. These may be used
> for any book-keeping purposes, contexts, eddy buffering
> etc etc by the adapter.
>
> Then there is memory pool that is owned by the "system".
> Adapters will DMA commands/data/responses directly into
> and out of this memory pool which is managed by the "system".
> However to prevent adapters from running over each other,
> and over the "system" processor, producer/consumer
> mechanisms have to be implemented.
>
> This is necessary to maintain order on the system. So
> on a long-term basis, the memory in the system is shared
> across the HBAs - absolutely yes. But at any given
> instance in time, it is partitioned among various
> usages.
>
> So all in all I agree with you. The only point of difference
> is that it is not possible that a given buffer (part of
> a buffer pool) is accessible to multiple adapters at
> the same instance in time. It is a necessary condition to
> prevent corruption and chaos.
>
> > -----Original Message-----
> > From: John Hufferd [mailto:hufferd@us.ibm.com]
> > Sent: Thursday, November 08, 2001 4:40 PM
> > To: somesh_gupta@silverbacksystems.com
> > Cc: Robert D. Russell; Julian Satran; ips@ece.cmu.edu
> > Subject: RE: iSCSI: Out of order commands
> >
> >
> >
> > I guess I disagree with this design approach.  The buffers on the iSCSI
> > HBAs are for Eddy buffers.  Once you have the iSCSI Header
> should be able
> > to place the data directly into a Shared buffer.  If you have a session
> > that spans HBAs, that should work just fine.  But the buffers match the
> > Session Window, and the Buffers need to be shared across the
> HBAs, not in
> > the HBAs.  Again, the HBAs should just have the Eddy Buffers.
> >
> > .
> > .
> > .
> > John L. Hufferd
> > Senior Technical Staff Member (STSM)
> > IBM/SSG San Jose Ca
> > Main Office (408) 256-0403, Tie: 276-0403,  eFax: (408) 904-4688
> > Home Office (408) 997-6136, Cell: (408) 499-9702
> > Internet address: hufferd@us.ibm.com
> >
> >
> > "Somesh Gupta" <somesh_gupta@silverbacksystems.com> on 11/08/2001
> 04:31:29
> > PM
> >
> > Please respond to <somesh_gupta@silverbacksystems.com>
> >
> > To:   John Hufferd/San Jose/IBM@IBMUS
> > cc:   "Robert D. Russell" <rdr@mars.iol.unh.edu>, Julian
> >       Satran/Haifa/IBM@IBMIL, <ips@ece.cmu.edu>
> > Subject:  RE: iSCSI: Out of order commands
> >
> >
> >
> > John,
> >
> > You are correct if all the connections are on
> > a single adapter. In this case the buffer pool
> > can be shared among all the connections on the
> > adapter.
> >
> > However this is not true if the connections are
> > on different adapters (the point being discussed).
> > Think of there being a big buffer pool in the board
> > into which multiple adapters/chips are "plugged" in.
> > However, at any given point in time, these buffers
> > are allocated (handed over through command queues or
> > whatever) to each of the individual adapters. And this
> > is where the problem arises.
> >
> > The assumption that adapters can share a typical host
> > command buffer pool on their own without host intervention
> > is incorrect.
> >
> > Somesh
> >
> > > -----Original Message-----
> > > From: John Hufferd [mailto:hufferd@us.ibm.com]
> > > Sent: Thursday, November 08, 2001 4:20 PM
> > > To: somesh_gupta@silverbacksystems.com
> > > Cc: Robert D. Russell; Julian Satran; ips@ece.cmu.edu
> > > Subject: RE: iSCSI: Out of order commands
> > >
> > >
> > >
> > > Most folks I know have a buffer that applies to all the
> > > connections, not to
> > > just one buffer per connection.  I fail to see the problem for most
> > > implimentations.  The window should apply to the session.
> > >
> > > .
> > > .
> > > .
> > > John L. Hufferd
> > > Senior Technical Staff Member (STSM)
> > > IBM/SSG San Jose Ca
> > > Main Office (408) 256-0403, Tie: 276-0403,  eFax: (408) 904-4688
> > > Home Office (408) 997-6136, Cell: (408) 499-9702
> > > Internet address: hufferd@us.ibm.com
> > >
> > >
> > > "Somesh Gupta" <somesh_gupta@silverbacksystems.com>@ece.cmu.edu on
> > > 11/08/2001 02:22:30 PM
> > >
> > > Please respond to <somesh_gupta@silverbacksystems.com>
> > >
> > > Sent by:  owner-ips@ece.cmu.edu
> > >
> > >
> > > To:   "Robert D. Russell" <rdr@mars.iol.unh.edu>
> > > cc:   Julian Satran/Haifa/IBM@IBMIL, <ips@ece.cmu.edu>
> > > Subject:  RE: iSCSI: Out of order commands
> > >
> > >
> > >
> > > Comments below
> > >
> > > > -----Original Message-----
> > > > From: Robert D. Russell [mailto:rdr@mars.iol.unh.edu]
> > > > Sent: Thursday, November 08, 2001 11:46 AM
> > > > To: Somesh Gupta
> > > > Cc: Julian Satran; ips@ece.cmu.edu
> > > > Subject: RE: iSCSI: Out of order commands
> > > >
> > > >
> > > >
> > > > Somesh:
> > > >
> > > > > > It seems to me that if a target can only handle x new commands,
> > > > > > then its queueing capacity is x, and in the SCSI Response PDU it
> > > > > > should set MaxCmdSN = (ExpCmdSn + x - 1), in accordance with the
> > > > > > formula in section 2.2.2.1.  This in turn controls the number of
> > > > > > commands the initiator can send, and thus prevents the incoming
> > > > > > commands from overfilling the target's queue.
> > > > >
> > > > >   This is actually a weakness of the multiple connections over
> > > > >   multiple adapters model (it is not related to ordering).
> > > > >   The target advertises a command window on a session wide basis.
> > > > >   At the same time, it has to provide the resource to the adapter
> > > > >   to be able to DMA those commands to. Since there is no
> restriction
> > > > >   on which connection the commands may be received, either the
> > > > >   target has to allocate the max resources needed to each adapter
> > > > >   (thus committing n times the resources actually required), or
> > > > >   has to "lie" (it would not be a complete lie) which could
> > > > >   result in running out of resources. One way to fix that would be
> > > > >   to have the credit on a per connection basis.
> > > >
> > > > If I understand you correctly, you are saying that deadlock can
> > > > occur even if we enforce ordered delivery by initiators and even
> > > > if the advertisements are correct and even if both parties obey
> > > > the advertisements.
> > >
> > >   I would let Julian enlighten us on that, or whether it just
> > >   leads to wholesale command drops and retries, or TASK SET FULL
> > >   is returned (is TASK SET FULL a valid response in this case
> > >   for a LINKED task when it is not the first command?)
> > >
> > > >
> > > > In other words, doesn't this mean that the standard can lead to a
> > > > trap in implementing targets and that, in fact, the advertisements
> > > > really should be on a per connection basis?
> > > > However, what would be the consequences for initiators if
> > > > the advertisements were on a per-connection basis?
> > >
> > >   There is really no issue with adveritisement on a per
> > >   connection basis. It leads to a protocol change. However, it
> > >   also allows a much more performant flow of commands.
> > >
> > > >
> > > >
> > > > Thanks,
> > > >
> > > > Bob Russell
> > > > InterOperability Lab
> > > > University of New Hampshire
> > > > rdr@iol.unh.edu
> > > > 603-862-3774
> > > >
> > > >
> > > >
> > >
> > >
> > >
> > >
> >
> >
> >
> >
>
>
>
>
>