RE: iSCSI : iSCSI parameter default values

["John Hufferd" <hufferd@us.ibm.com>]

Bob,
I really do understand your statements about the cache.  I just think that
they are over stated.

Ed,
First off let me also say to, that you points ( on another note regarding
Caches) are right on.  Especially your points with regards to having to do
something to match the new capabilities of iSCSI with old Cache Manager
designs. It is possible that not all are flexible enough to make a easy
adaption.  (But some/many are.)

Bob,
The Data parts of on immediate data are clearly identifiable in the PDU.
It is very reasonable to fork the input as it is coming in.  That is, put
the data into NVRAM cache memory as the data comes in, and place the CDB
etc, wherever you want it.  At that time, all the associations can be
marked as well.

I over stated the code path identity between Login handling and Write I/O
immediate Data.  But the receive code path is very similar, it is just that
the Data probably will not go to the NVRAM cache.  Instead it would be
forked into a Text parsing work area.  But the code is very close to the
same.  And my point is, to worry about the few line of code difference is
missing the key point and the value of Immediate Data.

.
.
.
John L. Hufferd
Senior Technical Staff Member (STSM)
IBM/SSG San Jose Ca
Main Office (408) 256-0403, Tie: 276-0403,  eFax: (408) 904-4688
Home Office (408) 997-6136
Internet address: hufferd@us.ibm.com


Robert Snively <rsnively@brocade.com> on 10/02/2001 09:40:01 AM

To:   John Hufferd/San Jose/IBM@IBMUS, Robert Snively
      <rsnively@brocade.com>
cc:   ips@ece.cmu.edu
Subject:  RE: iSCSI : iSCSI parameter default values



John,

What I really wanted to point out is that the destination
of the data is NOT the buffer, nor is it a memory space
behind the buffer.  It is a non-volatile protected storage medium,
which may be either a set of disk drives with the proper degrees
of RAID redundancy or (at least temporarily) some kind of
replicated non-volatile RAM.

It is that protected space which must be made available for received data.
If you tack the data into the command PDU, you must treat the
whole combination.  You may treat it as protected, or you may treat
it as recoverable, depending on how complex you want your PDU
retransmission algorithm to be, but in either case you will have
to post-process it to separate the command, interpret the command,
and move the data portion to the proper resting place, where it
can be available for subsequent accesses by other commands.

If you leave it separate, you can treat the command as a unit
which does not have to be separated out or analyzed until you get
around to it, placing the data in a pre-allocated portion of
the non-volatile storage for later tagging and referencing when
the command is interpreted.

Better yet, if you use R2T, you can request the data at the
proper time to have it delivered (in any convenient order of
course) directly to the proper destination disk drive or to a pre-tagged
portion of non-volatile storage.  It is this type of operation that
is truly "zero-copy" in the SCSI and storage sense.

Bob Snively                        e-mail:    rsnively@brocade.com
Brocade Communications Systems     phone:  408 487 8135
1745 Technology Drive
San Jose, CA 95110



> -----Original Message-----
> From: John Hufferd [mailto:hufferd@us.ibm.com]
> Sent: Sunday, September 30, 2001 2:29 AM
> To: Robert Snively
> Cc: ips@ece.cmu.edu
> Subject: RE: iscsi : iscsi parameter default values
>
>
>
> Robert,
> You only have to perform moves, if you have not coded your
> buffer manager
> to manage Chained Segments.  Since this is a very old and highly used
> technique in buffer management, especially involving
> Telecommunications, I
> am surprised that you have not considered it.  Perhaps I am missing
> something in what you are trying to say but I have used it
> often to avoid
> moving data and obtaining zero-copy.  It is also works well
> with devices
> that support Scatter Gather, which is the usual case with Disk.
>
> Immediate data is simple, perhaps the simplest of all our functions to
> code.  You are correct, that it means that there may have to
> be reasonably
> sized buffers.  This argument was valid years ago, but it is
> now just too
> hard not to have large buffers in a storage controller (in
> fact now small
> cost more).
>
> Yes, I know there are limits to what I said above, but that is why the
> ImmediateData=No can be used.  But your focus and arguments are old
> arguments, for devices that thought memory was too expensive
> to use a great
> deal of it.
>
> Example: since most uses of ImmediateData will probably
> average less then
> 4k (example Database pages),  four thousand connections means
> 16 Meg, I do
> not think you can even get a decent price on memories that
> small.  If you
> have a device that supports 4 thousand host connections, I
> suggest that 16
> meg is not an issue.  Why would we even discuss it?
>
>
> .
> .
> .
> John L. Hufferd
> Senior Technical Staff Member (STSM)
> IBM/SSG San Jose Ca
> Main Office (408) 256-0403, Tie: 276-0403,  eFax: (408) 904-4688
> Home Office (408) 997-6136
> Internet address: hufferd@us.ibm.com
>
>
> Robert Snively <rsnively@Brocade.COM>@ece.cmu.edu on
> 09/28/2001 04:31:12 PM
>
> Sent by:  owner-ips@ece.cmu.edu
>
>
> To:   Julian Satran/Haifa/IBM@IBMIL, ips@ece.cmu.edu
> cc:
> Subject:  RE: iscsi : iscsi parameter default values
>
>
>
> Julian,
>
> For SCSI Write operations, ImmediateData = yes is
> the most demanding mode for the target.  If I understand
> it correctly, it essentially over-rides the R2T state,
> allowing the first burst to be included as immediate data.
> In SCSI, except for special cases like this, the target
> is the device in charge of data transfers.
>
> This mode requires the target to have a guaranteed collection of
> received buffer space adequate to receive all possible
> outbound queued operations from all possible initiators
> through all possible target sessions (which may sum to
> 1000s of output operations), regardless of what other
> buffer-intensive operations may be going on in the device.
>
> It then requires the device to provide association with each
> of those commands instead of just the commands it has elected
> to activate.  Without immediate data, the command buffer can be
> separated and separately managed from the data buffer,
> limiting buffer requirements.
>
> It then requires the device to operate in a non-zero-copy
> mode (which raises buffer utilization and increases latency)
> while the device analyzes the command to determine what should
> be done with the data.  It further requires the subsequent
> data (if there is more than one PDU to be transmitted) to
> be separately managed, and perhaps actually stored in a
> separate operation if the buffer must be cleared to make
> space available for it.  It further requires the target to
> analyze the data for completeness before going on to determine
> what to do with it.
>
> Sure, it is easy for the initiator, but so is everything else
> except out-of-order reads.
> It is the target you are stressing with this.
>
> For local sub-LANs, and depending on the actual buffering
> available in the target, the performance may actually be lower with
> ImmediateData allowed, because zero copy behavior of the
> target to non-volatile media is not available, which raises
> buffer utilization.
>
> Have I missed something that would change my mind?
>
> Bob
>
>
> > -----Original Message-----
> > From: Julian Satran [mailto:Julian_Satran@il.ibm.com]
> > Sent: Friday, September 28, 2001 10:38 AM
> > To: ips@ece.cmu.edu
> > Subject: RE: iscsi : iscsi parameter default values
> >
> >
> >
> >
> > Robert,
> >
> > I disagree that Immediate is "the most demanding" mode.  That
> > is not my
> > experience and not what I heard from most of the developers.  On the
> > contrary immediate data do not require a separate indexing
> > operation on the
> > target (as non-immediate unsolicited do) and that was the
> base for the
> > consensus to have them negotiated separately.
> >
> > For the software initiator it is the most inexpensive way to
> > perform short
> > data transfers (4-8k) typical for major applications like
> > Database access
> > and so it is for lightweight target.
> >
> > Julo
> >
> >
> >
> >
> >
> >                     Robert Snively
> >
> >                     <rsnively@broc       To:     John
> > Hufferd/San Jose/IBM@IBMUS, Julian
> >                     ade.com>
> > Satran/Haifa/IBM@IBMIL
> >                                          cc:
> > ips@ece.cmu.edu
> >                     28-09-01 17:25       Subject:     RE:
> > iscsi : iscsi parameter default values
> >                     Please respond
> >
> >                     to Robert
> >
> >                     Snively
> >
> >
> >
> >
> >
> >
> >
> >
> > I vote no as the default value on ImmediateData.
> >
> > A default value of yes on ImmediateData requires implementation
> > of the most complex and demanding mode of operation as the
> > default.  SCSI has traditionally made its default behavior the
> > simplest and most encompassing, setting more sophisticated
> > behavior by subsequent agreement.  While it may be your earnest
> > desire to encourage the implementation of this function, it
> > is not appropriate to demand that as the default behavior
> > of all devices.  In particular, there is no special benefit
> > to providing ImmediateData in low-cost local sub-lans.
> >
> > If you want to encourage it in a profile, fine, but demanding
> > it as the default in the core standard is not appropriate.
> >
> > Note that the behavior of SCSI is traditionally managed
> > entirely by the target.  As such, there has never before now
> > been a requirement for the target to, as a default, accept
> > any PDU except a command or task management function
> > that was not explicitly solicited.  That is one of the mechanisms
> > that assists SCSI in achieving a low-overhead zero copy
> > capability while operating with a large number of initiators
> > and with deep command queues.
> >
> > Bob Snively                        e-mail:    rsnively@brocade.com
> > Brocade Communications Systems     phone:  408 487 8135
> > 1745 Technology Drive
> > San Jose, CA 95110
> >
> >
> >
> > > -----Original Message-----
> > > From: John Hufferd [mailto:hufferd@us.ibm.com]
> > > Sent: Friday, September 28, 2001 1:33 AM
> > > To: Julian Satran
> > > Cc: ips@ece.cmu.edu
> > > Subject: RE: iscsi : iscsi parameter default values
> > >
> > >
> > >
> > > I also agree with this.  It should be yes.
> > >
> > > .
> > > .
> > > .
> > > John L. Hufferd
> > > Senior Technical Staff Member (STSM)
> > > IBM/SSG San Jose Ca
> > > Main Office (408) 256-0403, Tie: 276-0403,  eFax: (408) 904-4688
> > > Home Office (408) 997-6136
> > > Internet address: hufferd@us.ibm.com
> > >
> > >
> > > Julian Satran/Haifa/IBM@IBMIL@ece.cmu.edu on 09/27/2001
> 09:50:21 AM
> > >
> > > Sent by:  owner-ips@ece.cmu.edu
> > >
> > >
> > > To:   ips@ece.cmu.edu
> > > cc:
> > > Subject:  RE: iscsi : iscsi parameter default values
> > >
> > >
> > >
> > >
> > > The one that I have trouble living with is ImmediateData.
> > > This is important
> > > for low-end desktops and hardly matters for large boxes.
> > > As such I would suggest it stays as yes.
> > >
> > > Julo
> > >
> > >
> > >
> > >                     "Eddy
> > >                     Quicksall"            To:     "'Santosh Rao'"
> > > <santoshr@cup.hp.com>,
> > >                     <EQuicksall@med        <ips@ece.cmu.edu>
> > >                     iaone.net>            cc:
> > >                     Sent by:              Subject:     RE:
> > > iscsi : iscsi
> > > parameter default values
> > >                     owner-ips@ece.c
> > >                     mu.edu
> > >
> > >
> > >                     27-09-01 17:22
> > >                     Please respond
> > >                     to "Eddy
> > >                     Quicksall"
> > >
> > >
> > >
> > >
> > >
> > > I like your defaults below.
> > >
> > > But, section 5 says:
> > >
> > >  The initial Login request MUST include the InitiatorName and
> > >  SessionType key=value pairs.
> > >
> > > Since SessionType is REQUIRED, naming a default would imply a
> > > possible typo
> > > in the spec.
> > >
> > > Eddy
> > >
> > > -----Original Message-----
> > > From: Santosh Rao [mailto:santoshr@cup.hp.com]
> > > Sent: Wednesday, September 26, 2001 10:29 PM
> > > To: ips@ece.cmu.edu
> > > Subject: iscsi : iscsi parameter default values
> > >
> > >
> > > All,
> > >
> > > With the issue of mode page vs. login keys having [almost]
> > drawn to a
> > > close, I would like to
> > > raise the below issues again, on the subject of default
> > > values for login
> > > keys and other iscsi
> > > parameters :
> > >
> > >
> > >    * In keeping with traditional use of scsi mode pages,
> > iscsi should
> > > not specify any default
> > >      settings for any mode pages that continue to exist for iscsi.
> > > "Default values" for mode
> > >      pages are target specific and should not be bound to
> > the protocol
> > > draft.
> > >
> > >    * MORE IMPORTANTLY, I read the default for EMDP as
> being set to 1
> > > :-<  This implies that
> > >      targets must be always prepared to deal with out of
> > > order data and
> > > initiators must be
> > >      prepared to deal with out of order data, unless the initiator
> > > performs a mode select to
> > >      disable it. [which defeats all the previous advantages
> > > gained thru
> > > mandating use of login
> > >      keys for other negotiations.]. A default, if it were
> to exist,
> > > should be 0. (in-order, by
> > >      default).
> > >
> > >    * Conservative specification of defaults for login
> keys along the
> > > following lines :
> > >                             MaxConnections = 1
> > >                             FMarker = "no"
> > >                             InitialR2T = "yes"
> > >                             BidiInitialR2T = "yes"
> > >                             ImmediateData = "no"
> > >                             DataPDULength = 16
> > >                             MaxOutstandingR2T = 1
> > >                             DataPDUInOrder = "yes"
> > >                             ErrorRecoveryLevel = 0
> > >                             SessionType = "normal"
> > >
> > >    * Should the iscsi protocol require a "Lun Control Mode
> > Page"? IOW,
> > > is an EnableCRN bit
> > >      required at the transport layer ? If the device server
> > capability
> > > is to be negotiated , I
> > >      suggest this bit be moved to a SCSI ULP Mode Page such as the
> > > "Control Mode Page", through a
> > >      T10 change as a part of the SCSI changes being
> driven by iscsi.
> > >
> > > Comments ?
> > >
> > > Thanks,
> > > Santosh
> > >
> > >
> > > Santosh Rao wrote:
> > >
> > > > There are the separate issues of :
> > > >
> > > >    * iscsi's specification of defaults for its mode pages
> > > which is not
> > > in line with mode page
> > > >      usage. This impacts the target's ability to enforce
> > > values other
> > > than the protocol
> > > >      specified default, if the initiator were to not use mode
> > > sense/select.
> > > >
> > > >    * default settings for login keys.
> > > >
> > > >    * Is there a need for the "LUN Control Mode Page" and whether
> > > "Enable CRN" should be in a
> > > >      transport specific mode page ?
> > > >
> > > > which need to be driven to closure as well.
> > > >
> > > > Regards,
> > > > Santosh
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> >
> >
> >
> >
> >
>
>
>
>