RE: iSCSI: No Framing

["Julian Satran" <Julian_Satran@il.ibm.com>]

Not exacly markers - those appeared in Haifa-summer of 2000 but something similar - aligned PDUs.

Julo

"Somesh Gupta" <somesh_gupta@silverbacksystems.com> Sent by: owner-ips@ece.cmu.edu 05-02-02 21:20 Please respond to somesh_gupta

To:        John Hufferd/San Jose/IBM@IBMUS         cc:        "Jim Pinkerton" <jpink@microsoft.com>, "Mukund, Shridhar" <Shridhar_Mukund@adaptec.com>, <ips@ece.cmu.edu>         Subject:        RE: iSCSI: No Framing

John,

I have to point for the record, that I was the one
that proposed markers over a year ago (not that many
other did not also have this idea). I have also tried
to keep this debate at a technical level. There
are reasonable folks on both sides of the debate,
and we should just continue in that spirit.

Somesh

> -----Original Message-----
> From: John Hufferd [mailto:hufferd@us.ibm.com]
> Sent: Tuesday, February 05, 2002 10:55 AM
> To: somesh_gupta@silverbacksystems.com
> Cc: Jim Pinkerton; Mukund, Shridhar; ips@ece.cmu.edu
> Subject: RE: iSCSI: No Framing
>
>
>
> Somesh,
> I usually give higher weight to the opinions of folks that take their
> opinions and put their money on it, than the folks that do not.   So I am
> not sure you can so easily dismiss the ideas of folks that are committed
> with actual money.
>
> .
> .
> .
> John L. Hufferd
> Senior Technical Staff Member (STSM)
> IBM/SSG San Jose Ca
> Main Office (408) 256-0403, Tie: 276-0403,  eFax: (408) 904-4688
> Home Office (408) 997-6136, Cell: (408) 499-9702
> Internet address: hufferd@us.ibm.com
>
>
> "Somesh Gupta" <somesh_gupta@silverbacksystems.com> on 02/05/2002 10:25:45
> AM
>
> Please respond to <somesh_gupta@silverbacksystems.com>
>
> To:    "Jim Pinkerton" <jpink@microsoft.com>, John Hufferd/San
>        Jose/IBM@IBMUS
> cc:    "Mukund, Shridhar" <Shridhar_Mukund@adaptec.com>, <ips@ece.cmu.edu>
> Subject:    RE: iSCSI: No Framing
>
>
>
> Jim,
>
> > -----Original Message-----
> > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of
> > Jim Pinkerton
> > Sent: Tuesday, February 05, 2002 6:53 AM
> > To: John Hufferd; somesh_gupta@silverbacksystems.com
> > Cc: Mukund, Shridhar; ips@ece.cmu.edu
> > Subject: RE: iSCSI: No Framing
> >
> >
> >
> > In reviewing the mail alias, and after having many conversations with
> > various folks, I would like to retract my prior opinion on ripping out
> > markers. It appears to me as though _every_ NIC vendor is in support of
> > markers.
>
>   Every NIC vendor may implement markers (or as is obvious from
>   interoperability results, may have the capability to implement
>   markers). On whether every (person who works at a) NIC/Chip vendor
>   supports it from a technical perspective (i.e. whether it solves
>   anything, or is a useful solution, or whether the analysis people
>   are using is correct) supports it, I think the count seems to be
>   evenly split.
> >
> > To me the evidence in support of keeping in "Sync and Steering with
> > Fixed Interval Markers" is pretty compelling (now that I've been
> > re-educated). Without going into proprietary issues - how often do NIC
> > vendors agree unanimously on something? Also, a software implementation
> > of markers is tractable.
> >
> > I have not heard a single NIC vendor in support of COWS, and I
> > personally don't support COWS either. Thus I would recommend that COWS
> > be "ripped out".
> >
> >
> > Jim
> >
>
>   Julian indicates in his message that we are not really sure what
>   a good solution is and if we put something in, we are likely
>   to get to a better one in the second draft. I also concur that
>   the whole problem/solution space is experimental in nature, and
>   ought to be treated as such.
>
>   I think it is meaningless to assume that one person or another
>   has a magic solution here. The problems are fairly well understood.
>   One can talk about it from an algorithmic perspective (like
>   error recovery) which will expose some of the assumptions/compromises
>   behind this. Of course, whether that will pass muster with
>   the community at large is another issue.
> >
> >
> >
> > -----Original Message-----
> > From: John Hufferd [mailto:hufferd@us.ibm.com]
> > Sent: Friday, February 01, 2002 11:31 PM
> > To: somesh_gupta@silverbacksystems.com
> > Cc: Mukund, Shridhar; ips@ece.cmu.edu
> > Subject: RE: iSCSI: No Framing
> >
> >
> > Somesh,
> > This is not fair, we asked for Chip Vendors that were planning on using
> > it,
> > to make themselves known and to tell us why.  Without exposing internal
> > designs, he has tried to do that, likewise Trebia, and some others have
> > said they want FIMs.  You may not want FIMs for your design, but you
> > should
> > respect these folks that have been following our spec and found it to be
> > valuable in their designs.  They are putting their money on it not just
> > talk, so we should at least give them the credit which their position
> > deserves.  Yes, of course they are disappointed with the rhetoric around
> > this issue, since they and others have been following the spec, and can
> > not
> > understand why others haven't, since they have determined it to be of
> > significant value.
> >
> > His note does speak to the issues. (And clearly has some silly parts :-)
> >
> > In fact he has found it of enough value to use now, since he believes
> > they
> > can not wait for iWARP to happen before the issue is addressed (and he
> > really wants iWARP).  It should be useful to listen to folks that are
> > willing to put their money where their mouth is.
> >
> > OK, that said: don't you think we can find a way to address both your
> > position and their position, since they both seem strong and heart felt.
> > I
> > would think that the "Should Implement" or at least "May Implement"
> > would
> > give both sides enough reasonable room to meet on an even business
> > plane.
> >
> >
> > .
> > .
> > .
> > John L. Hufferd
> > Senior Technical Staff Member (STSM)
> > IBM/SSG San Jose Ca
> > Main Office (408) 256-0403, Tie: 276-0403,  eFax: (408) 904-4688
> > Home Office (408) 997-6136, Cell: (408) 499-9702
> > Internet address: hufferd@us.ibm.com
> >
> >
> > "Somesh Gupta" <somesh_gupta@silverbacksystems.com>@ece.cmu.edu on
> > 02/01/2002 07:10:13 PM
> >
> > Please respond to <somesh_gupta@silverbacksystems.com>
> >
> > Sent by:    owner-ips@ece.cmu.edu
> >
> >
> > To:    "Mukund, Shridhar" <Shridhar_Mukund@adaptec.com>, "'WENDT,JIM
> >        \(HP-Roseville,ex1\)'" <jim_wendt@hp.com>, <ips@ece.cmu.edu>
> > cc:
> > Subject:    RE: iSCSI: No Framing
> >
> >
> >
> > Mukund,
> >
> > With all due respect, taking on the mantle of
> > the noble engineer being denied by politically
> > motivated people, belittles the technical expertise
> > and strengths of the people who have expressed an
> > opinion against having any of the proposed schemes
> > at this time in the spec.
> >
> > I would be very happy to work alongside a number of
> > the people on both sides of the issue (and that does
> > include you).
> >
> > None of your arguments below refutes the technical
> > and some non-technical opinions expressed against
> > FIM.
> >
> > Somesh
> >
> > > -----Original Message-----
> > > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of
> > > Mukund, Shridhar
> > > Sent: Friday, February 01, 2002 5:24 PM
> > > To: 'WENDT,JIM (HP-Roseville,ex1)'; ips@ece.cmu.edu
> > > Subject: RE: iSCSI: No Framing
> > >
> > >
> > > Dear Colleagues,
> > >
> > > PART 1 of 3 :
> > >
> > > The pertinent points that "I" see in Jim's email are:
> > >
> > > A. Need to hear from iSCSI chip implementers ...
> > >    The issues that you raise for e.g. in #2&#4 are simply
> > >    circumstantial( See PART 2 ). Answers to those questions
> > >    unnecessarily call for data flows and implementation
> > >    details that silicon vendors are not allowed to share
> > >    in public. While I do not know of many silicon vendors
> > >    in the multi gig space, clearly one of the competitions
> > >    I respect, namely Trebia, point to FIM as well. Granted,
> > >    no matter what, we are going to see several
> > >    trebia-on-the-other-end and adaptec-on-the-other-end
> > >    type of cost optimizations.
> > >
> > > B. The iWARP/TUF/SCTP under-current ...
> > >    The "only" message of significance in Jim's email is #5.
> > >    It seems like the iWARP/SCTP/TUF enthusiasts somehow feel
> > >    threatened that FIM-iSCSI will dilute the perceived value
> > >    proposition of iWARP :-) I for one am an enthusiast of
> > >    iWARP ideals myself, barring the proposed mechanics. I
> > >    would love to make a buck or two along with you in
> > >    building iWARP NICs, "at the right time". In the
> > >    meanwhile, iSCSI is the flag ship effort that has the
> > >    unique opportunity to make a dent in enabling IP Storage
> > >    visions. ( See PART 3 )
> > >
> > > My assertions are :
> > >     i) TUF/SCTP/iWARP is ruled out in the near term. The
> > >        mechanics are unclear as hell.
> > >     ii) FIM is the least intrusive, TCP transparent, means for
> > >        enabling low-cost(power,b/w,latency,memory,space,CPU)
> > >        RDMA transport of bulk data.
> > >     iii) I do not see significant technical reasons that
> > >        merit major changes to the iSCSI draft at this late
> > >        date.
> > >     iii) Making it OPTIONAL to use, and SHOULD only on send
> > >        provides a graceful and incremental deployment path
> > >        for "real":-) providers and users to succeed.
> > >
> > > I have personally contributed nothing to the iSCSI effort
> > > and do applaud the pains that several of you are taking to
> > > pull it all together. In that very same spirit, I respect
> > > David's decision w.r.t. consensus, whatever that turns out
> > > to be.
> > >
> > > Thanks.
> > >
> > > -Shridhar Mukund + Todd Sperry, Siva Munnangi, Dev ...
> > >
> > > ------------------------------------------------------------
> > > PART 2 of 3 : MUST delete, SHOULD read
> > >
> > > Dear Jim,
> > >
> > > Congratulations Jim! Seems like when you bowl, pins
> > > roll, unintentional as they may be. You make a "seemingly"
> > > well balanced set of arguments and "tactfully" tilt the
> > > balance towards your chosen side. I would love to be on
> > > your side ... maybe in another effort :-)
> > >
> > > I would like to introduce you to my respectable friend,
> > > who "vehemently" contradicts you on all accounts. One of
> > > his numerous quotes goes as follows:
> > > http://groups.yahoo.com/group/rdma/message/486
> > > "Much of today's usage of the Internet and IP networks
> > > is for buffer-to-buffer data transfers, often in the
> > > form of bulk data ... Gigabit and faster network transfers
> > > incur 'heavy' system resource costs, including both CPU
> > > use and system bus bandwidth, particularly on the
> > > receiving side ... The costs incurred on hosts for protocol
> > > processing and management has 'inhibited' the use of IP
> > > in the high speed bulk data domain. ... Bulk data transfer
> > > is dominated by the costs of copying and validating
> > > incoming data in order to place it at its ultimate
> > > destination."
> > >
> > > The good friend I quote above is none other than Jim
> > > Wendt himself!!! I REST MY CASE.
> > >
> > > Thanks.
> > >
> > > -Shridhar Mukund
> > >
> > > ----------------------------------------------------------------
> > > PART 3 of 3: MUST delete, OPTIONAL read
> > >
> > > On the lighter side ... since the opponents have "no" technical
> > > arguments whatsoever against FIM and it is all turning out to
> > > be a pure political gimmick, I will put on my rusting tin
> > > politician hat :-)
> > >
> > > My dear fellow iSCSI country (wo)men: If our goals are anything
> > > short of enabling the HP's, Cisco's, uSoft's, IBM's and EMC's of
> > > the world in globalization of storage for McDonald's and Macy's
> > > from Kabul to Somalia, via iSCSI, we have lost our identity!
> > > ( \insert APPLAUSE for 13 seconds )We are no more protected by
> > > the vast oceans between us and other Storage efforts. The
> > > freedom of iSCSI America is threatened from within by elements
> > > who will not blink twice when it comes to using the world's most
> > > potent BOO-bombs ... and grinnn while we end up sifting through
> > > the rubbles for iSCSI, youSCSI and theySCSI.
> > > ( \insert APPLAUSE for 11 seconds ) Beware of the elements that
> > > sleep with "our" very ideals in their privacy(burkha clad
> > > though) and yet attempt to destabilize us, only to accomplish
> > > their mutated agenda.( \insert APPLAUSE for 57 seconds )
> > >
> > > No offense folks. I respect each and every one of you and
> > > especially Jim. I think that he was only attempting to
> > > question, "are we sure ... should we commit ...".
> > > I disagree with him anyway!
> > >
> > > - the running(actually limping) mate :-) :-) :-)
> > >
> > > -----Original Message-----
> > > From: WENDT,JIM (HP-Roseville,ex1) [mailto:jim_wendt@hp.com]
> > > Sent: Tuesday, January 29, 2002 10:47 PM
> > > To: ips@ece.cmu.edu
> > > Subject: iSCSI: No Framing
> > >
> > >
> > > Perhaps we should discuss the possibility of not
> > > specifying any framing mechanism (FIM or COWS) in the
> > > first version of iSCSI.
> > >
> > > The arguments for not specifying framing include
> > > (there may be others as well):
> > > 1) The brute force approach of putting TCP receiver
> > > reassembly memory on the iSCSI NIC will work for both
> > > 1Gbps and 10Gbps. Cost is incurred for memory chips,
> > > ASIC pins, power, and board space. But, it is a
> > > feasible approach.
> > >
> > > 2) I/O bus latency (or bandwidth limitations) could
> > > mandate inbound buffering (as a 'smoothing' buffer)
> > > from the iSCSI NIC to the host system. If this buffer
> > > memory is large enough to have to be off-chip, then
> > > it will require some minimum number of memory chips
> > > to provide the necessary bandwidth, and the
> > > memory/pins/power/space costs will be incurred
> > > anyway.
> > >
> > > 3) Very large receive buffering on the NIC is only
> > > required for high-bandwidth links traversing large
> > > geographic distances (which may not be the
> > > predominate case). These specialized implementations
> > > can cost more (e.g. more memory/power/pins/etc) and
> > > customers will likely be willing to pay accordingly.
> > >
> > > 4) Many NICs will likely aspire to be combo
> > > iSCSI+TOE+Ethernet NICs allowing the host system to
> > > use a single Ethernet port for all of its network
> > > communications. The TOE function on this combo NIC
> > > will already require TCP receive buffering and off-
> > > chip buffer memory to support the existing sockets
> > > interface receive model.  More importantly, to allow
> > > senders to assume ownership of the buffer upon return
> > > from a socket send call, the TOE NIC would need to
> > > copy application send buffers into NIC memory as
> > > well.
> > >
> > > 5) The framing/marker mechanism will be an iSCSI-only
> > > solution. It is quite possible that the problem will
> > > be solved via a different, and hopefully more
> > > general, mechanism for other ULPs.
> > >
> > > 6) Both FIM and COWS are poor solutions for software
> > > implementation. COWS requires, at a minimum, the
> > > sender to read every word in the buffer. FIM either
> > > imposes additional sender gather processing (iovecs)
> > > or requires a data buffer copy (on systems that don't
> > > support HW gather on sends).
> > >
> > > 7) Unless all senders thoroughly test framing/markers
> > > now (i.e. unless the framing method is a MUST
> > > implement), there is potential for future
> > > interoperability problems as framing/marker receivers
> > > are deployed in the future.
> > >
> > > 8) Neither FIM nor COWS is an elegant solution. Are
> > > we comfortable with the legacy we are creating under
> > > the umbrella of state-of-the-art IP networked
> > > storage?
> > >
> > > 9) Is it essential to build in forward compatibility
> > > now, or will there be a different solution in the
> > > 10Gbps or 40Gbps timeframe - perhaps involving iSCSI-
> > > 2?  Will it be reasonable to update or bridge initial
> > > 1Gbps deployments?
> > >
> > >
> > > So, it would be good to hear from several iSCSI
> > > NIC/chip implementors who:
> > > - have or plan to implement FIM or COWS (or some
> > > other framing mechanism) and take advantage of it to
> > > minimize demands on on-NIC buffer memory
> > > bandwidth/quantity.
> > > - believe that all-buffers-on-chip solutions are
> > > feasible and valid (wrt the points above, including
> > > #2)
> > > - can quantify the memory/pin/power/space cost
> > > savings for all-buffers-on-chip solutions
> > >
> > > Jim
> > >
> >
> >
> >
> >
>
>
>
>