RE: iSCSI: No Framing

["Somesh Gupta" <somesh_gupta@silverbacksystems.com>]

Jim,

> -----Original Message-----
> From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of
> Jim Pinkerton
> Sent: Tuesday, February 05, 2002 6:53 AM
> To: John Hufferd; somesh_gupta@silverbacksystems.com
> Cc: Mukund, Shridhar; ips@ece.cmu.edu
> Subject: RE: iSCSI: No Framing
> 
> 
> 
> In reviewing the mail alias, and after having many conversations with
> various folks, I would like to retract my prior opinion on ripping out
> markers. It appears to me as though _every_ NIC vendor is in support of
> markers. 

  Every NIC vendor may implement markers (or as is obvious from
  interoperability results, may have the capability to implement
  markers). On whether every (person who works at a) NIC/Chip vendor
  supports it from a technical perspective (i.e. whether it solves
  anything, or is a useful solution, or whether the analysis people
  are using is correct) supports it, I think the count seems to be
  evenly split.
> 
> To me the evidence in support of keeping in "Sync and Steering with
> Fixed Interval Markers" is pretty compelling (now that I've been
> re-educated). Without going into proprietary issues - how often do NIC
> vendors agree unanimously on something? Also, a software implementation
> of markers is tractable.
> 
> I have not heard a single NIC vendor in support of COWS, and I
> personally don't support COWS either. Thus I would recommend that COWS
> be "ripped out".
> 
> 
> Jim
> 

  Julian indicates in his message that we are not really sure what
  a good solution is and if we put something in, we are likely
  to get to a better one in the second draft. I also concur that
  the whole problem/solution space is experimental in nature, and
  ought to be treated as such.

  I think it is meaningless to assume that one person or another
  has a magic solution here. The problems are fairly well understood.
  One can talk about it from an algorithmic perspective (like
  error recovery) which will expose some of the assumptions/compromises
  behind this. Of course, whether that will pass muster with
  the community at large is another issue.
> 
> 
> 
> -----Original Message-----
> From: John Hufferd [mailto:hufferd@us.ibm.com] 
> Sent: Friday, February 01, 2002 11:31 PM
> To: somesh_gupta@silverbacksystems.com
> Cc: Mukund, Shridhar; ips@ece.cmu.edu
> Subject: RE: iSCSI: No Framing
> 
> 
> Somesh,
> This is not fair, we asked for Chip Vendors that were planning on using
> it,
> to make themselves known and to tell us why.  Without exposing internal
> designs, he has tried to do that, likewise Trebia, and some others have
> said they want FIMs.  You may not want FIMs for your design, but you
> should
> respect these folks that have been following our spec and found it to be
> valuable in their designs.  They are putting their money on it not just
> talk, so we should at least give them the credit which their position
> deserves.  Yes, of course they are disappointed with the rhetoric around
> this issue, since they and others have been following the spec, and can
> not
> understand why others haven't, since they have determined it to be of
> significant value.
> 
> His note does speak to the issues. (And clearly has some silly parts :-)
> 
> In fact he has found it of enough value to use now, since he believes
> they
> can not wait for iWARP to happen before the issue is addressed (and he
> really wants iWARP).  It should be useful to listen to folks that are
> willing to put their money where their mouth is.
> 
> OK, that said: don't you think we can find a way to address both your
> position and their position, since they both seem strong and heart felt.
> I
> would think that the "Should Implement" or at least "May Implement"
> would
> give both sides enough reasonable room to meet on an even business
> plane.
> 
> 
> .
> .
> .
> John L. Hufferd
> Senior Technical Staff Member (STSM)
> IBM/SSG San Jose Ca
> Main Office (408) 256-0403, Tie: 276-0403,  eFax: (408) 904-4688
> Home Office (408) 997-6136, Cell: (408) 499-9702
> Internet address: hufferd@us.ibm.com
> 
> 
> "Somesh Gupta" <somesh_gupta@silverbacksystems.com>@ece.cmu.edu on
> 02/01/2002 07:10:13 PM
> 
> Please respond to <somesh_gupta@silverbacksystems.com>
> 
> Sent by:    owner-ips@ece.cmu.edu
> 
> 
> To:    "Mukund, Shridhar" <Shridhar_Mukund@adaptec.com>, "'WENDT,JIM
>        \(HP-Roseville,ex1\)'" <jim_wendt@hp.com>, <ips@ece.cmu.edu>
> cc:
> Subject:    RE: iSCSI: No Framing
> 
> 
> 
> Mukund,
> 
> With all due respect, taking on the mantle of
> the noble engineer being denied by politically
> motivated people, belittles the technical expertise
> and strengths of the people who have expressed an
> opinion against having any of the proposed schemes
> at this time in the spec.
> 
> I would be very happy to work alongside a number of
> the people on both sides of the issue (and that does
> include you).
> 
> None of your arguments below refutes the technical
> and some non-technical opinions expressed against
> FIM.
> 
> Somesh
> 
> > -----Original Message-----
> > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of
> > Mukund, Shridhar
> > Sent: Friday, February 01, 2002 5:24 PM
> > To: 'WENDT,JIM (HP-Roseville,ex1)'; ips@ece.cmu.edu
> > Subject: RE: iSCSI: No Framing
> >
> >
> > Dear Colleagues,
> >
> > PART 1 of 3 :
> >
> > The pertinent points that "I" see in Jim's email are:
> >
> > A. Need to hear from iSCSI chip implementers ...
> >    The issues that you raise for e.g. in #2&#4 are simply
> >    circumstantial( See PART 2 ). Answers to those questions
> >    unnecessarily call for data flows and implementation
> >    details that silicon vendors are not allowed to share
> >    in public. While I do not know of many silicon vendors
> >    in the multi gig space, clearly one of the competitions
> >    I respect, namely Trebia, point to FIM as well. Granted,
> >    no matter what, we are going to see several
> >    trebia-on-the-other-end and adaptec-on-the-other-end
> >    type of cost optimizations.
> >
> > B. The iWARP/TUF/SCTP under-current ...
> >    The "only" message of significance in Jim's email is #5.
> >    It seems like the iWARP/SCTP/TUF enthusiasts somehow feel
> >    threatened that FIM-iSCSI will dilute the perceived value
> >    proposition of iWARP :-) I for one am an enthusiast of
> >    iWARP ideals myself, barring the proposed mechanics. I
> >    would love to make a buck or two along with you in
> >    building iWARP NICs, "at the right time". In the
> >    meanwhile, iSCSI is the flag ship effort that has the
> >    unique opportunity to make a dent in enabling IP Storage
> >    visions. ( See PART 3 )
> >
> > My assertions are :
> >     i) TUF/SCTP/iWARP is ruled out in the near term. The
> >        mechanics are unclear as hell.
> >     ii) FIM is the least intrusive, TCP transparent, means for
> >        enabling low-cost(power,b/w,latency,memory,space,CPU)
> >        RDMA transport of bulk data.
> >     iii) I do not see significant technical reasons that
> >        merit major changes to the iSCSI draft at this late
> >        date.
> >     iii) Making it OPTIONAL to use, and SHOULD only on send
> >        provides a graceful and incremental deployment path
> >        for "real":-) providers and users to succeed.
> >
> > I have personally contributed nothing to the iSCSI effort
> > and do applaud the pains that several of you are taking to
> > pull it all together. In that very same spirit, I respect
> > David's decision w.r.t. consensus, whatever that turns out
> > to be.
> >
> > Thanks.
> >
> > -Shridhar Mukund + Todd Sperry, Siva Munnangi, Dev ...
> >
> > ------------------------------------------------------------
> > PART 2 of 3 : MUST delete, SHOULD read
> >
> > Dear Jim,
> >
> > Congratulations Jim! Seems like when you bowl, pins
> > roll, unintentional as they may be. You make a "seemingly"
> > well balanced set of arguments and "tactfully" tilt the
> > balance towards your chosen side. I would love to be on
> > your side ... maybe in another effort :-)
> >
> > I would like to introduce you to my respectable friend,
> > who "vehemently" contradicts you on all accounts. One of
> > his numerous quotes goes as follows:
> > http://groups.yahoo.com/group/rdma/message/486
> > "Much of today's usage of the Internet and IP networks
> > is for buffer-to-buffer data transfers, often in the
> > form of bulk data ... Gigabit and faster network transfers
> > incur 'heavy' system resource costs, including both CPU
> > use and system bus bandwidth, particularly on the
> > receiving side ... The costs incurred on hosts for protocol
> > processing and management has 'inhibited' the use of IP
> > in the high speed bulk data domain. ... Bulk data transfer
> > is dominated by the costs of copying and validating
> > incoming data in order to place it at its ultimate
> > destination."
> >
> > The good friend I quote above is none other than Jim
> > Wendt himself!!! I REST MY CASE.
> >
> > Thanks.
> >
> > -Shridhar Mukund
> >
> > ----------------------------------------------------------------
> > PART 3 of 3: MUST delete, OPTIONAL read
> >
> > On the lighter side ... since the opponents have "no" technical
> > arguments whatsoever against FIM and it is all turning out to
> > be a pure political gimmick, I will put on my rusting tin
> > politician hat :-)
> >
> > My dear fellow iSCSI country (wo)men: If our goals are anything
> > short of enabling the HP's, Cisco's, uSoft's, IBM's and EMC's of
> > the world in globalization of storage for McDonald's and Macy's
> > from Kabul to Somalia, via iSCSI, we have lost our identity!
> > ( \insert APPLAUSE for 13 seconds )We are no more protected by
> > the vast oceans between us and other Storage efforts. The
> > freedom of iSCSI America is threatened from within by elements
> > who will not blink twice when it comes to using the world's most
> > potent BOO-bombs ... and grinnn while we end up sifting through
> > the rubbles for iSCSI, youSCSI and theySCSI.
> > ( \insert APPLAUSE for 11 seconds ) Beware of the elements that
> > sleep with "our" very ideals in their privacy(burkha clad
> > though) and yet attempt to destabilize us, only to accomplish
> > their mutated agenda.( \insert APPLAUSE for 57 seconds )
> >
> > No offense folks. I respect each and every one of you and
> > especially Jim. I think that he was only attempting to
> > question, "are we sure ... should we commit ...".
> > I disagree with him anyway!
> >
> > - the running(actually limping) mate :-) :-) :-)
> >
> > -----Original Message-----
> > From: WENDT,JIM (HP-Roseville,ex1) [mailto:jim_wendt@hp.com]
> > Sent: Tuesday, January 29, 2002 10:47 PM
> > To: ips@ece.cmu.edu
> > Subject: iSCSI: No Framing
> >
> >
> > Perhaps we should discuss the possibility of not
> > specifying any framing mechanism (FIM or COWS) in the
> > first version of iSCSI.
> >
> > The arguments for not specifying framing include
> > (there may be others as well):
> > 1) The brute force approach of putting TCP receiver
> > reassembly memory on the iSCSI NIC will work for both
> > 1Gbps and 10Gbps. Cost is incurred for memory chips,
> > ASIC pins, power, and board space. But, it is a
> > feasible approach.
> >
> > 2) I/O bus latency (or bandwidth limitations) could
> > mandate inbound buffering (as a 'smoothing' buffer)
> > from the iSCSI NIC to the host system. If this buffer
> > memory is large enough to have to be off-chip, then
> > it will require some minimum number of memory chips
> > to provide the necessary bandwidth, and the
> > memory/pins/power/space costs will be incurred
> > anyway.
> >
> > 3) Very large receive buffering on the NIC is only
> > required for high-bandwidth links traversing large
> > geographic distances (which may not be the
> > predominate case). These specialized implementations
> > can cost more (e.g. more memory/power/pins/etc) and
> > customers will likely be willing to pay accordingly.
> >
> > 4) Many NICs will likely aspire to be combo
> > iSCSI+TOE+Ethernet NICs allowing the host system to
> > use a single Ethernet port for all of its network
> > communications. The TOE function on this combo NIC
> > will already require TCP receive buffering and off-
> > chip buffer memory to support the existing sockets
> > interface receive model.  More importantly, to allow
> > senders to assume ownership of the buffer upon return
> > from a socket send call, the TOE NIC would need to
> > copy application send buffers into NIC memory as
> > well.
> >
> > 5) The framing/marker mechanism will be an iSCSI-only
> > solution. It is quite possible that the problem will
> > be solved via a different, and hopefully more
> > general, mechanism for other ULPs.
> >
> > 6) Both FIM and COWS are poor solutions for software
> > implementation. COWS requires, at a minimum, the
> > sender to read every word in the buffer. FIM either
> > imposes additional sender gather processing (iovecs)
> > or requires a data buffer copy (on systems that don't
> > support HW gather on sends).
> >
> > 7) Unless all senders thoroughly test framing/markers
> > now (i.e. unless the framing method is a MUST
> > implement), there is potential for future
> > interoperability problems as framing/marker receivers
> > are deployed in the future.
> >
> > 8) Neither FIM nor COWS is an elegant solution. Are
> > we comfortable with the legacy we are creating under
> > the umbrella of state-of-the-art IP networked
> > storage?
> >
> > 9) Is it essential to build in forward compatibility
> > now, or will there be a different solution in the
> > 10Gbps or 40Gbps timeframe - perhaps involving iSCSI-
> > 2?  Will it be reasonable to update or bridge initial
> > 1Gbps deployments?
> >
> >
> > So, it would be good to hear from several iSCSI
> > NIC/chip implementors who:
> > - have or plan to implement FIM or COWS (or some
> > other framing mechanism) and take advantage of it to
> > minimize demands on on-NIC buffer memory
> > bandwidth/quantity.
> > - believe that all-buffers-on-chip solutions are
> > feasible and valid (wrt the points above, including
> > #2)
> > - can quantify the memory/pin/power/space cost
> > savings for all-buffers-on-chip solutions
> >
> > Jim
> >
> 
> 
> 
>