RE: iSCSI: No Framing

["John Hufferd" <hufferd@us.ibm.com>]

Somesh,
This is not fair, we asked for Chip Vendors that were planning on using it,
to make themselves known and to tell us why.  Without exposing internal
designs, he has tried to do that, likewise Trebia, and some others have
said they want FIMs.  You may not want FIMs for your design, but you should
respect these folks that have been following our spec and found it to be
valuable in their designs.  They are putting their money on it not just
talk, so we should at least give them the credit which their position
deserves.  Yes, of course they are disappointed with the rhetoric around
this issue, since they and others have been following the spec, and can not
understand why others haven't, since they have determined it to be of
significant value.

His note does speak to the issues. (And clearly has some silly parts :-)

In fact he has found it of enough value to use now, since he believes they
can not wait for iWARP to happen before the issue is addressed (and he
really wants iWARP).  It should be useful to listen to folks that are
willing to put their money where their mouth is.

OK, that said: don't you think we can find a way to address both your
position and their position, since they both seem strong and heart felt.  I
would think that the "Should Implement" or at least "May Implement" would
give both sides enough reasonable room to meet on an even business plane.


.
.
.
John L. Hufferd
Senior Technical Staff Member (STSM)
IBM/SSG San Jose Ca
Main Office (408) 256-0403, Tie: 276-0403,  eFax: (408) 904-4688
Home Office (408) 997-6136, Cell: (408) 499-9702
Internet address: hufferd@us.ibm.com


"Somesh Gupta" <somesh_gupta@silverbacksystems.com>@ece.cmu.edu on
02/01/2002 07:10:13 PM

Please respond to <somesh_gupta@silverbacksystems.com>

Sent by:    owner-ips@ece.cmu.edu


To:    "Mukund, Shridhar" <Shridhar_Mukund@adaptec.com>, "'WENDT,JIM
       \(HP-Roseville,ex1\)'" <jim_wendt@hp.com>, <ips@ece.cmu.edu>
cc:
Subject:    RE: iSCSI: No Framing



Mukund,

With all due respect, taking on the mantle of
the noble engineer being denied by politically
motivated people, belittles the technical expertise
and strengths of the people who have expressed an
opinion against having any of the proposed schemes
at this time in the spec.

I would be very happy to work alongside a number of
the people on both sides of the issue (and that does
include you).

None of your arguments below refutes the technical
and some non-technical opinions expressed against
FIM.

Somesh

> -----Original Message-----
> From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of
> Mukund, Shridhar
> Sent: Friday, February 01, 2002 5:24 PM
> To: 'WENDT,JIM (HP-Roseville,ex1)'; ips@ece.cmu.edu
> Subject: RE: iSCSI: No Framing
>
>
> Dear Colleagues,
>
> PART 1 of 3 :
>
> The pertinent points that "I" see in Jim's email are:
>
> A. Need to hear from iSCSI chip implementers ...
>    The issues that you raise for e.g. in #2&#4 are simply
>    circumstantial( See PART 2 ). Answers to those questions
>    unnecessarily call for data flows and implementation
>    details that silicon vendors are not allowed to share
>    in public. While I do not know of many silicon vendors
>    in the multi gig space, clearly one of the competitions
>    I respect, namely Trebia, point to FIM as well. Granted,
>    no matter what, we are going to see several
>    trebia-on-the-other-end and adaptec-on-the-other-end
>    type of cost optimizations.
>
> B. The iWARP/TUF/SCTP under-current ...
>    The "only" message of significance in Jim's email is #5.
>    It seems like the iWARP/SCTP/TUF enthusiasts somehow feel
>    threatened that FIM-iSCSI will dilute the perceived value
>    proposition of iWARP :-) I for one am an enthusiast of
>    iWARP ideals myself, barring the proposed mechanics. I
>    would love to make a buck or two along with you in
>    building iWARP NICs, "at the right time". In the
>    meanwhile, iSCSI is the flag ship effort that has the
>    unique opportunity to make a dent in enabling IP Storage
>    visions. ( See PART 3 )
>
> My assertions are :
>     i) TUF/SCTP/iWARP is ruled out in the near term. The
>        mechanics are unclear as hell.
>     ii) FIM is the least intrusive, TCP transparent, means for
>        enabling low-cost(power,b/w,latency,memory,space,CPU)
>        RDMA transport of bulk data.
>     iii) I do not see significant technical reasons that
>        merit major changes to the iSCSI draft at this late
>        date.
>     iii) Making it OPTIONAL to use, and SHOULD only on send
>        provides a graceful and incremental deployment path
>        for "real":-) providers and users to succeed.
>
> I have personally contributed nothing to the iSCSI effort
> and do applaud the pains that several of you are taking to
> pull it all together. In that very same spirit, I respect
> David's decision w.r.t. consensus, whatever that turns out
> to be.
>
> Thanks.
>
> -Shridhar Mukund + Todd Sperry, Siva Munnangi, Dev ...
>
> ------------------------------------------------------------
> PART 2 of 3 : MUST delete, SHOULD read
>
> Dear Jim,
>
> Congratulations Jim! Seems like when you bowl, pins
> roll, unintentional as they may be. You make a "seemingly"
> well balanced set of arguments and "tactfully" tilt the
> balance towards your chosen side. I would love to be on
> your side ... maybe in another effort :-)
>
> I would like to introduce you to my respectable friend,
> who "vehemently" contradicts you on all accounts. One of
> his numerous quotes goes as follows:
> http://groups.yahoo.com/group/rdma/message/486
> "Much of today's usage of the Internet and IP networks
> is for buffer-to-buffer data transfers, often in the
> form of bulk data ... Gigabit and faster network transfers
> incur 'heavy' system resource costs, including both CPU
> use and system bus bandwidth, particularly on the
> receiving side ... The costs incurred on hosts for protocol
> processing and management has 'inhibited' the use of IP
> in the high speed bulk data domain. ... Bulk data transfer
> is dominated by the costs of copying and validating
> incoming data in order to place it at its ultimate
> destination."
>
> The good friend I quote above is none other than Jim
> Wendt himself!!! I REST MY CASE.
>
> Thanks.
>
> -Shridhar Mukund
>
> ----------------------------------------------------------------
> PART 3 of 3: MUST delete, OPTIONAL read
>
> On the lighter side ... since the opponents have "no" technical
> arguments whatsoever against FIM and it is all turning out to
> be a pure political gimmick, I will put on my rusting tin
> politician hat :-)
>
> My dear fellow iSCSI country (wo)men: If our goals are anything
> short of enabling the HP's, Cisco's, uSoft's, IBM's and EMC's of
> the world in globalization of storage for McDonald's and Macy's
> from Kabul to Somalia, via iSCSI, we have lost our identity!
> ( \insert APPLAUSE for 13 seconds )We are no more protected by
> the vast oceans between us and other Storage efforts. The
> freedom of iSCSI America is threatened from within by elements
> who will not blink twice when it comes to using the world's most
> potent BOO-bombs ... and grinnn while we end up sifting through
> the rubbles for iSCSI, youSCSI and theySCSI.
> ( \insert APPLAUSE for 11 seconds ) Beware of the elements that
> sleep with "our" very ideals in their privacy(burkha clad
> though) and yet attempt to destabilize us, only to accomplish
> their mutated agenda.( \insert APPLAUSE for 57 seconds )
>
> No offense folks. I respect each and every one of you and
> especially Jim. I think that he was only attempting to
> question, "are we sure ... should we commit ...".
> I disagree with him anyway!
>
> - the running(actually limping) mate :-) :-) :-)
>
> -----Original Message-----
> From: WENDT,JIM (HP-Roseville,ex1) [mailto:jim_wendt@hp.com]
> Sent: Tuesday, January 29, 2002 10:47 PM
> To: ips@ece.cmu.edu
> Subject: iSCSI: No Framing
>
>
> Perhaps we should discuss the possibility of not
> specifying any framing mechanism (FIM or COWS) in the
> first version of iSCSI.
>
> The arguments for not specifying framing include
> (there may be others as well):
> 1) The brute force approach of putting TCP receiver
> reassembly memory on the iSCSI NIC will work for both
> 1Gbps and 10Gbps. Cost is incurred for memory chips,
> ASIC pins, power, and board space. But, it is a
> feasible approach.
>
> 2) I/O bus latency (or bandwidth limitations) could
> mandate inbound buffering (as a 'smoothing' buffer)
> from the iSCSI NIC to the host system. If this buffer
> memory is large enough to have to be off-chip, then
> it will require some minimum number of memory chips
> to provide the necessary bandwidth, and the
> memory/pins/power/space costs will be incurred
> anyway.
>
> 3) Very large receive buffering on the NIC is only
> required for high-bandwidth links traversing large
> geographic distances (which may not be the
> predominate case). These specialized implementations
> can cost more (e.g. more memory/power/pins/etc) and
> customers will likely be willing to pay accordingly.
>
> 4) Many NICs will likely aspire to be combo
> iSCSI+TOE+Ethernet NICs allowing the host system to
> use a single Ethernet port for all of its network
> communications. The TOE function on this combo NIC
> will already require TCP receive buffering and off-
> chip buffer memory to support the existing sockets
> interface receive model.  More importantly, to allow
> senders to assume ownership of the buffer upon return
> from a socket send call, the TOE NIC would need to
> copy application send buffers into NIC memory as
> well.
>
> 5) The framing/marker mechanism will be an iSCSI-only
> solution. It is quite possible that the problem will
> be solved via a different, and hopefully more
> general, mechanism for other ULPs.
>
> 6) Both FIM and COWS are poor solutions for software
> implementation. COWS requires, at a minimum, the
> sender to read every word in the buffer. FIM either
> imposes additional sender gather processing (iovecs)
> or requires a data buffer copy (on systems that don't
> support HW gather on sends).
>
> 7) Unless all senders thoroughly test framing/markers
> now (i.e. unless the framing method is a MUST
> implement), there is potential for future
> interoperability problems as framing/marker receivers
> are deployed in the future.
>
> 8) Neither FIM nor COWS is an elegant solution. Are
> we comfortable with the legacy we are creating under
> the umbrella of state-of-the-art IP networked
> storage?
>
> 9) Is it essential to build in forward compatibility
> now, or will there be a different solution in the
> 10Gbps or 40Gbps timeframe - perhaps involving iSCSI-
> 2?  Will it be reasonable to update or bridge initial
> 1Gbps deployments?
>
>
> So, it would be good to hear from several iSCSI
> NIC/chip implementors who:
> - have or plan to implement FIM or COWS (or some
> other framing mechanism) and take advantage of it to
> minimize demands on on-NIC buffer memory
> bandwidth/quantity.
> - believe that all-buffers-on-chip solutions are
> feasible and valid (wrt the points above, including
> #2)
> - can quantify the memory/pin/power/space cost
> savings for all-buffers-on-chip solutions
>
> Jim
>