RE: Wot I `know' about COWS in hardware

["Somesh Gupta" <somesh_gupta@silverbacksystems.com>]

I think the COBS/COWS should more potential than the markers proposal, or

the ULP framing without COBS. There is one case where it does add more overhead,

but the question is how prevelent is the scenario - when outbound zero copy is

enabled/possible and the NIC does checksum offload and cannot be changed to

do COBS. Of course changes are required :-).

 

I also hope that data-centers will use acclerated iSCSI/Clustering HBAs/NICs rather

than the current solutions. The current solutions MAY be useful for desktops/laptops

where hopefully there are plenty of spare cycles to do COBS in software.

 

COBS also has alignment benefits - the header could be aligned with the ULP PDU,

and the ULP PDU can be aligned with the TCP header and there are no false

positives. The alignment with the TCP header may not always happen (the mythical

box in the middle that does TCP resegmentation), but can be detected - in the

presence of such a box, the performance could reduce to the levels encountered

when IP fragmentation happens.

 

I think it is better to have a couple of inter-operable implementations that demonstrate

the benefit of any of the alternate proposals (especially markers vs cobs) before selecting

one.

-----Original Message-----
From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of Julian Satran
Sent: Monday, December 17, 2001 9:27 AM
To: Jim Pinkerton; ips@ece.cmu.edu
Cc: allyn@cisco.com; csapuntz@cisco.com; howard.c.herbert@intel.com; Stephen Bailey; uri@broadcom.com
Subject: RE: Wot I `know' about COWS in hardware

Jim,

There are some things attractive about COWS -

1. the hard work - touching every data word has to be done only by the sender (on the normal path) and can be easily included in NIC with accelerator cards that seem to do a good job on the send side

2. If you are doing CRC or IPsec on a client in software there is no additional penalty (provided you can include the code in the right layer of software) as no data gets moved

3.It does not have to associated with a TCP packet alignment - and can work in face of TCP segmentation

Julo

"Jim Pinkerton" <jpink@microsoft.com> wrote on 17-12-2001 17:32:04:

>
> My main concern with this approach is that we could kill the product but
> win the spec wars. Specifically, this approach means that an
> end-customer has one of two choices in deploying the technology:
>
>    1) Upgrade both ends, and they'll see the full benefit
>    2) Upgrade only the server side, and see roughly 2-4 times the
> CPU
>       utilization on the client if their current
>       implementation is optimized
>       on the client side (a mere 2x if they are doing
> significant
>       receives that already require a copy, more like 4x if
> they
>       are primarily doing sends, which currently has no bcopy
> in
>       many OS implementations).
>
> This means that if they pick option 2) and their machines are CPU bound,
> that the data center capacity to handle requests will actually
> *decrease* if they deploy the technology. If the front end has enough
> idle CPU cycles, then they probably could select option 2).
>
> In my experience, we need to make sure we have a volume solution to
> enable NIC vendors to make enough profit to fund the next generation
> (otherwise RDMA/TOE is a one-shot deal and won't keep up with the CPU
> architecture). This means we need a path to the front-end boxes in the
> data center. My concern is that there is no half-measure in the above
> scenario - the IT manager must upgrade the thousand front-end boxes at
> the same time as they upgrade the back end, or deploy asymmetric configs
> where some front end boxes are upgraded. I'm not sure how attractive
> this deployment scenario is.
>
> Howard and Uri, can you comment on this issue?
>
>
>
> Jim
>
>
>
>
>
>
>
> > -----Original Message-----
> > From: Stephen Bailey [mailto:steph@cs.uchicago.edu]
> > Sent: Monday, December 17, 2001 7:13 AM
> > To: uri@broadcom.com; howard.c.herbert@intel.com
> > Cc: csapuntz@cisco.com; Jim Pinkerton; Julian_Satran@il.ibm.com;
> > allyn@cisco.com
> > Subject: Wot I `know' about COWS in hardware
> >
> > Hi,
> >
> > I haven't gotten a chance to do a full implementation yet, but here's
> > some architectural properties I believe to be true of a hardware COWS
> > implementation:
> >
> >   1) can be implemented `in line' on receive
> >   2) requires an MTU-sized RAM on send
> >   3) expected touches to send RAM is 2 per data word (just the `fifo'
> >      write and read ops, no editing), assuming the headers are merged
> >      on the outbound side.
> >   4) worst case touches to send RAM is 3 per data word (assuming every
> >      word must be edited)
> >   5) eliminates the need for the funky `make sure you don't send
> >      anything that's false positive under non-nominal conditions'
> >      behavior of the key/length proposal (I kinda doubted hardware
> >      impls were going to do this anyway, since it was a SHOULD).
> >
> > Basically, it looks OK to me.  Slowing the sender is much better than
> > slowing the receiver.  Theoretically, we could reverse the pointer
> > chain and allow in-line send, but RAM on receive, but that seems
> > stupid to me.
> >
> > It's clearly a design tradeoff whether you chose to use the COWS
> > send-side RAM for other purposes, or not.
> >
> > I'm hoping you guys can tell me whether you think this blows your
> > budget, or other noteworthy (unfortunate) properties.  As you can
> > tell, I have the utmost enthusiasm for the mission (Dr. Chandra...).
> >
> > Thanks,
> >   Steph