Re: header and data digest issue

["Bob Mastors" <bmastors@aciesnetworks.com>]

IPsec ESP cryptographic integrity is too expensive (at least for me).

It seems likely that a iSCSI aware middle box will DMA the header
and data separately. If this is the case than it seems likely that
eventually the data will be DMAed to the wrong place.
This could be caused by software, firmware, or hardware.

It also seems likely that a software error in the target operating system
will also cause this problem.
Again because it is likely that the header and data will be manipulated
differently.
It is a very minor coding error to cause this behavior. Just mess up a circular
queue, or some off by one bug.

I am writing software that processes and stores iSCSI data.
I will have the iSCSI target (whether software or off-board processor) pass me
the CRC digests in addition to the header and data.
The CRC digests are verified and stored on disk with the header and data.
This provides a high degree of confidence that the data on disk is either correct
and at its correct address, or that it can be detected that it is not.
Additionally this architecture allows fsck type software to be written to
walk the disk looking for CRC errors.

By using the CRCs generated by the initiator and verified by my software, I have
a high degree of confidence that my own software or the operating system
software did not introduce errors.

The iSCSI header and data digests are almost perfect for implementing this
architecture. But by computing CRCs separately for the address and the data,
there is a non-negligible chance that my software will write the wrong data
to a block. Not only will I be unable to detect the problem when it occurs,
a scan of the disk and CRCs will indicate that nothing is wrong.
Only the application will know that the data is wrong.

You are correct to point out that there are all sorts of errors brain damaged
middle boxes could introduce that CRCs will never solve.
But I think that this is a very simple bug to cause.
All it takes is an address line in a DMA being held for just a bit too long
once a month.

I hope you can see from the explanation that to me this is not a theoretical
problem.

Bob

----- Original Message ----- 
From: <Black_David@emc.com>
To: <bmastors@aciesnetworks.com>; <ips@ece.cmu.edu>
Sent: Wednesday, February 27, 2002 12:47 PM
Subject: RE: header and data digest issue


> > I think there is a problem with the current design of the
> > header and data digest.
> > There is a non-zero chance that an error could cause
> > the target to process a single pdu that has a header from
> > one pdu and data from a second pdu.
> > 
> > This type of error is undetectable by the target with the CRC32C
> > digests turned on.
> > 
> > For example:
> >         Initiator sends pdu 1 which contains header H1 and data D1
> >         Initiator sends pdu 2 which contains header H2 and data D2
> >         Target receives pdu 1 which contains header H1 and data D1
> >         Target receives pdu 2 which contains header H2 and data D1 (bad)
> > In this example the header and data digests will checksum correctly.
> > However the wrong data will be written to addresses indicated in H2.
> > 
> > There are three obvious places this error can occur:
> >         1. the initiator
> >         2. an intelligent router/switch
> >         3. the target
> > 
> > Not much can be done about initiator problems.
> > But I would like to detect problems introduced by the other areas.
> 
> The middlebox problem is inherent in allowing middlebox functionality
> which was the rationale for using separate CRCs.  One can stop a middlebox
> from making this mistake by running IPsec ESP's keyed cryptographic
> integrity
> check end-to-end - if some middlebox tries to get clever, the ESP integrity
> check fails and the packets that make up the corrupted PDU are dropped; if
> the middlebox error is persistent, the TCP connection will probably close.
> Changing the CRC check won't stop middleboxes - if some CRC gets in the way,
> the middlebox will strip the offending CRC and regenerate it - the result
> is *still* vulnerable to implementation bugs in the middlebox (and cleverer
> things like putting the data digest in the header or vice versa are 
> vulnerable to similarly cleverer implementation bugs).
> 
> > I think it is actually likely that my target software will run
> > in an environment that produces this type of error.
> > Imagining how an intelligent router/switch or off-board iSCSI target
> > hardware might operate, it is probably only a single bit logic error
> > that could cause this type of error.
> 
> If "intelligent" means the router/switch is iSCSI-aware, ESP cryptographic
> integrity is the right means to protect against modifications - anything
> less (especially omitting the keying) is an invitation to the "strip the
> CRC and regenerate" scenario akin to the one in the middlebox paragraph
> above.
> 
> If an iSCSI target is off-board, it has stripped, parsed, and discarded
> the iSCSI header - no CRC of any form will protect against a bug in that
> target that got confused about which data went with which header.  Protocol
> implementations remove headers from data and have to keep track of what
> they're doing - there's no free lunch here (e.g., if a CRC covering header
> and data is added, an implementation will likely to check that and strip it,
> after which it can still get confused.
>  
> > Maybe it is not appropriate to address this problem in the iSCSI spec
> > because I have described a problem that would not occur on the wire.
> > However iSCSI is the mechanism that allows more hardware and software
> > to be interposed between the SCSI initiator and the SCSI target.
> > iSCSI should not allow a single bit logic error in this additional
> > hardware and software to produce an undetectable error.
> 
> The only problem scenario I see involves middleboxes, and iSCSI
> provides a tool (IPsec ESP cryptographic integrity) that addresses
> that problem.  Did I miss anything?
> 
> Thanks,
> --David
> 
> ---------------------------------------------------
> David L. Black, Senior Technologist
> EMC Corporation, 42 South St., Hopkinton, MA  01748
> +1 (508) 249-6449 *NEW*      FAX: +1 (508) 497-8500
> black_david@emc.com         Cell: +1 (978) 394-7754
> ---------------------------------------------------