RE: sector alignment for DataOut PDUs?

["Martin, Nick" <Nick.Martin@compaq.com>]

Pat,

Are you considering a bridge between iSCSI initiator and FC target, or the reverse?

For the former case, the bridge is an iSCSI target.  The buffer size reported to the iSCSI initiator as DataPDUAlignment would be one which simplifies command and memory management within the bridge.  If the bridge can conveniently manage its memory in 1 byte increments, then report 1 byte which does not restrict the initiator.  If it prefers 1024 byte increments, it can send that to the iSCSI initiator.

For the latter case, the bridge is an iSCSI initiator.  Perhaps the iSCSI target has requested DataPDUAlignment such as 1024 bytes, but the FC initiator sends unaligned data, or perhaps 512 byte data.  This is the same situation the iSCSI target is in with no data alignment in iSCSI.  Data has arrived, but nothing can be done with it until more data arrives to fill the next transfer unit.

This is perhaps acceptable for some store-and-forward "bridge" designs, but rules out some "switch" or other low latency pass-through middle device designs.  When the alignment requirement can not be enforced for the FC or other protocol device on the other side, store-and-forward buffering would be required.

So we come to, DataPDUAlignment as a requirement is good for some designs and bad for others.  I believe that most initiators, even without this parameter=value, will break data PDUs on convenient boundaries.  If an initiator does not do so, the target needs to do the bookkeeping.  When the target implements bookkeeping and the initiator sends aligned data, the target operates efficiently.  When the initiator does not send conveniently aligned data, the target must not fail.

DataPDUAlignment could be advisory in nature.  The target advises the initiator of the transfer increment which will be optimal for the target.  When the initiator can not (or does not) break its transfers on these boundaries, the target may be less efficient.  No requirement is removed from the target, so no development work is saved developing or testing it.  The target is required to accept data PDUs of any otherwise legal length.  No new requirement is imposed on the initiator, nor on a bridge excepting to accept this parameter.  However, iSCSI initiators can now be given enough information to enable them to avoid unaligned data transfers to the target.  Targets will be able to operate at maximum efficiently when the initiators utilize this information.

If it needs to be advisory in nature, therefore optional to implement, I suggest that it could be tabled so as not to interfere with progress toward iSCSI 1.0.  On the other hand, being optional to implement, only required to understand, it is a small impact to implementations currently underway.

As someone else mentioned, there are targets whose block size is not a power of two.  Without such a parameter=value, aligned transfers to these targets could be rare. 

Thanks,
Nick


> -----Original Message-----
> From: THALER,PAT (A-Roseville,ex1) [mailto:pat_thaler@agilent.com]
> Sent: Monday, March 04, 2002 11:34 AM
> To: Robert Snively; Paul Koning
> Cc: ips@ece.cmu.edu
> Subject: RE: sector alignment for DataOut PDUs?
> 
> 
> All,
> 
> Putting in a requirement for sector alignment will make 
> implementation of
> bridges difficult. I've been looking through the Fibre 
> Channel specs and I
> cannot find any requirement that transfers be in block size 
> multiples. If we
> put a sector alignment requirement in, then bridges would 
> have to buffer
> data that arrived in a non-multiple until further data 
> arrived to add to it.
> They would also have to keep an accounting to know whether a 
> transfer was
> the last and therefore should be allowed to go even if it 
> isn't a multiple
> of block size. This would complicate bridging excessively.
> 
> Regards,
> Pat
> 
> -----Original Message-----
> From: Robert Snively [mailto:rsnively@Brocade.COM]
> Sent: Monday, March 04, 2002 7:29 AM
> To: Paul Koning; rsnively@Brocade.COM
> Cc: ips@ece.cmu.edu
> Subject: RE: sector alignment for DataOut PDUs?
> 
> 
> > > There has always been a feeling that it was nice to do things
> > > on convenient boundaries, including memory page boundaries and
> > > device physical block boundaries, but SCSI has long since 
> > > elected to perform any operation on almost any boundary.  
> > SCSI drives
> > > and related operating systems have commonly used block 
> sizes of 512
> > > bytes and 520 bytes and less commonly of other values.  
> > > 
> > > The only requirement we were able to enforce in previous
> > > SCSI protocols is that all but the last PDU of a transfer
> > > were required to be on 4 byte boundaries.  We were also
> > > able to enforce the maximum values and a requirement that
> > > the transferred data count exactly match the requested byte
> > > count.
> > 
> > I don't see that last point.  Certainly not for unsolicited data --
> > and for R2T I can find no stated requirement to send exactly the
> > requested count either.  Or did you mean the total for the entire
> > operation as opposed to the individual bursts?
> >  
> 
> I haven't reviewed all of iSCSI's rules with respect to this
> particular issue.  There appears to be an assumption of 
> non-congested processor-sized buffers that allow unsolicited
> write data transfer.  That may be a convenient assumption for
> some devices, but a problem for those that want more control
> of their buffering.  In general, the requests for write data 
> are part of the logical unit's write buffer management
> and data striping management.  If you don't supply
> exactly what is asked for, you foul up the buffer management of the
> target, sometimes causing pathological performance problems.
> 
> > > I would strongly suggest that you not bother to enforce any
> > > other boundary restrictions.
> > > 
> > > I believe Eddy was right when he asked:
> > > 
> > > 	Given that the TCP issues of reassembly 
> > > 	are so much more complicated than
> > > 	the concatenation issues necessary for 
> > > 	target data alignment, is there
> > > 	really a problem here?
> > > 
> > > My answer to him would be, no, there is not a problem.
> > 
> > I answered it before when Eddy said that, but I'll 
> summarize it again:
> > I'm not talking about a problem, I'm talking about an 
> opportunity for
> > simplifying the target and making it more efficient.
> > 
> > Given alignment, each iSCSI PDU that carries data can be 
> sent to disk
> > by itself, because it corresponds exactly to one or more 
> disk blocks.
> > Without alignment, doing writes when the data arrives is still
> > possible, but it clearly adds complexity because PDU 
> boundaries don't
> > match disk block boundaries.
> > 
> > I made the proposal because it clearly helps the target and 
> appears to
> > add no significant burden on the initiator.  The feedback to date
> > indicates that this is indeed the case.
> > 
> > Are you saying "I don't care one way or the other" or are you saying
> > "I feel this is a bad idea because it creates problems you haven't
> > thought of"?  I'm reading your comment as the former; if 
> you meant the
> > latter, could you elaborate?
> 
> I think I am saying that I care, and that it is an idea that
> will trap you into limiting simplifications.  In general, the 
> initiator's data flow engine has no knowledge of the logical 
> unit's physical block structure.  In addition, the page structure
> from which you are obtaining data in the host will usually 
> have boundaries
> that are unrelated to the logical unit's physical block structure.
> While it is perfectly possible to align PDU's with blocks, it
> requires SCSI ULP information to be forced in to the lower level
> data transfer hardware.  This is a significant inconvenience and
> will cost performance and/or complexity.  Note that the blocks
> in a logical unit are not all required to be the same size, and in
> tape drives and some optical devices are typically not the same size.
> 
> I still think it is best for the logical unit to ask for what 
> it wants,
> then expect to get it.  The logical unit is the only device in
> the system that really knows what it needs and when it needs it.
> That is one of the important SCSI principles, allowing the 
> logical unit
> to take over all the media dependent functions from the initiator
> and the host programming.
>