|
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] Re: iSCSI/iWARP drafts and flow controlFrom sec.3.5.2.1 of the iSCSI spec, "Asychronoous Messages are used to carry SCSI asyncrhonous events (AEN) and iSCSI asynchronous messages." Sec.10.9.1 lists the codes for Asyncrhonous Messages which include 0) SCSI Asynchronous Event, 1) target requests Logout, 2) target indicates it will drop the connection, 3) target indicates it will drop all connections of this session, and 4) target requests parameter negotiation. Since the iSER layer can estimate the number of Asynchronous Messages needed in a typical situation for the AsyncEvent codes listed in the iSCSI spec, it can determine how much to overprovision when a dedicated receive queue is used. Additionally, if one is concerned about the atypical case, the optional features of graceful handling and the shared receive queue as you have pointed out can be used. So it doesn't seem to me that flow control for Asynchronous Messages are needed. Can you provide an example where an unbounded number of Asynchronous Messages would need to be sent? Mike Ko IBM Almaden Research San Jose, CA 95120 To: Mike Ko <mako@almaden.ibm.com> cc: ips@ece.cmu.edu Subject: Re: iSCSI/iWARP drafts and flow control On Saturday, July 26, 2003, at 03:57 PM, Mike Ko wrote: > In iSER, we expect the flow control to be regulated by the Command > Numbering mechanism in iSCSI. In other words, since the queuing > capacity > of the receiving iSCSI layer is MaxCmdSN - ExpCmdSN + 1, the receiving > iSER layer can use this information to determine the minimum number of > untagged buffers. In addition, it needs to provision a sufficient > number > of untagged buffers to allow enough time for the iSER layer to respond > to > incoming immediate commands, asynchronous messages, etc., and replenish > the buffers. The use of a buffer pool shared across multiple > connections > will allow the iSER layer to replenish the buffers on a statistical > basis. > iSCSI does indeed provide control for untagged messages that have CmdSNs. The issue relates to Asynchronous Messages. Is there any protocol restraint on the number of Asynchronous Messages that can be sent? The only one that I could find is in section 3.2.2.2 where it is limited to an "implementation defined constant that MUST NOT exceed 2**31-1". You cannot know an implementation defined constant of your peer, unless the protocol explicitly communicates it. So the only ULP constraint on the number of in-flight Asynchronous messages is 2**31-1. That constraint already existed in iWARP (the Message Sequence Number has the same need to determine which of two 32 bit numbers that wrap-around is "later"). This is an important change from iSCSI over a conventional transport protocol. The asynchronous messages would be flow controlled by the transport itself. The Data Sink is expected to advertise the amount of in-flight data that it is willing to accept. It is important to separate buffering from flow control. Flow control deals with what is legal to send. Buffering is strictly a local issue. Even with conventional transports, there is no requirement that every buffer byte advertised on a TCP connection or SCTP association be pre-allocated and reserved in advance. How a host manages its receive buffers is a local issue. The protocol issue is what it tells its peer that the peer is authorized to send. Aside from the theoretical issues (such as definitions of "reliable transport" and "flow control") there is a very nitty gritty issue that buffer pools are an *optional* feature even under the RDMAC proposed verbs. You also cannot "share" resources across connections if there is only a single connection to deal with. To achieve the goal of allowing iSER to work on a generic RDMA card it must work with a *simple* Receive Queue. That requires that the Data Sink be able to predict the maximum number of untagged messages that can be in-flight from a peer that is following the protocol. This actually would not be that hard to deal with. Several of the Asynchronous Messages are inherently unique. There is no reason to send two messages announcing the intent to terminate a connection over a reliable connection. If you simply add a constraint on the number of Asynchronous Messages that may be sent without confirmation that they have been processed there is no problem. iSER becomes a reliable protocol. Peers are not required to estimate how fast the other side can process messages. (And, to hit the theoretical point, if you have to *estimate* how fast your peer operates then you are no longer using a *reliable* protocol). There are already "credits" for CmdSN-bearing untagged message. The methods for establishing and modifying these credits already exist in iSCSI so there is no need for iSER to do anything other than be aware of the value. iSER then has to factor in additional credits for Asychs. This could be a ULP specified constant, or it could be negotiated just as is proposed for RDMA Read credits. Credits would be drained by sending. All that is required is the addition of of a rule on how credits are restored. I would suggest using reply to a *later* CmdSN and/or a command that explicitly waits for all prior Asynch Messages to have been processed.
Home Last updated: Tue Aug 05 12:46:09 2003 12771 messages in chronological order |