|
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] Re: iSCSI/iWARP drafts and flow controlOn Saturday, July 26, 2003, at 03:57 PM, Mike Ko wrote: > In iSER, we expect the flow control to be regulated by the Command > Numbering mechanism in iSCSI. In other words, since the queuing > capacity > of the receiving iSCSI layer is MaxCmdSN - ExpCmdSN + 1, the receiving > iSER layer can use this information to determine the minimum number of > untagged buffers. In addition, it needs to provision a sufficient > number > of untagged buffers to allow enough time for the iSER layer to respond > to > incoming immediate commands, asynchronous messages, etc., and replenish > the buffers. The use of a buffer pool shared across multiple > connections > will allow the iSER layer to replenish the buffers on a statistical > basis. > iSCSI does indeed provide control for untagged messages that have CmdSNs. The issue relates to Asynchronous Messages. Is there any protocol restraint on the number of Asynchronous Messages that can be sent? The only one that I could find is in section 3.2.2.2 where it is limited to an "implementation defined constant that MUST NOT exceed 2**31-1". You cannot know an implementation defined constant of your peer, unless the protocol explicitly communicates it. So the only ULP constraint on the number of in-flight Asynchronous messages is 2**31-1. That constraint already existed in iWARP (the Message Sequence Number has the same need to determine which of two 32 bit numbers that wrap-around is "later"). This is an important change from iSCSI over a conventional transport protocol. The asynchronous messages would be flow controlled by the transport itself. The Data Sink is expected to advertise the amount of in-flight data that it is willing to accept. It is important to separate buffering from flow control. Flow control deals with what is legal to send. Buffering is strictly a local issue. Even with conventional transports, there is no requirement that every buffer byte advertised on a TCP connection or SCTP association be pre-allocated and reserved in advance. How a host manages its receive buffers is a local issue. The protocol issue is what it tells its peer that the peer is authorized to send. Aside from the theoretical issues (such as definitions of "reliable transport" and "flow control") there is a very nitty gritty issue that buffer pools are an *optional* feature even under the RDMAC proposed verbs. You also cannot "share" resources across connections if there is only a single connection to deal with. To achieve the goal of allowing iSER to work on a generic RDMA card it must work with a *simple* Receive Queue. That requires that the Data Sink be able to predict the maximum number of untagged messages that can be in-flight from a peer that is following the protocol. This actually would not be that hard to deal with. Several of the Asynchronous Messages are inherently unique. There is no reason to send two messages announcing the intent to terminate a connection over a reliable connection. If you simply add a constraint on the number of Asynchronous Messages that may be sent without confirmation that they have been processed there is no problem. iSER becomes a reliable protocol. Peers are not required to estimate how fast the other side can process messages. (And, to hit the theoretical point, if you have to *estimate* how fast your peer operates then you are no longer using a *reliable* protocol). There are already "credits" for CmdSN-bearing untagged message. The methods for establishing and modifying these credits already exist in iSCSI so there is no need for iSER to do anything other than be aware of the value. iSER then has to factor in additional credits for Asychs. This could be a ULP specified constant, or it could be negotiated just as is proposed for RDMA Read credits. Credits would be drained by sending. All that is required is the addition of of a rule on how credits are restored. I would suggest using reply to a *later* CmdSN and/or a command that explicitly waits for all prior Asynch Messages to have been processed.
Home Last updated: Tue Aug 05 12:46:09 2003 12771 messages in chronological order |