DRAFT Nashua interim meeting minutes

[Black_David@emc.com]

This is one loquacious bunch - sorry it took a while
to get these out.  WG consensus (will be considered
affirmed if not objected to on the list) is flagged
by **, action items are flagged by ->.

Comments, corrections, etc. to
the list by Friday, May 18th.

Thanks,
--David



IETF IP Storage (ips) Working Group 
Nashua Interim Meeting - April 30 and May 1, 2001
DRAFT Minutes

-----------------------------------------------------------
April 30, 2001 - FC encapsulation protocols (FCIP and iFCP)
-----------------------------------------------------------

FC Common Encapsulation - draft-ietf-ips-fcencapsulation-00.txt
-----------------------

CRC discussion - some protocols want to use a CRC, some
don't, hence there's a CRC Valid flag in the header.
Using one of the bits from the protocol number field to
encode CRC Valid was discussed and rejected as too clever by half

** Rough Consensus: Retain CRC Valid as a separate flag as currently
	defined.
-> Action: David Black to send email to list describing allocation
	approach that would (initially) avoid two protocol numbers that
	differ only in the encoded CRC Valid bit without permanently
	using two bits from the protocol number field to do this.

** Rough Consensus - Use different default TCP ports for FCIP and iFCP.
-> Action: David Black to pursue causing these ports plus one
	for iSCSI to be allocated.
-> Action: David Black will work with authors on revising the
	IANA text.

** Rough Consensus on the Reserved flag bits:
	(1) MUST be zero on send.
	(2) Protocol MAY specify MUST be zero on receive.
	(3) How to allow future protocols to make use of these?
		- Actual use requires modification of common encap.
		- Protocol-generic use of this encap SHOULD not
			assume that reserved flags are zero.

Later discussion on use of inverted check fields (<X> followed by
	1's complement of <X>) reached the following
** Rough Consensus: All uses of inverted check fields are to have
	the structure of a 32 bit word consisting of 16 bits of <X>
	followed by 16 bits of the 1's complement of <X>, where <X>
	may consist of multiple fields.  This applies to the common
	encapsulation header, and both the SOF and EOF encodings.

Fibre Channel MIBs - General Discussion
---------------------------------------

** Rough consensus - iFCP and FCIP will do separate protocol-specific
	MIBs.  Use existing FC MIBs (ipfc WG) for common stuff, similar
	to iSCSI's approach/direction of abstracting SCSI stuff into
	separate MIB.


FCIP MIB - draft-anil-fcip-mib-00.txt
--------

This is being developed as an extension of the FC Framework MIB
	(draft-ietf-ipfc-fsmgmt-int-mib-06.txt).

Several comments were made on things needing attention in the MIB,
	including clarifying the distinction between port and endpoint,
	and the likely need for RowStatus support for row creation.

TCP connection parameters are whether FCIP is configured to use them.
TCP statistics - aggregated across connections that constitute a single
	FCIP link.  This includes tracking of totals across closed TCP
	connections in a still-open link.

Some more global suggestions:
	- Beware of just duplicationg TCP parameters from other MIBs.
		Aggregation across TCP connections in an FCIP link should be
ok.
	- Look at RMON and follow its style to define an FCIP RMON MIB.

FCIP use of Common Encapsulation
--------------------------------

** Rough Consensus: FCIP will not use CRC.

The fact that this leaves the timestamp field unprotected
was characterized as acceptable because the probability of an error causing
a
frame that should have been discarded to not be discarded is very small due
to
the timestamp needing to fall within an approximately 40 second window to
cause
the frame to be forwarded.  40 seconds is more than enough for satellite
transmission - this won't allow FCIP to go to the moon, but that's probably
out of scope.

Protocol-specific fields for FCIP are not needed (protocol works fine
	if words 1 and 2 are zero), The decision on their use is:
** Rough Consensus: For FCIP, word 1 is a copy of 0, the first 16 bits
	of word 2 are reserved, and the last 16 bits are the 1's complement
	of the first 16 bits.  Consensus called over Charles Monia's
objection.

Back to protocol-specific data.  Bob S wants to use Word 1 as a copy
	of Word 0, and reserve Word 2 (reserved/-reserved)  (replicate
	3 into 2? - would consume all extra space).

-> Action: David Black will work with the FCIP authors to provide right
	Quality of Service (diffserv) words.

-- Model Update (Jim Nelson, Vixel)

Work in progress based on direction Mike O'Donnell presented in Minneapolis,
	update prior to T11 June meeting (week of June 4 in Minneapolis -
BB-2
	is after FC-SW-2 on Monday, afternoon).
Full model goes in FC-BB-2, a piece of this goes in FCIP to replace BSW
	(Backbone switch) discussion.  Full discussion when the text becomes
	available (August, except that London IETF meeting conflicts w/T11)

--> Action: David and Elizabeth are already in touch with the Area Directors
	regarding the IETF and T11 meeting conflict, and what to do about
it.
 
-- FCIP interaction with TCP

Changes made in -02 version of FCIP draft:
- Now distributing FC flows across multiple TCP connections (each port
	pair [N_Port login session] has to be limited to one TCP connection
	for in-order delivery, and FC ACKs need to be on same conn. as ACKed
	traffic).
- Removed discussion of TCP parameters.
- Error recovery (TCP keep-alive too large): 2 possible approaches
	-1- Fibre Channel ELS (Extended Link Sequence) to detect loss of
connectivity.
		This would not propagate across endpoints, and a zero-length
		frame might be better as that would not be a valid frame to
		forward into Fibre Channel.

-> Action: Bob Snively will check whether FSPF (Fibre Channel routing
protocol
	that will always be present on an FCIP link as FCIP is currently
specified)
	already contains heartbeats to detect a lost connection.  If so,
FCIP can
	leave this issue to FSPF to handle.

	-2- Check time stamps.  Added text to indicate that horribly late
		frames get bit-bucketed.  Common Encapsulation says "put
timestamp
		here", but doesn't provide much in the way of details

Resulting discussion was inconclusive.  FC experts need to write the
timestamp
rules to make sure that timestamp processing adheres to FC requirements -
these rules probably belong in the common encapsulation document as they
apply to any Fibre Channel traffic.  The following issues are open:
	(1) How do we ensure that the times at both ends of an FCIP or iFCP
		link are synchronized?  FC-GS-4's time protocol or requiring
		use of NTP across the link are possibilities.  FC B ports
can't
		source or sink FC-GS-4 frames, so may have to use NTP.
	(2) Need a concise and strict definition of when it's ok to *not*
		timestamp traffic.

Current text on loss of sync and related error recovery issues is incorrect
	in several ways and needs to be revised.  Revison *must* not be
susceptible
	to misinterpreting frame payload as a frame when sync is lost.

Multiple TCP connections
	- iFCP classifies by destination
	- FCIP can classify by traffic class (frame type), and also by
destination
	- FC class 4 is defining virtual channels (can have > 1 per
destination).

--> Action: David Black to run some version of the above summary past the
Area
	Directors to make sure this doesn't run afoul of general principle
of not
	using multiple TCP connections to increase a single application's
performance.

-- FCIP QoS and Performance

-> Action: David Black will check QoS text in Section 9.1 of -02 draft and
review
	first sentence of Section 9.2
-> Action: Authors will look at QoS text in iFCP draft.

The TCP performance section of the draft contains some
informational/advisory
	text on dealing with some TCP issues (e.g., long/fat networks
	[high bandwidth x delay product]).


Flow control management (Section 10)  This is about flow control mapping
	between FC and TCP.  Section title is misleading and may be changed.
-> Action: Remove reference to Fibre Channel Class 1.  T11.3 is removing
	support for it in FC-MI.

There are open issues in Section 10.  This is about reflecting TCP traffic
	propagation issues into FC credit-based flow control.  Source
throttling.

-- Data Integrity (Bob Snively, Brocade)

This was a discussion of detailed edits to the draft to make sure that there
	are no serious date integrity issues.

Section 3 error handling text will be deleted in favor of a pointer to 8.3.

Section 4.3 on ordering will be connected this to the multi-path
classification
	characteristics above.

Section 8.1 - Bob Snbively wants to add ability to do data-based resync

-> Action: Bob will write the text to explain how to do this for serious
	review.  Will also rewrite the text to distinguish use of another
	framing mechanism vs. the data scan.  Resync should be optional, and
	data scan must not be capable of misinterpreting what appears to
	be a Fibre Channel frame in an FCIP data payload as an actual frame.

Section 8.3 - additional text on dealing with CRC errors for frames going
	onto Fibre Channel.

Section 4.2 - use usual login procedure for fabric devices.  FCIP need not
	do some sort of TCP/IP logins, but FC mechanisms must work
unmodified.

Endpoint definition - distinguish Fibre Channel port from TCP connection
endpoint.
	

Naming and Discovery for FCIP
-----------------------------

-- FCIP and iSNS - Josh Tseng, Nishan

Hostnames will name FCIP boxes - discussed as possibility in Orlando,
presented
	as proposal at this time.  DNS or iSNS can be used to resolve them -
this
	is about tunnel configuration for FCIP.

Simple DNS approach - have to enter hostnames of remote FCIP devices in each
	FCIP device.  Each device resolves names with DNS and sets up
tunnels.
	All configuration is manual.

iSNS can automate configuration and discovery.  Domain concept makes
group-based
	managment of FCIP connectivity possible - scales better than manual
	configuration.  iSNS server can be discovered via DHCP option or
SLP.

FCIP box identifiers don't have to be DNS hostnames, but those are
convenient.
Can automate hostname assignment to FCIP boxes.  Management station can
	alias names.

-- FCIP use of SLP - Dave Peterson, Cisco [used Excel spreadsheet + MS Word
document]

iSNS and SLPv2 use same authentication.

Dave Peterson doesn't anticipate using SLP scopes for FCIP.
New SLP RFC (3082) provides sufficient (somewhat timely) notification
	of devices appearing and disappearing.

Proposal - static config, SLP for dynamic config, also iSNS, both are
optional.

DHCP - self-discovery mechanism, can't discover info about others.
DNS - too limited, SLP improves
LDAP - database interface, may be used by either SLP or iSNS

Putting together FCIP discovery model for SLPv2.

Suggestion - just use SLP, limit iSNS to functionality beyond SLP.

Josh:
RFC 3082 is Experimental, SLP scope is different from iSNS Discovery
domains.
SLP periodic re-registration has to be set correctly.
SLP can't store non-string attributes (e.g., X.509 certificates).
SLP is a good discovery mechanism, but not good for subsequent/sophisticated
mgt.

After much discussion, it was not possible to call consensus on the
relationship
	between iSNS and SLP usage/requirements for FCIP.

-> Action: Josh and Dave Petersen will post their positions to the reflector
	with David Black and Elizabeth summarizing areas of agreement and
open issues.

iFCP - Charles Monia, Nishan
-----------------------------

Changes to draft - iFCP addressing model updated, including how to manage
	address tables, and incorporation of transparent mode.
	ELS handling changed for common encapsulation.  No longer need
	extended header (this was discussed in Minneapolis).  Lots of
editorial
	changes. QoS text added, now 1 TCP/IP connection per N_Port session.

Next version targeted for May 15, major work is to incorporate common
encapsulation.

Changes to incorporate Common Encapsulation:

(1) Protocol-specific fields:
	- LS_CMD - identify command for Accepts of augmented ELS's.
	- iFCP flags (see below), copy of SOF, copy of EOF.
		iFCP uses these copies, and skips over SOF and EOF words.
	- CRC is always checked, CRCV field is ignored on receive, but
		must be set on send.

(2) Flags 
	- Compliance level (which version of document, watch out for
		version # change).  This is a 1-bit version number.
	- Augmented (part of an ELS w/augmented data)
	- Address transparent vs. translation mode
	- iFCP session control frame (must be non-augmented, translation)

Header CRC error causes connection close.  CRC error in 7-word header
	is unlikely, and only a single N_port to N_port connection is
affected.

(3) Timestamp format and content will be same as FCIP.  Likely to be able to
	put this syntax and semantics into the common encapsulation
document.

iFCP address-transparent mode.  This assembles iFCP gateways into a single
	logical fabric, within which addresses are global and untranslated.
	Maximum of 239 gateways in a fabric (this is the FC domain limit
	for number of switches, each gateway counts as a switch).
Address translation mode will probably be mandatory.  Translation and
	Transparent mode do not interoperate.  Transparent mode does not
	require any FC header or CRC changes.

iFCP transparent mode is similar to FCIP, but implements FC services
	via IP equivalents rather than using FC switches.
Gateways implement F or FL port, could even do an E port.  For E port,
	gateway MUST become the principle switch.

ELS modifications - out of 73 total ELSs, 14 of them, and 3 of their
	ACC (Accept) responses require iFCP intervention/changes.
-> Action: Authors to look at FCP-2 document for more ELSs.  REC now comes
from
	there, and SRR appears to be missing from the list.

One of the types of augmentation appends WWN to ELS sent between iFCP
gateways.
TPRLO (Third PaRty LogOut) can require a massive augmentation that can
span multiple FC frames due to the ability to to specify up to 4095 devices
to be logged out.

Gateways have to maintain state to track augmented Accepts and handle them
properly - this is for the gateway that proxies for target of ELS.  Exchange
identifier associates ACCs with ELSs on the FC side of that proxy.

Fabric Service Profiles.  This is about making sure that a iFCP-base
	fabric uniformly exports services to all attached devices, even
	if gateways are from different vendors.
	- Fabric services to be provided at well-known addresses (mandatory,
		optional, prohibited, behavior description for each
service).
	-  Transport (details of Class 2 and 3 behavior, prohibit Class 1,
		FC QoS mapping [when FC does QoS]).

iFCP Naming and Discovery - Josh Tseng
--------------------------------------

iSNS is required for iFCP - nothing else works.
Overview of iSNS use by iFCP.

Security for iFCP - use IPSec, probably tunnel mode.
FCIP is leaning in the same direction.

Security Comments - David Black
-------------------------------

These comments apply to all IPS protocols, and are particularly applicable
to the iSCSI security discussion scheduled for tomorrow.

Confidentiality is not required.  The IPS WG may have the freedom
to "profile" IPSec, e.g., make AH optional to implement.  The expensive
parts of IPSec are Confidentiality (cycles) and negotiation/key exchange
[IKE/ISAKMP] (code) - IKE/ISAKMP could be optional to implement (i.e.,
manual configuration, with attendant management problems).  One of the
ipsec WG chairs should be here tomorrow to provide additional information.

---------------------------
May 1, 2001 - iSCSI
---------------------------

iSCSI Naming and Discovery
--------------------------

-- Overview - Kaladhar Voruganti, IBM

-- Naming and Addressing - Mark Bakke, Cisco

WWUI acronym in earlier versions of draft gone, set of naming types
has been reduced to eui (WWN) and fqn.  fqn acronyn will be changed to
iqn (i = iSCSI) to avoid possibly rude (mis)-pronunciation

An iSCSI name is a persistent location-independent identifier, an iSCSI
	address is how one gets to it, with DNS recommended in preference to
	specifying the IP address.

There is a problem with domain names changing ownership without record of
what iSCSI names iqns were created by the old owner.  This risks
unacceptable
duplication.  Suggestions for using IANA private enterprise identifier,
date,
and some sort of existing vendor identifier were made.  The simple approach
of using WWNs runs into administrative difficulties like a $1500 fee to get
initial allocation.

-> Action: Authors to propose resolution to this issue.

Third party command authentication/authorization is an open issue.

-- SAM2 and iSCSI (concept mapping) - Jim Hafner, IBM

This is about how the concepts in the SCSI architecture (SAM2) map
to the concepts in iSCSI, and the implications of this mapping.

Basic concepts and mappings:
	o Network entity - has an IP network address (and TCP port)
	o iSCSI Node - within network entity, identified by iSCSI Name
		Contains at most ONE SCSI device (canonical iSCSI Target
		does not have a SCSI device).
	o SCSI Device Name (T10 is working on this) = iSCSI Node Name
	o SCSI Device - contains SCSI Ports = iSCSI Node Name + ISID or
TSID.
	o SCSI Port has one or more <IP address, TCP port> pairs, may change
		dynamically.
	o SCSI I_T Nexus = iSCSI Session, connects two iSCSI Ports.

Q: Multiple sessions per node pair?
A: Yes, each session has unique SCSI Port instance on each side.
	Result is that SCSI Ports are dynamic entities.  This is different
	from models used in other SCSI tranports because the SCSI Port
	entities are bound one-to-one by a connection.

The iSCSI Name is intended to name OS image (actually
	SCSI instance in OS image) - goal here is to avoid problem with
	FC Node Name, which is associated with HBA - iSCSI Node Name is
	supposed to be associated with the "SCSI entity" in the OS.
	This requires the SCSI instance to manage ISIDs or TSIDs globally
	across all iSCSI interfaces.

SCSI Port Address - not clear what this T10 concept maps to, could
	just use SCSI Port Name (iSCSI name + ISID or TSID).

Mapping Consequences:
	- ISIDs have to be unique initiator-wide (across all NICs/Adapters
connected
		to same target).
	- Persistent reservations get linked to both ISID and TSID --
reservation
		comes back only if initiatiator reuses ISID and target
reuses TSID.
		This has implications on target saving/tracking of state,
and requires
		initiator to remember which ISID was used with which target.

-> Action: Next version of draft will need a table of how nexus state reacts
to
	events (e.g., Resets).

Discussion of Persistent Reservation behavior wrt this model.  SCSI has
things
	to say about Target reboot.  Host reboot is generally not visible,
but
	only exception is that if addresses are reassigned (e.g., by Fibre
Channel
	fabric), Target has to cope with this reassignment.

Names span sessions to make authentication simpler.  Keeping ISID separate
	from the name makes the authentication stuff simpler.

Q: Do persistent reservations open up a denial of service attack route?
A: No - can't do this without prior authentication, and there are ways
		to blow away reservations from other initiators.
	Difference from FCP is that iSCSI persistent reservations are keyed
		to session IDs vs. FCP keying them to FC ports (via WWN).

SCSI allows sharing of reservations.  At the iSCSI level, each reservation
	is tied to a session.  SCSI reservation sharing is how reservations
get
	shared across more than one I_T_Nexus.  T10 is considering
reservation
	semantics that ignore ports and make reservations at SCSI Device
level,
	which would yield reservations at iSCSI node granularity.

On session establishment, TSID choice may be influenced by existence
	of reservation state - this results in iSCSI having to check for
	SCSI persistent reservations for that initiator name and ISID.

Third party reservations were brought up.  They're volatile, essentially
obsolete, and not needed by anything in SCSI, included extended copy.  There
are also command format problems that would make it very difficult for iSCSI
to pass a node name + session ID to the third party copy device.
 
Persistent reservation key *does not* contain iSCSI node name or ISID.
Cluster
	software (or whatever) creates its own reservation keys.

--> Action: There may be an issue of scope of ACA with respect to the model.
	Jim Hafner will check into this.  CA should not be an issue because
	iSCSI requires Autosense.

Access controls need to use iSCSI Initiator node name.  T10 will handle
	the required changes, as part of long identifier format changes
	(e.g., for things like SCSI alias designation), which also cover
	third-party commands.  Additional address info for third-party
	commands could be definitive, or just hints.  SRP and FCP use
	definitive addresses.

-- Discovery - Mark Bakke, Cisco  draft-ietf-ips-iscsi-name-disc-01.txt

Review of types of configuration:
	- Static/manual
	- SLP for simple discovery
	- iSNS for centralized management
"Send Targets" - an additional mechanism that can be used in all three
cases.
	Send Targets is allowed to send info on targets that are elsewhere,
		or just targets behind its port.  This can also implement
		some "soft zoning" (i.e. restricted what an Initiator can
see),
		and cascading (return a target that is elsewhere to which
		"Send Targets" must be sent).

If there's only one iSCSI target behind a port, a login to the "iscsi"
target
	could return that name and immediately allow commands.  This would
	lead to needing to distinguish between login only for "Send Targets"
	and login that results in the one and only target coming back and
	going into full-feature mode.  It adds complexity but might simplify
	booting.  OPEN ISSUE.

There is a need for some sort of iterator interface or something else that
	avoids an Initiator having to allocate potentially unlimited space
to
	hold the results of Send Targets.  This issue will be taken to the
mailing
	list.  OPEN ISSUE.

Will be adding tags to addresses that specify which addresses can be
	aggregated into a session.  Absence of tag indicates that
aggregation
	is not possible.  Tag unique to a single address would indicate that
	multiple connections to that address are aupported.  Only semantics
	of tag is whether these values match, and context is within a single
	response to Send Targets.  One tag per address for now.

The ability to specify another initiator in Send Targets will not be
	supported, as there is T10 work to be done in the access control
	area (this is about giving an Initiator's view to
	a third party copy device) and it raises security issues.

-- SLP - Mark Bakke, Cisco 	draft-ietf-ips-iscsi-slp-00.txt

SLP template has been submitted as WG draft.
SLP and iSNS now do *exactly* the same authentication.

Working on interoperability with iSNS.
Working on host/device taxonomy to make recommendations about what should
	be implemented where (SLP and/or iSNS in drives, arrays, hosts,
	gateways, etc.?).

Open source for SLP authentication is now available.
Need to look at RFC 3082 for event propagation to see what it does and
figure
	out how to use it.  iSCSI may need a few minor extensions to this.
The iSCSI SLP Template may need changes to better accomodate copy managers.

None of the discovery mechanisms are REQUIRED at this point in time.
A question was asked about how network management tools discover network
	configuration and devices - this is generally based on MIBs and
	reading information from them.  One can have a conformance statement
	for a MIB that mandates a relatively small portion of it that is
	necessary for this sort of discovery.

-- iSNS - Josh Tseng, Nishan  draft-ietf-ips-isns-02.txt

SNS requirements in naming and discovery draft are based on FC-GS-3.

Discussion of iSNS functionality and alternatives:

- LDAP: This is a directory service, but seems to lack info about when and
where
	to send state change notifications.
- SNMP: Discovery domains cannot be implemented, cannot do complex queries.
- SLP: Now has a notification mechanism, but needs some changes.  SLP
	depends on multicast (in absence of SLP, have to manually configure
	addresses of SLP directory agent).  DHCP can discover SLP, hence
	can piggyback on DHCP infrastructure.  SLP scopes aren't the same
	as discovery domains - have to add something.  SLP requires on
	service agents to reregister with directory agents, iSNS does
	things in the reverse direction.

Nishan will open source iSNS client and server within next month.

iSCSI Requirements Last Call Issues - Marjorie Krueger, HP
-----------------------------------

This discussion was about closing the issues raised during informal Last
Call of
	this draft on the mailing list.

** Rough Consensus: iSCSI should not be making changes to SCSI-3, except
where the
	relevant SCSI document says something is "transport dependent".
Wording will
	be crafted to require "minimum changes" or something like that
outside of
	this case.

** Rough Consensus: Layering wording will not be added to this document; the
	exhortation in the IPS WG charter is sufficient.
** Rough Consensus: "packet formats similar to" wording will NOT be added.

** Rough Consensus: Accept Bob Snively's change to require ordering only
	at I_T_L level, not I_T, but don't exclude implentations that do I_T
ordering.

After a long discussion of ordering, the conclusion was:
** Rough Consensus: Strictly ordered command delivery to target is required
across
	a session, even in error recovery cases (this is stricter than
either FCP or
	parallel SCSI require).  Those wanting looser ordering should use
multiple
	sessions.

Rough consensus on the last point was called over visible opposition
(significant
	minority of the room).
	
Security
--------

-- David Black (EMC, co-chair) presentation on security requirements

See slides - this was an informational presentation/discussion about
requirements
	(MUST implement authentication and cryptographic integrity, MUST
have a
	required mechanism for each, confidentiality is OPTIONAL, IP vs.
iSCSI
	level mechanisms [iSCSI can multiplex initiators and targets on a
single
	<IP address, TCP port> pair).

-- Ofer Biran (IBM, Haifa)

Lots of slides - see slides for details.

After much discussion ...

** Rough Consensus:
(1) Need iSCSI level authentication in addition to what is done for IP or
TCP/IP.
(2) ESP with null authentication is mandatory to implement
	- TLS was considered and rejected as part of this.
(3) SRP is mandatory to implement.

Beyond this, the use of SRP (or another inband authentication mechanism)
	to key ESP was discussed as a promising direction.  It's premature
	to require this - the minimum requirement for now is ESP with
pre-shared
	keys and a separate implementation of SRP.  

-> Action: David Black, Ted Ts'o, and Ofer Birman will write 
	up details of getting ESP key material from in-band authentication
	mechanism and related "Start Here" text key to turn on ESP
dynamically.

Error Recovery - Randy Haagens, HP and Kalman Meth, IBM
-------------------------------------------------------

NOTE: Due to the late hour and diminishing attendance, no consensus calls
	were attempted in this area.

This is a progress report from some extensive and on-going off-line work
	on details of errror recovery.  The goal is to provide a set of
	mechanisms that allow implementations to choose where they want
	to be between "Trust TCP Explicitly" (Detect errors and flag
	to SCSI) and "Can't Trust TCP" (comprehensive error recovery.
	This work will be folded into the basic iSCSI document.

Session recovery appears to be cleanly separable from finer-grained
	mechanisms.  Not sure about separations among within-session,
	within-connection, and within command recovery.

Not every situation will need fine-grained recovery.  Session recovery
	will be mandatory - close session and open a new session.  ** Need
	to make it clear that explicit logout on last connection in session.

Within-session recovery reissues commands on another connection in
	same session - login of new connection includes instruction
	to logout old connection.  Have to make sure that old connection
	is successfully logged out before issuing new commands with
	retry (X) bit set.  This is optional.  Have to use same tags
	and CmdSNs in doing the retry.

Terminology note: This is really "resend".  Whether or not a "retry" of
	the command is actually done is at the device's option.  Need
	to be careful to distinguish these terms.

Comment: Wording is needed to indicate that if the initiator
	chooses not to reissue a command that was pending on the
	terminated connection, it MUST terminate the session.

Q: Is having both implicit and explicit logout mechanisms overkill?
A: Needed to cover resend on an existing connection vs.
	opening a new connection for this purpose.  Target can
	reject attempt to open new connection, and then initiator
	has to clean up.

Comment: This portion of the draft contains a recommendation
	to use Target Reset that needs to be removed.

Within-connection recovery reissues command on same connection to fill
	CmdSN holes.  Has to be on same connection due to connection
	allegiance.  These errors require framing support over TCP -
	in the absence of framing, the connection has to
	be terminated.

Comment: When there are multiple connections, target needs
	to delay for a bit before telling Initiator that commands are
missing.
	On a single connection, this is easy.  When CmdSN has advanced past
	the hole on all connections, target knows that a command is missing.
	A request was made for an explicit way to request a resend rather
	than the implicit monitoring of ExpCmdSN by the Initiator and
possible
	use of NOP by the target.  This will be considered.

Discussion of (lack of) trust in TCP.  Between ESP at IP
	level and encouraging SCTP to use a CRC, we're getting stronger
	integrity in the stack at a lower level.  TCP (or SCTP) will retry
	is a wonderful answer to a whole bunch of problems.  This lead to
	some interest in figuring out how to shim a CRC between TCP and IP,
	which would then allow at least within-connection and within-command
	error recovery to be dropped.  OPEN ISSUE: for further discussion. 

Discussion of data resends differing between reads and writes.  On read,
initiator resends command causing target to resend all the data.  On write,
the target can issue only some of the R2Ts (for the missing data) and it
should not be a big deal for the initiator to cope.  OPEN ISSUE - will
require confirmation/further discussion on list.

Status recovery within connection is based on status SNACK.

-> Action:  Need to revise text to indicate that SNACK is optional.  SNACK
is
	required to do within-connection recovery of status.

Comment: This may be a big deal for backup - put in a negotiation
	key/value pair to allow initiator to discover whether target
	does SNACK.  Backup may want to avoid using a target that doesn't
	do status SNACK.  OPEN ISSUE: Take to list.

Within-command recovery from dropped data.  If data CRC fails, issue
	immediate Data SNACK.  If header CRC fails, find the hole, issue
	Data SNACK at that point.