RE: iSCSI: ISID issue summary

[Black_David@emc.com]

Rob,

> As Nick noted, the problem is where do you store these ISIDs?
> For that matter, where do you store the iSCSI name? [..snip..]
> With iSCSI you have no guarantee of hardware.

Software-based implementations will have access to a file or
registry or something like that.  Boot involves BIOS nonsense
and pain of varying forms and has access to BIOS parameter
store (may be battery-backed RAM rather than flash) if needed.
Booting over a software implementation can be very difficult
- in essence, the iSCSI software gets added to the main system
BIOS (not something to be done lightly, especially as there
are existing protocols that can retrieve a boot image over IP).

> For the SCSI RDMA Protocol for InfiniBand, we chose:
>   initiator port name = 64-bit worldwide identifier from
>                         the InfiniBand channel adapter plus
>                         64 extra bits
> All software using the same InfiniBand channel adapter have to
> coordinate usage of the extra bits, but single-OS single-driver
> systems can just fill them with zeros.

But SRP only needs to coordinate within "the same InfiniBand channel
adapter".  iSCSI currently requires the ISID to be coordinated across
adapters/network interfaces.  That's an important difference.

> Persistent reservations, access controls, extended copy,
> third-party XOR commands, and the supporting alias commands
> are all potential users of port names today.  Protocol bridges
> and management tools may also benefit from using names 
> rather than addresses.

I'll take that as a strong hint from the T10 direction that this
set of issues cannot be deferred (matching Jim's comments).

> Are the iSCSI name and ISID used as part of authentication?
> How does a device driver prove it has the right to use
> a certain iSCSI name/ISID?  How does it prevent some other
> software from using its own iSCSI name/ISID?

Only the iSCSI name is used, and iSCSI inband authentication
(which requires state on the target) is used to deal with both
of the latter questions.

> If we didn't have ISIDs we'd still have the same debate about
> managing the iSCSI names.  Two software implementations
> on the same machine could choose the same iSCSI name and
> conflict - the same problems result.

Actually, the iSCSI name has to be statically configured -
it's somewhat akin to an IP address in that using the same
one twice on two different independent systems will have
unintended (and potentially unpleasant) results.  Picking
the name dynamically is not anticipated because that name
is expected to show up in the configuration of LUN masking/
mapping access controls on the Targets.

> Jim's partitioning lets the OS pick an iSCSI name (unique 
> from any other OS on the fabric, with any luck) and requires 
> the OS coordinate ISIDs for any iSCSI drivers sharing
> that iSCSI name.  A dedicated iSCSI HBA could also hand
> out ISIDs for drivers using it.  

With the exception of having "the OS pick an iSCSI name",
that's correct.  The problem is one system with two HBAs
that don't know about each other.  Also, unlike InfiniBand,
one can expect an iSCSI HBA to have a dedicated driver.

> This should limit the 
> scope of conflicts to one system rather than the whole 
> fabric.  It'd be helpful to have standards for the OS 
> or HBAs to solve this, but that seems outside the scope 
> of the iSCSI protocol spec.

A solution that works here is to statically configure the
ISIDs along with the iSCSI name and write the "conservative
reuse" rule into the iSCSI protocol specification.

Thanks,
--David

> -----Original Message-----
> From: Elliott, Robert [mailto:Robert.Elliott@compaq.com]
> Sent: Wednesday, October 10, 2001 8:19 PM
> To: ips@ece.cmu.edu
> Subject: RE: iSCSI: ISID issue summary
> 
> 
> David Black wrote:
> 
> > SCSI architecture (SAM2) is based on an intuition that SCSI
> > ports are fixed (e.g., physical) things that have names.  So
> > far, the existence of the names has been an intellectual
> > curiosity as they haven't mattered to any visible SCSI
> > functionality.
> 
> Not quite true.  The Fibre Channel port's Worldwide_Name -
> what SAM-2 now calls the initiator port name - is used 
> today for persistent reservations.  SPC-2 has this wording:
> 
>   For those SCSI protocols for which the initiator port's 
>   world wide identification is available to the device 
>   server the initiator port's world wide identification 
>   shall be used to determine if the initiator identifier 
>   has changed.
> 
> FCP-2 indicates that the Worldwide_Name of the Fibre Channel 
> port serves as the "initiator port's world wide 
> identification."  (that phrase will change to "initiator 
> port name" as Jim's (accepted) T10 name proposal is 
> digested.)
> 
> >                Single port Initiators predominate in 
> > both Parallel SCSI and Fibre Channel (e.g., a dual ported
> > Fibre Channel HBA is usually treated as two SCSI Initiators,
> > not a single dual-ported one).
> 
> According to the SAM-2 multiport model, you can upgrade that
> "usually" to "always."  The key to the multiport model is:
> 
> 	Initiator = initator port
> 	Target = target port
> 
> In FCP-2, initiator ports equal Fibre Channel ports.  There's 
> no way to treat two physical ports as the same initiator
> without violating the standards.
> 
> The multiport model defines "initiator device" as the object
> that contains multiple initiator ports.  The initiator device 
> has neither an address nor a name in any existing protocol.
> There are no commands that rely on it.  Jim wanted to start
> using that concept for iSCSI access controls.
> 
> > Jim tells us that T10 is about to define persistent reserve
> > functionality that depends on the identity of the SCSI Initiator
> > Port - in particular that under some circumstances, persistent
> > reservations will be associated with the Initiator Port independent
> > of the Target Port.  Persistent reservations are currently bound
> > to the Initiator Port and Target Port (as well as the LU they
> > affect).  For the rest of this message, I'm going to assume
> 
> Again, the reason for this is the multiport model that says
> initiator = initiator port and target = target port.
> 
> > that we want to support this functionality.  Assuming this and
> > that I got the description in this paragraph right ...
> > 
> >  ... I need to take a slight detour into pragmatics.  Those
> > who configure storage networks rapidly discover the value
> > of consistency and matching configurations.  Multi-path I/O
> > configurations tend to want to see access to the same set of LUs
> > consistently across a set of interfaces (i.e., Ports).  Between
> > suitable configuration and the aggressive setup of connections
> > to all possible targets, all the required connections come into
> > existence as part of the boot process.  Applying
> > this experience to the new persistent reservation functionality,
> > one would expect persistent reservations that are bound only to
> > an Initiator Port to be independent of the Target Port, in that
> > the reservation should span connections to all of the relevant
> > Target Ports and those connections should come into existence
> > as part of the boot process.
> 
> That's the goal of one of the T10 proposals.  If initiators
> are identified only by their port addresses, it's not safe.  
> The two target ports could be on different fabrics, where
> initiators are different but happen to have the same
> addresses.  On a fabric where the initiator port has a name
> that is known to be worldwide unique, this becomes a
> non-issue if the initiator port name is used instead of the
> initiator port address.
> 
> > Turning to iSCSI, the situation gets complicated by the
> > interaction of two factors:
> > - Parallel sessions are permitted down to the interface
> > 	level (e.g., more than one iSCSI session may exist
> > 	that uses IP addresses A and B to connect iSCSI Initiator
> > 	I to iSCSI Target T).
> > - SCSI disallows parallel nexii (i.e., more than one session
> > 	between the same two SCSI Ports).
> > If SCSI Ports are mapped to hardware entities such as network
> > interfaces in iSCSI, the opportunity for parallel sessions
> > between the same pair of network interfaces vanishes.  In order
> > to avoid that outcome, iSCSI Initiator Ports have to be
> > mapped to software entities. To date, the ISID has been
> > used to identify those software entities, and the only rule
> > on their allocation is:
> > 
> >   ISID RULE: Between a given iSCSI Initiator and iSCSI 
> >   Target Portal Group (SCSI target port), then can be only
> >   one session with a given ISID (identifying a SCSI 
> >   initiator port).  See 3.12.8. 
> 
> There's a bit more than that.  The "iSCSI Name" has to
> be included with the ISID to identify the initiator port.
> Jim wanted the iSCSI name to be the "initiator device name."
> 
> > In order to get the expected outcome of the new functionality
> > being defined by T10, the same ISID has to be used to establish
> > all the connections that the persistent reservation is supposed
> > to span.  Unfortunately, that's above and beyond what iSCSI
> > requires - the relevant text about doing this in -08 is:
> > 
> >         The "ISID rule" does not preclude 
> >         the use of the same ISID from the same iSCSI Initiator with
> >         different Target Portal Groups on the same iSCSI target or
> >         on other iSCSI targets 
> > 
> > "does not preclude" is rather weak language for something that's
> > essential to making a piece of SCSI functionality work.  If an
> > iSCSI implementation uses a different ISID for every session
> > (which is also not precluded by the current text), then persistent
> > reservations will never span Targets or Target Ports for that
> > implementation despite T10's best efforts and intentions.  IMHO,
> > allowing that outcome is *wrong* (and this is the source of the
> > difference of opinion between Jim and myself).
> > 
> > The correction to this involves what Jim has called "conservative
> > reuse" of ISIDs.  This is something like for each ISID that an
> > implementation uses, a connection with that ISID is made to
> > every possible Target Portal Group of every possible Target.  I
> > suspect a longer explanation than this will be needed, including
> > a discussion of the desirability of avoiding a situation in which
> > the same ISID is used by two iSCSI implementations that are part
> > of the same Initiator and set up sessions concurrently.
> > 
> > IMHO, if we want to support persistent reservations at this stage
> > that span target ports, we need to replace the "does not preclude"
> > language with a REQUIREMENT for conservative reuse to avoid a
> > situation in which SCSI functionality that works consistently with
> > other transports works inconsistently with iSCSI (i.e., doesn't
> > work with iSCSI implementations that don't do conservative reuse).
> 
> As Nick noted, the problem is where do you store these ISIDs?
> For that matter, where do you store the iSCSI name?  With
> Fibre Channel, the Worldwide_name is in a ROM associated with
> the port.  With iSCSI you have no guarantee of hardware.  You
> can usually find an Ethernet MAC address and could base a
> name off of that.  However, there's no guarantees.
> 
> For the SCSI RDMA Protocol for InfiniBand, we chose:
>   initiator port name = 64-bit worldwide identifier from
>                         the InfiniBand channel adapter plus
>                         64 extra bits
> All software using the same InfiniBand channel adapter have to
> coordinate usage of the extra bits, but single-OS single-driver
> systems can just fill them with zeros.
> 
> > Stepping back from the assumption that support for port-spanning
> > persistent reservations is required, I observe that the conservative
> > reuse rule impacts all implementations, even those that will never
> > use such persistent reservations because ISID is a mandatory part
> > of the iSCSI protocol.
> 
> Persistent reservations, access controls, extended copy,
> third-party XOR commands, and the supporting alias commands
> are all potential users of port names today.  Protocol bridges
> and management tools may also benefit from using names 
> rather than addresses.
> 
> >                       If a text key were used instead, only those
> > Initiators that want to support this functionality on which T10
> > is "crossing a few t's and dotting a few i's" are affected (the
> > text key is optional).  The text key would be subject to a
> > conservative reuse rule similar to the one that would be needed
> > for ISIDs, and once T10 finishes the i's and t's, we can precisely
> > reference the SCSI features (persistent reservation expansion)
> > that require use of this text key.
> > 
> > In addition, text keys have much better alternatives for dealing
> > with conflicts than ISIDs - if a text key conflicts, it is possible
> > to continue  the negotiation with a different text key, whereas
> > ISID conflict is fatal to one of the two sessions involved.
> > As Jim has previously observed, changing the text key (e.g.,
> > generate a large random number in response to a conflict),
> > results in a situation that can be dealt with by software that
> > uses persistent reservations, although this departure from
> > conservative reuse should only occur as a consequence of some
> > sort of configuration mistake or bug.  At the tail end of this
> > reasoning chain is the observation that use of this text key
> > leaves the ISID without value/purpose, and hence would call
> > for its removal (or perhaps designation as a reserved field).
> 
> This just seems to make the names more vague and volatile.
> 
> Are the iSCSI name and ISID used as part of authentication?
> How does a device driver prove it has the right to use
> a certain iSCSI name/ISID?  How does it prevent some other
> software from using its own iSCSI name/ISID?
> 
> > Putting on my WG chair hat for a moment, I can accept either
> > alternative outlined above:
> > - Add conservative reuse to the ISID rule.
> > - Use a text key for iSCSI port identification.
> > But I'm not comfortable with the current situation in which iSCSI
> > implementations are permitted to break T10-defined SCSI features
> > that will work as expected in other SCSI transports.
> > 
> > I hope this now makes sense to more people than just Jim and I,
> > and I apologize for the length of this message, as this is a
> > somewhat subtle issue.
> > 
> > Comments?
> 
> If we didn't have ISIDs we'd still have the same debate about
> managing the iSCSI names.  Two software implementations
> on the same machine could choose the same iSCSI name and
> conflict - the same problems result.
> 
> Jim's partitioning lets the OS pick an iSCSI name (unique 
> from any other OS on the fabric, with any luck) and requires 
> the OS coordinate ISIDs for any iSCSI drivers sharing
> that iSCSI name.  A dedicated iSCSI HBA could also hand
> out ISIDs for drivers using it.  This should limit the 
> scope of conflicts to one system rather than the whole 
> fabric.  It'd be helpful to have standards for the OS 
> or HBAs to solve this, but that seems outside the scope 
> of the iSCSI protocol spec.
> 
> > --David
> > 
> > ---------------------------------------------------
> > David L. Black, Senior Technologist
> > EMC Corporation, 42 South St., Hopkinton, MA  01748
> > +1 (508) 435-1000 x75140     FAX: +1 (508) 497-8500
> > black_david@emc.com       Mobile: +1 (978) 394-7754
> > ---------------------------------------------------
> > 
> ---
> Rob Elliott, Compaq Server Storage
> Robert.Elliott@compaq.com
>