[Open-FCoE] FCoE target failure

Discussion:

Natti Shwarts

2014-01-15 13:23:11 UTC

Hello,

I'm trying to work with SCSI target over FCoE. (using fileio sample of SCST)
I'm using Linux 3.8.13 , Intel 10G card and HP5900 switch with VSAN 10 for
FCoE.
The problem is that the target doesn't send FLOGI to the switch.
(I have windows client that works with the same switch and configuration
and succeed to login)

The following is the operation sequence I perform:
- vconfig add eth2 10
- load scst, scst_user
- fileio_tgt disk0 file0 disk1 file1
- modprobe fcoe
- echo eth2.10 > /sys/module/libfcoe/parameters/create_vn2vn (fcoeadm -c
eth2.10 returned an error)
- modprobe fcst
- echo "add disk0 0" >
/sys/kernel/scst_tgt/targets/fcst/20\:01\:b8\:ca\:3a\:62\:82\:f9/luns/mgmt
- echo "add disk1 1" >
/sys/kernel/scst_tgt/targets/fcst/20\:01\:b8\:ca\:3a\:62\:82\:f9/luns/mgmt
- echo 1 >
/sys/kernel/scst_tgt/targets/fcst/20\:01\:b8\:ca\:3a\:62\:82\:f9/enabled

My questions are:
1. What is wrong with the above sequence ?
2. Do you have any idea why fcoeadm fails ?
3. Is "fcoeadm -c eth2.10" is exactly the same as "echo eth2.10 >
/sys/module/libfcoe/parameters/create_vn2vn " ?

Thanks
Natti

Robert Love

2014-01-15 20:36:02 UTC

Permalink

Post by Natti Shwarts
Hello,
I'm trying to work with SCSI target over FCoE. (using fileio sample of SCST)
I'm using Linux 3.8.13 , Intel 10G card and HP5900 switch with VSAN 10 for
FCoE.

I am not familiar with the SCST commands to be able to tell you if
you're using it right. I might be able to give you some pointers to help
debug though.

Post by Natti Shwarts
The problem is that the target doesn't send FLOGI to the switch.
(I have windows client that works with the same switch and configuration
and succeed to login)
- vconfig add eth2 10
- load scst, scst_user
- fileio_tgt disk0 file0 disk1 file1
- modprobe fcoe
- echo eth2.10 > /sys/module/libfcoe/parameters/create_vn2vn (fcoeadm -c
eth2.10 returned an error)

This is definitely a problem.

* Does eth2.10 exist? Was the VLAN configured on the switch? How did you
configure it on the host/target? Did you manually create the vlan or did
you use the 'fipvlan' application to discover/create it?

* What was the output of the 'echo' command? Why did it fail?

* What did the log files say (dmesg/messages)? You can also turn on
debug_logging for the fcoe kernel modules to get more details. I
wouldn't bother with fcoe.ko at this point as its debug_logging is very
verbose. You could try the following commands after the modprobe.

# echo 0xFFFF > /sys/module/libfcoe/parameters/debug_logging
# echo 0xFFFF > /sys/module/libfc/parameters/debug_logging

Post by Natti Shwarts
- modprobe fcst
- echo "add disk0 0" >
/sys/kernel/scst_tgt/targets/fcst/20\:01\:b8\:ca\:3a\:62\:82\:f9/luns/mgmt
- echo "add disk1 1" >
/sys/kernel/scst_tgt/targets/fcst/20\:01\:b8\:ca\:3a\:62\:82\:f9/luns/mgmt
- echo 1 >
/sys/kernel/scst_tgt/targets/fcst/20\:01\:b8\:ca\:3a\:62\:82\:f9/enabled
1. What is wrong with the above sequence ?

You should have stopped at the fcoeadm fail, no need to proceed with the
failure.

Post by Natti Shwarts
2. Do you have any idea why fcoeadm fails ?

No, I don't have any information about the failure to draw a conclusion.

Post by Natti Shwarts
3. Is "fcoeadm -c eth2.10" is exactly the same as "echo eth2.10 >
/sys/module/libfcoe/parameters/create_vn2vn " ?

No. 'fcoeadm -c eth2.10' with do 'Fabric Mode' and not VN2VN.

I'm not sure what kernel version you're using. For a while we did not
have VN2VN support in fcoeadm. Only in more recent kernels did we change
the sysfs interfaces to use files in /sys/bus/fcoe/. This change allowed
us to add VN2VN support to fcoeadm with the '-m|--mode' option. The
'echo' method is how you enable VN2VN on older kernels. On newer kernels
that method will still work, but it is deprecated because of the new
interfaces.

Hope this helps,

//Rob

Robert Love

2014-01-15 21:53:12 UTC

Permalink

Post by Robert Love

Post by Natti Shwarts
Hello,
I'm trying to work with SCSI target over FCoE. (using fileio sample of SCST)
I'm using Linux 3.8.13 , Intel 10G card and HP5900 switch with VSAN 10 for

Oops... Missed the kernel version here on first read, also this seems to
imply that you configured the switch.

Post by Robert Love

Post by Natti Shwarts
FCoE.

I am not familiar with the SCST commands to be able to tell you if
you're using it right. I might be able to give you some pointers to
help debug though.

This is definitely a problem.
* Does eth2.10 exist? Was the VLAN configured on the switch? How did
you configure it on the host/target? Did you manually create the vlan
or did you use the 'fipvlan' application to discover/create it?
* What was the output of the 'echo' command? Why did it fail?
* What did the log files say (dmesg/messages)? You can also turn on
debug_logging for the fcoe kernel modules to get more details. I
wouldn't bother with fcoe.ko at this point as its debug_logging is
very verbose. You could try the following commands after the modprobe.
# echo 0xFFFF > /sys/module/libfcoe/parameters/debug_logging
# echo 0xFFFF > /sys/module/libfc/parameters/debug_logging

You should have stopped at the fcoeadm fail, no need to proceed with
the failure.

Post by Natti Shwarts
2. Do you have any idea why fcoeadm fails ?

No, I don't have any information about the failure to draw a conclusion.

Post by Natti Shwarts
3. Is "fcoeadm -c eth2.10" is exactly the same as "echo eth2.10 >
/sys/module/libfcoe/parameters/create_vn2vn " ?

No. 'fcoeadm -c eth2.10' with do 'Fabric Mode' and not VN2VN.
I'm not sure what kernel version you're using. For a while we did not
have VN2VN support in fcoeadm. Only in more recent kernels did we
change the sysfs interfaces to use files in /sys/bus/fcoe/. This
change allowed us to add VN2VN support to fcoeadm with the '-m|--mode'
option. The 'echo' method is how you enable VN2VN on older kernels. On
newer kernels that method will still work, but it is deprecated
because of the new interfaces.

fcoe_sysfs went into the kernel on 2012-12-14 and 3.8 was tagged on
2013-02-18, so I think you should have the new interfaces. Does
'fcoeadm' have a '-m|--mode' option?

Bart Van Assche

2014-01-16 15:18:31 UTC

Permalink

Post by Robert Love

Post by Natti Shwarts
Hello,
I'm trying to work with SCSI target over FCoE. (using fileio sample of SCST)
I'm using Linux 3.8.13 , Intel 10G card and HP5900 switch with VSAN 10 for
FCoE.

I am not familiar with the SCST commands to be able to tell you if
you're using it right. I might be able to give you some pointers to help
debug though.

This is definitely a problem.
* Does eth2.10 exist? Was the VLAN configured on the switch? How did you
configure it on the host/target? Did you manually create the vlan or did
you use the 'fipvlan' application to discover/create it?
* What was the output of the 'echo' command? Why did it fail?
* What did the log files say (dmesg/messages)? You can also turn on
debug_logging for the fcoe kernel modules to get more details. I
wouldn't bother with fcoe.ko at this point as its debug_logging is very
verbose. You could try the following commands after the modprobe.
# echo 0xFFFF > /sys/module/libfcoe/parameters/debug_logging
# echo 0xFFFF > /sys/module/libfc/parameters/debug_logging

You should have stopped at the fcoeadm fail, no need to proceed with the
failure.

Post by Natti Shwarts
2. Do you have any idea why fcoeadm fails ?

No, I don't have any information about the failure to draw a conclusion.

Post by Natti Shwarts
3. Is "fcoeadm -c eth2.10" is exactly the same as "echo eth2.10 >
/sys/module/libfcoe/parameters/create_vn2vn " ?

No. 'fcoeadm -c eth2.10' with do 'Fabric Mode' and not VN2VN.
I'm not sure what kernel version you're using. For a while we did not
have VN2VN support in fcoeadm. Only in more recent kernels did we change
the sysfs interfaces to use files in /sys/bus/fcoe/. This change allowed
us to add VN2VN support to fcoeadm with the '-m|--mode' option. The
'echo' method is how you enable VN2VN on older kernels. On newer kernels
that method will still work, but it is deprecated because of the new
interfaces.

I have tried the sequence mentioned by Natti myself. What I see on the
initiator system with kernel 3.13-rc8 is the following:
* When not using VLAN (eth0), login works fine - the SCSI devices exported
by the target system appear at the initiator side.
* When using VLAN, no SCSI devices appear after reloading the fcoe module
on the initiator system. However, about 22 seconds later a kernel crash
is triggered.

Robert Love

2014-01-23 01:34:06 UTC

Permalink

Post by Bart Van Assche

Post by Robert Love

Post by Natti Shwarts
Hello,
I'm trying to work with SCSI target over FCoE. (using fileio sample of SCST)
I'm using Linux 3.8.13 , Intel 10G card and HP5900 switch with VSAN 10 for
FCoE.

I am not familiar with the SCST commands to be able to tell you if
you're using it right. I might be able to give you some pointers to help
debug though.

This is definitely a problem.
* Does eth2.10 exist? Was the VLAN configured on the switch? How did you
configure it on the host/target? Did you manually create the vlan or did
you use the 'fipvlan' application to discover/create it?
* What was the output of the 'echo' command? Why did it fail?
* What did the log files say (dmesg/messages)? You can also turn on
debug_logging for the fcoe kernel modules to get more details. I
wouldn't bother with fcoe.ko at this point as its debug_logging is very
verbose. You could try the following commands after the modprobe.
# echo 0xFFFF > /sys/module/libfcoe/parameters/debug_logging
# echo 0xFFFF > /sys/module/libfc/parameters/debug_logging

You should have stopped at the fcoeadm fail, no need to proceed with the
failure.

Post by Natti Shwarts
2. Do you have any idea why fcoeadm fails ?

No, I don't have any information about the failure to draw a conclusion.

Post by Natti Shwarts
3. Is "fcoeadm -c eth2.10" is exactly the same as "echo eth2.10 >
/sys/module/libfcoe/parameters/create_vn2vn " ?

No. 'fcoeadm -c eth2.10' with do 'Fabric Mode' and not VN2VN.
I'm not sure what kernel version you're using. For a while we did not
have VN2VN support in fcoeadm. Only in more recent kernels did we change
the sysfs interfaces to use files in /sys/bus/fcoe/. This change allowed
us to add VN2VN support to fcoeadm with the '-m|--mode' option. The
'echo' method is how you enable VN2VN on older kernels. On newer kernels
that method will still work, but it is deprecated because of the new
interfaces.

I have tried the sequence mentioned by Natti myself. What I see on the
* When not using VLAN (eth0), login works fine - the SCSI devices exported
by the target system appear at the initiator side.
* When using VLAN, no SCSI devices appear after reloading the fcoe module
on the initiator system. However, about 22 seconds later a kernel crash
is triggered.

Natti Shwarts

2014-01-23 05:55:50 UTC

Permalink

*'fcoeadm -c eth2.10' *
doesn't work, I receive the error: ' fcoeadm: Failed to open
/sys/module/fcoe/parameters/create'

*'echo eth2.10 > /sys/module/libfcoe/parameters**/create_vn2vn'*
Nothing happens, in Wireshark trace I see the following messages:
Probe request
Claim notification
and a lot of beacon messages

*'echo eth2.10 > /sys/module/libfcoe/parameters**/create'*
I have the successful FLOGI , but the client can't connect me (I can't find
the PLOGI and PRLI accept, see attached file)

Natti

Post by Bart Van Assche

Post by Robert Love

Post by Natti Shwarts
Hello,
I'm trying to work with SCSI target over FCoE. (using fileio sample of SCST)
I'm using Linux 3.8.13 , Intel 10G card and HP5900 switch with VSAN 10 for
FCoE.

I am not familiar with the SCST commands to be able to tell you if
you're using it right. I might be able to give you some pointers to help
debug though.
The problem is that the target doesn't send FLOGI to the switch.

Post by Natti Shwarts
(I have windows client that works with the same switch and configuration
and succeed to login)
- vconfig add eth2 10
- load scst, scst_user
- fileio_tgt disk0 file0 disk1 file1
- modprobe fcoe
- echo eth2.10 > /sys/module/libfcoe/parameters/create_vn2vn (fcoeadm -c
eth2.10 returned an error)

This is definitely a problem.
* Does eth2.10 exist? Was the VLAN configured on the switch? How did you
configure it on the host/target? Did you manually create the vlan or did
you use the 'fipvlan' application to discover/create it?
* What was the output of the 'echo' command? Why did it fail?
* What did the log files say (dmesg/messages)? You can also turn on
debug_logging for the fcoe kernel modules to get more details. I
wouldn't bother with fcoe.ko at this point as its debug_logging is very
verbose. You could try the following commands after the modprobe.
# echo 0xFFFF > /sys/module/libfcoe/parameters/debug_logging
# echo 0xFFFF > /sys/module/libfc/parameters/debug_logging
- modprobe fcst

Post by Natti Shwarts
- echo "add disk0 0" >
82\:f9/luns/mgmt
- echo "add disk1 1" >
82\:f9/luns/mgmt
- echo 1 >
82\:f9/enabled
1. What is wrong with the above sequence ?

You should have stopped at the fcoeadm fail, no need to proceed with the
failure.

Post by Natti Shwarts
2. Do you have any idea why fcoeadm fails ?

No, I don't have any information about the failure to draw a conclusion.
3. Is "fcoeadm -c eth2.10" is exactly the same as "echo eth2.10 >

Post by Natti Shwarts
/sys/module/libfcoe/parameters/create_vn2vn " ?

No. 'fcoeadm -c eth2.10' with do 'Fabric Mode' and not VN2VN.
I'm not sure what kernel version you're using. For a while we did not
have VN2VN support in fcoeadm. Only in more recent kernels did we change
the sysfs interfaces to use files in /sys/bus/fcoe/. This change allowed
us to add VN2VN support to fcoeadm with the '-m|--mode' option. The
'echo' method is how you enable VN2VN on older kernels. On newer kernels
that method will still work, but it is deprecated because of the new
interfaces.

Robert Love

2014-01-23 17:30:33 UTC

Permalink

_'fcoeadm -c eth2.10' _
doesn't work, I receive the error: ' fcoeadm: Failed to open
/sys/module/fcoe/parameters/create'

This is not how you start VN2VN, as I stated in my first reply, do not
do this anymore.

_'echo eth2.10 > /sys/module/libfcoe/parameters__/create_vn2vn'_
Probe request
Claim notification
and a lot of beacon messages

This is the legacy way of starting VN2VN, it should work. It worked for
me, but I did not try with a VLAN. When I get a chance to get back to
this I will try with a VLAN.

_'echo eth2.10 > /sys/module/libfcoe/parameters__/create'_
I have the successful FLOGI , but the client can't connect me (I can't
find the PLOGI and PRLI accept, see attached file)

This is not how you start VN2VN, do not do this anymore.

Natti

I am unclear if you're just running these command successively or after
reboots/rmmod. If you just run them sequentially you're going to
initialize data structures with the 'fcoeadm -c eth2.10' and the
initiator will be in Fabric mode, so any commands after this won't do
anything. You need to make sure you destroy the interface before you try
a different style of creating.

I'd suggest that you reboot, i.e. start frest, and then just do 'echo
eth2.10 > /sys/module/libfcoe/parameters/create_vn2vn'.

I will take a look at your logs when I get a chance. I'm just replying
with this info because I won't have time to look at them today.

//Rob

Robert Love

2014-01-24 18:50:56 UTC

Permalink

Post by Robert Love

Post by Natti Shwarts
Hello,
I'm trying to work with SCSI target over FCoE. (using fileio sample of SCST)
I'm using Linux 3.8.13 , Intel 10G card and HP5900 switch with VSAN 10 for
FCoE.

I am not familiar with the SCST commands to be able to tell you if
you're using it right. I might be able to give you some pointers to
help debug though.

I'm using TCM, not SCST, but VN2VN with a VLAN worked for me.

I have debug logging on for libfcoe on both systems.

# echo 0xFFFF > /sys/module/libfcoe/parameters/debug_logging

On Target:

[***@vmF18t ~]# uname -r
3.13.0+
[***@vmF18t ~]# modprobe fcoe
[***@vmF18t ~]# echo eth2.10 > /sys/module/libfcoe/parameters/create_vn2vn
[***@vmF18t ~]# tail -10 /var/log/messages
Jan 24 13:46:27 vmF18t kernel: [61662872.081832] host8: fip: state
VNMP_START -> VNMP_PROBE1
Jan 24 13:46:27 vmF18t kernel: [61662872.181897] host8: fip: state
VNMP_PROBE1 -> VNMP_PROBE2
Jan 24 13:46:27 vmF18t kernel: [61662872.583338] host8: fip: state
VNMP_PROBE2 -> VNMP_CLAIM
Jan 24 13:46:27 vmF18t kernel: [61662872.584493] host8: Assigned Port ID
0091fb
Jan 24 13:46:27 vmF18t kernel: [61662872.584498] host8: lport 0091fb:
Entered READY from state FLOGI
Jan 24 13:46:27 vmF18t kernel: [61662872.584502] host8: lport 0091fb:
Discovery succeeded
Jan 24 13:46:28 vmF18t kernel: [61662872.985577] host8: fip: state
VNMP_CLAIM -> VNMP_UP

On Initiator:

[***@vmF18i open-fcoe-initiator]# uname -r
3.13.0+
[***@vmF18i open-fcoe-initiator]# modprobe fcoe
[***@vmF18i open-fcoe-initiator]# echo eth2.10 >
/sys/module/libfcoe/parameters/create_vn2vn
[***@vmF18i open-fcoe-initiator]# lsscsi
[1:0:0:0] cd/dvd QEMU QEMU DVD-ROM 1.0. /dev/sr0
[17:0:0:1] disk LIO-ORG FILEIO 4.0 /dev/sda
[***@vmF18i open-fcoe-initiator]# tail -10 /var/log/messages
Jan 24 13:49:04 vmF18i kernel: [61654280.239674] host17: rport 0091fb:
Received a PRLI response, but in state Ready
Jan 24 13:49:04 vmF18i kernel: [61654280.240496] host17: rport 0091fb:
Received a FLOGI response, but in state Ready
Jan 24 13:49:04 vmF18i kernel: [61654280.245301] host17: rport 0091fb:
callback ev 1
Jan 24 13:49:04 vmF18i kernel: [61654280.253803] scsi 17:0:0:1:
Direct-Access LIO-ORG FILEIO 4.0 PQ: 0 ANSI: 5
Jan 24 13:49:04 vmF18i kernel: [61654280.259583] sd 17:0:0:1: [sda]
8388608 512-byte logical blocks: (4.29 GB/4.00 GiB)
Jan 24 13:49:04 vmF18i kernel: [61654280.261441] sd 17:0:0:1: Attached
scsi generic sg1 type 0
Jan 24 13:49:04 vmF18i kernel: [61654280.263090] sd 17:0:0:1: [sda]
Write Protect is off
Jan 24 13:49:04 vmF18i kernel: [61654280.265034] sd 17:0:0:1: [sda]
Write cache: enabled, read cache: enabled, supports DPO and FUA
Jan 24 13:49:04 vmF18i kernel: [61654280.278197] sda: unknown partition
table
Jan 24 13:49:04 vmF18i kernel: [61654280.291158] sd 17:0:0:1: [sda]
Attached SCSI disk

Natti Shwarts

2014-01-16 13:08:20 UTC

Permalink

Hi,
I didn't receive the message directly so I copied it to here. Please see my
response inline

I installed Ubuntu 12.04.3, than downloaded sources (3.8.13.12) and
compiled the kernel as described in the SCST howto document for working
with Qlogic.
To use the fcoeadm I performed apt-get install, should I perform this
differently ?

Thanks a lot
Natti

---------- Forwarded message ----------
From: Robert Love <***@intel.com>
To: fcoe-***@open-fcoe.org
Cc:
Date: Wed, 15 Jan 2014 13:53:12 -0800
Subject: Re: [Open-FCoE] FCoE target failure

Post by Natti Shwarts
Hello,
I'm trying to work with SCSI target over FCoE. (using fileio sample of
SCST)
I'm using Linux 3.8.13 , Intel 10G card and HP5900 switch with VSAN 10 for

Oops... Missed the kernel version here on first read, also this seems to
imply that you configured the switch.

FCoE.

I am not familiar with the SCST commands to be able to tell you if you're
using it right. I might be able to give you some pointers to help debug
though.
The problem is that the target doesn't send FLOGI to the switch.

* I created eth2.10 using 'vconfig add eth2 10" , *
* fipvlan response "No Fibre Channel Forwarders Found", how should I use
it ?*

* What was the output of the 'echo' command? Why did it fail?
*The echo command didn't fail (the fcoeadm -c failed) *

* What did the log files say (dmesg/messages)? You can also turn on
debug_logging for the fcoe kernel modules to get more details. I wouldn't
bother with fcoe.ko at this point as its debug_logging is very verbose. You
could try the following commands after the modprobe.

# echo 0xFFFF > /sys/module/libfcoe/parameters/debug_logging
# echo 0xFFFF > /sys/module/libfc/parameters/debug_logging

- modprobe fcst

Post by Natti Shwarts
- echo "add disk0 0" >
/sys/kernel/scst_tgt/targets/fcst/20\:01\:b8\:ca\:3a\:62\:82\:f9/luns/mgmt
- echo "add disk1 1" >
/sys/kernel/scst_tgt/targets/fcst/20\:01\:b8\:ca\:3a\:62\:82\:f9/luns/mgmt
- echo 1 >
/sys/kernel/scst_tgt/targets/fcst/20\:01\:b8\:ca\:3a\:62\:82\:f9/enabled
1. What is wrong with the above sequence ?

You should have stopped at the fcoeadm fail, no need to proceed with the
failure.

*I sent a question to this forum and received a response to use the "echo"
command instead... *

2. Do you have any idea why fcoeadm fails ?
No, I don't have any information about the failure to draw a conclusion.
* The output of "fcoeadm -c eth2.10" is: "fcoeadm: Failed to open
/sys/module/fcoe/parameters/create"*
* No special message in the system log (debug_loggi**ng of libfcoe and
libfc is on as you suggested)*
3. Is "fcoeadm -c eth2.10" is exactly the same as "echo eth2.10 >

Post by Natti Shwarts
/sys/module/libfcoe/parameters/create_vn2vn " ?

No. 'fcoeadm -c eth2.10' with do 'Fabric Mode' and not VN2VN.
I'm not sure what kernel version you're using. For a while we did not have
VN2VN support in fcoeadm. Only in more recent kernels did we change the
sysfs interfaces to use files in /sys/bus/fcoe/. This change allowed us to
add VN2VN support to fcoeadm with the '-m|--mode' option. The 'echo' method
is how you enable VN2VN on older kernels. On newer kernels that method will
still work, but it is deprecated because of the new interfaces.

fcoe_sysfs went into the kernel on 2012-12-14 and 3.8 was tagged on
2013-02-18, so I think you should have the new interfaces. Does 'fcoeadm'
have a '-m|--mode' option?
*No*