Archive for December, 2015

Dec 14

Solaris Multipath Incorrect Totals

From time to time we notice that some LUN's are not optimal. It could be because of off-lining a LUN, changes on the switches I am not sure why exactly it happens. If multipath is not showing the correct Path Counts you may need to run cfgadm.

See how some LUN's here are showing 4 paths only. We expect 8.

# mpathadm list lu
        /dev/rdsk/c0t5000CCA04385ED60d0s2
                Total Path Count: 1
                Operational Path Count: 1
[..]
        /dev/rdsk/c0t600144F09D7311B500005605A40C0006d0s2
                Total Path Count: 8
                Operational Path Count: 8
        /dev/rdsk/c0t600144F09D7311B50000561ED8AB0007d0s2
                Total Path Count: 8
                Operational Path Count: 8
[..]
        /dev/rdsk/c0t600144F09D7311B50000538507080021d0s2
                Total Path Count: 8
                Operational Path Count: 8
        /dev/rdsk/c0t600144F09D7311B50000534309C40011d0s2
                Total Path Count: 4
                Operational Path Count: 4
        /dev/rdsk/c0t600144F09D7311B500005342FE86000Fd0s2
                Total Path Count: 4
                Operational Path Count: 4
        /dev/rdsk/c0t600144F09D7311B5000053D13E130029d0s2
                Total Path Count: 4
                Operational Path Count: 4
        /dev/rdsk/c0t600144F09D7311B50000566AE1CC0008d0s2
                Total Path Count: 4
                Operational Path Count: 4

I tried a few things and it looks like the cfgadm worked. It could also be a couple other thing that triggered it like destroying an unused LUN or changing a recently added LUN's target group to be more restrictive but I doubt that is it. Most likely cfgadm.

# cfgadm -o show_SCSI_LUN -al
Ap_Id                          Type         Receptacle   Occupant     Condition
c1                             fc           connected    unconfigured unknown
c8                             fc-fabric    connected    configured   unknown
c8::20520002ac000f02,254       ESI          connected    configured   unknown
c8::21000024ff3db11d,0         disk         connected    configured   unknown
c8::21000024ff3db11d,1         disk         connected    configured   unknown
[..]
c8::21000024ff57d646,54        disk         connected    configured   unknown
c8::21000024ff57d646,56        disk         connected    configured   unknown
c8::21520002ac000f02,254       ESI          connected    configured   unknown
c8::22520002ac000f02,254       ESI          connected    configured   unknown
c8::23520002ac000f02,254       ESI          connected    configured   unknown
c9                             fc           connected    unconfigured unknown
c13                            fc-fabric    connected    configured   unknown
c13::20510002ac000f02,254      ESI          connected    configured   unknown
c13::21000024ff3db11c,0        disk         connected    configured   unknown
[..]
c13::21000024ff3db11c,44       disk         connected    configured   unknown
c13::21000024ff3db11c,46       disk         connected    configured   unknown
c13::21000024ff3db11c,48       disk         connected    configured   unknown
c13::21000024ff3db11c,50       disk         connected    unconfigured unknown
c13::21000024ff3db11c,52       disk         connected    unconfigured unknown
c13::21000024ff3db11c,54       disk         connected    unconfigured unknown
c13::21000024ff3db11c,56       disk         connected    configured   unknown
c13::21000024ff3db1b4,0        disk         connected    configured   unknown
c13::21000024ff3db1b4,1        disk         connected    configured   unknown
[..]
c13::21510002ac000f02,254      ESI          connected    configured   unknown
c13::22510002ac000f02,254      ESI          connected    configured   unknown
c13::23510002ac000f02,254      ESI          connected    configured   unknown

After I ran cfgadm..

# mpathadm list lu
        /dev/rdsk/c0t5000CCA04385ED60d0s2
                Total Path Count: 1
                Operational Path Count: 1
[..]
        /dev/rdsk/c0t600144F09D7311B500005605A40C0006d0s2
                Total Path Count: 8
                Operational Path Count: 8
        /dev/rdsk/c0t600144F09D7311B50000561ED8AB0007d0s2
                Total Path Count: 8
                Operational Path Count: 8
[..]
        /dev/rdsk/c0t600144F09D7311B5000053BE90620024d0s2
                Total Path Count: 8
                Operational Path Count: 8
        /dev/rdsk/c0t600144F09D7311B50000533C012A0009d0s2
                Total Path Count: 8
                Operational Path Count: 8
        /dev/rdsk/c0t600144F09D7311B50000533AAFF00007d0s2
                Total Path Count: 8
                Operational Path Count: 8
        /dev/rdsk/c0t600144F09D7311B50000538507080021d0s2
                Total Path Count: 8
                Operational Path Count: 8
        /dev/rdsk/c0t600144F09D7311B50000534309C40011d0s2
                Total Path Count: 8
                Operational Path Count: 8
        /dev/rdsk/c0t600144F09D7311B500005342FE86000Fd0s2
                Total Path Count: 8
                Operational Path Count: 8
        /dev/rdsk/c0t600144F09D7311B5000053D13E130029d0s2
                Total Path Count: 8
                Operational Path Count: 8
        /dev/rdsk/c0t600144F09D7311B50000566AE1CC0008d0s2
                Total Path Count: 8
                Operational Path Count: 8

I can also see the changes in messages...

# dmesg
Nov  9 03:59:21 solaris11 mac: [ID 469746 kern.info] NOTICE: ldoms-vsw0.vport15 registered
Nov  9 05:41:55 solaris11 scsi: [ID 583861 kern.info] ssd98 at scsi_vhci0: unit-address g600144f09d7311b50000534309c40011: f_tpgs
[..]
Dec 14 09:38:01 solaris11 genunix: [ID 483743 kern.info] /scsi_vhci/ssd@g600144f09d7311b50000566ae1cc0008 (ssd101) multipath status: optimal: path 343 fp16/ssd@w21000024ff3db1b5,38 is standby
Dec 14 09:38:01 solaris11 last message repeated 1 time
Dec 14 09:38:01 solaris11 genunix: [ID 483743 kern.info] /scsi_vhci/ssd@g600144f09d7311b5000053d13e130029 (ssd100) multipath status: optimal: path 344 fp16/ssd@w21000024ff3db1b5,36 is standby
Dec 14 09:38:01 solaris11 genunix: [ID 483743 kern.info] /scsi_vhci/ssd@g600144f09d7311b500005342fe86000f (ssd99) multipath status: optimal: path 345 fp16/ssd@w21000024ff3db1b5,34 is standby
Dec 14 09:38:01 solaris11 genunix: [ID 483743 kern.info] /scsi_vhci/ssd@g600144f09d7311b50000534309c40011 (ssd98) multipath status: optimal: path 346 fp16/ssd@w21000024ff3db1b5,32 is standby
Dec 14 09:38:07 solaris11 genunix: [ID 483743 kern.info] /scsi_vhci/ssd@g600144f09d7311b50000566ae1cc0008 (ssd101) multipath status: optimal: path 347 fp16/ssd@w21000024ff3db11d,38 is standby
Dec 14 09:38:07 solaris11 last message repeated 1 time
Dec 14 09:38:07 solaris11 genunix: [ID 483743 kern.info] /scsi_vhci/ssd@g600144f09d7311b5000053d13e130029 (ssd100) multipath status: optimal: path 348 fp16/ssd@w21000024ff3db11d,36 is standby
Dec 14 09:38:08 solaris11 genunix: [ID 483743 kern.info] /scsi_vhci/ssd@g600144f09d7311b500005342fe86000f (ssd99) multipath status: optimal: path 349 fp16/ssd@w21000024ff3db11d,34 is standby
Dec 14 09:38:08 solaris11 genunix: [ID 483743 kern.info] /scsi_vhci/ssd@g600144f09d7311b50000534309c40011 (ssd98) multipath status: optimal: path 350 fp16/ssd@w21000024ff3db11d,32 is standby
Dec 14 09:38:16 solaris11 genunix: [ID 483743 kern.info] /scsi_vhci/ssd@g600144f09d7311b50000566ae1cc0008 (ssd101) multipath status: optimal: path 351 fp21/ssd@w21000024ff3db1b4,38 is standby
Dec 14 09:38:16 solaris11 last message repeated 1 time
Dec 14 09:38:17 solaris11 genunix: [ID 483743 kern.info] /scsi_vhci/ssd@g600144f09d7311b5000053d13e130029 (ssd100) multipath status: optimal: path 352 fp21/ssd@w21000024ff3db1b4,36 is standby
Dec 14 09:38:17 solaris11 genunix: [ID 483743 kern.info] /scsi_vhci/ssd@g600144f09d7311b500005342fe86000f (ssd99) multipath status: optimal: path 353 fp21/ssd@w21000024ff3db1b4,34 is standby
Dec 14 09:38:17 solaris11 genunix: [ID 483743 kern.info] /scsi_vhci/ssd@g600144f09d7311b50000534309c40011 (ssd98) multipath status: optimal: path 354 fp21/ssd@w21000024ff3db1b4,32 is standby
Dec 14 09:38:22 solaris11 genunix: [ID 483743 kern.info] /scsi_vhci/ssd@g600144f09d7311b50000566ae1cc0008 (ssd101) multipath status: optimal: path 355 fp21/ssd@w21000024ff3db11c,38 is standby
Dec 14 09:38:22 solaris11 last message repeated 1 time
Dec 14 09:38:22 solaris11 genunix: [ID 483743 kern.info] /scsi_vhci/ssd@g600144f09d7311b5000053d13e130029 (ssd100) multipath status: optimal: path 356 fp21/ssd@w21000024ff3db11c,36 is standby
Dec 14 09:38:23 solaris11 genunix: [ID 483743 kern.info] /scsi_vhci/ssd@g600144f09d7311b500005342fe86000f (ssd99) multipath status: optimal: path 357 fp21/ssd@w21000024ff3db11c,34 is standby
Dec 14 09:38:23 solaris11 genunix: [ID 483743 kern.info] /scsi_vhci/ssd@g600144f09d7311b50000534309c40011 (ssd98) multipath status: optimal: path 358 fp21/ssd@w21000024ff3db11c,32 is standby

Comments Off on Solaris Multipath Incorrect Totals
comments

Dec 10

Solaris Snoop on File Access

If you find yourself trying to figure out where your operating system is spending time with reads and writes try this little dtrace gem. Script is here: http://dtracebook.com/index.php/File_System:rwsnoop

I ran it like below. Unknown is socket access and filtering out ssh and grep explains itself.

# ./rwsnoop.dtrace | egrep -v "sshd|grep|unknown"
  UID    PID CMD          D   BYTES FILE
    0    637 utmpd        R       4 /var/adm/wtmpx
  324   2884 java         W      77 /scratch/agtst1ML/MemoryMonitorLog.log
  324   2884 java         W      77 /scratch/agtst1ML/MemoryMonitorLog.log
  324   2884 java         W      77 /scratch/agtst1ML/MemoryMonitorLog.log
  324   2884 java         W      16 /devices/pseudo/poll@0:poll
  324   2884 java         W       8 /devices/pseudo/poll@0:poll
    1    593 nfsmapid     R      78 /etc/resolv.conf
    1    593 nfsmapid     R       0 /etc/resolv.conf
  324   2884 java         W      77 /scratch/agtst1ML/MemoryMonitorLog.log
    0      1 init         R    1006 /etc/inittab
    0      1 init         R       0 /etc/inittab
    0      1 init         W     412 /etc/svc/volatile/init-next.state
    0      1 init         W     412 /etc/svc/volatile/init-next.state
    0      1 init         R    1006 /etc/inittab
    0      1 init         R       0 /etc/inittab
    1    180 kcfd         R     976 /usr/lib/security/pkcs11_kernel.so.1

Comments Off on Solaris Snoop on File Access
comments

Dec 02

Service Response Time

There are several ways to get this done but I found using curl gives a good quick test of how long a web response was. You can use a formatted text file or just command line.

Command line can be something like this:

# curl -w "\ntime_namelookup:\t%{time_namelookup}\ntime_connect:\t\t%{time_connect}\ntime_appconnect:\t%{time_appconnect}\ntime_pretransfer:\t%{time_pretransfer}\ntime_redirect:\t\t%{time_redirect}\ntime_starttransfer:\t%{time_starttransfer}\n----------\ntime_total:\t\t%{time_total}\n" -o /dev/null -s https://www.wormly.com
 
time_namelookup:        0.366
time_connect:           0.367
time_appconnect:        0.402
time_pretransfer:       0.402
time_redirect:          0.000
time_starttransfer:     0.408
----------
time_total:             0.408

Or using a text file for formatting:

# curl -w "@curl-format.txt" -o /dev/null -s https://www.wormly.com

            time_namelookup:  0.127
               time_connect:  0.128
            time_appconnect:  0.142
           time_pretransfer:  0.142
              time_redirect:  0.000
         time_starttransfer:  0.150
                            ----------
                 time_total:  0.151

Text file like this:

# cat curl-format.txt 
\n
            time_namelookup:  %{time_namelookup}\n
               time_connect:  %{time_connect}\n
            time_appconnect:  %{time_appconnect}\n
           time_pretransfer:  %{time_pretransfer}\n
              time_redirect:  %{time_redirect}\n
         time_starttransfer:  %{time_starttransfer}\n
                            ----------\n
                 time_total:  %{time_total}\n
\n

Comments Off on Service Response Time
comments

Dec 01

Linux tabbed SSH connection manager

I like to work in a tabbed SSH connection manager. Especially when I have hundred's or thousands of machines to connect to. A connection manager like putty keeps track of machine names and login info. Using a tabbed interface like MTPutty can make your life a whole lot easier with treeview/groups and side by side terminals. And additionally if you can cluster the terminal commands it can be an added bonus.

So far I have not really liked anything in the Linux world as far as a SSH connection manager. Ubuntu does come with putty which seems to work the same as in the Windows world. Best I could find is an application written in Python called Gnome Connection Manager (gcm). In the Ubuntu 15.10 repos the package is called gnome-connection-manager. Be warned of a few things:

1. Seems the code is at least a few years old and the website does not seem to have documentation or any kind of discussion. The code is all python so you can look at fixing and emailing the owner.
2. Be sure to check the paste-right-click in ~/.gcm/gcm.conf. This can be a nasty setting if you did not expect it. I copy and paste a lot between a Windows desktop and a Linux guest and almost accidentally pasted garbage into a critical device.
3. Also check auto-copy-selection if you like putty style behavior where anything selected in your SSH terminal should be in the copy buffer.

I think if the developer put a little bit more love into gnome-connection-manager it would definitely be a keeper and first rate gnome app. I have looked at some other options like hotssh but worth checking out is Remmina. Unfortunately for me Remmina was very buggy.

1
comments