Category: Linux

Nov 24

Linux WakeOnLAN Issue

Wake On LAN Issue

I had a strange issue where my ZFS and restic backups to an Ubuntu backup server stopped working. The server had an interesting issue that was totally unrelated. It would boot on manual power-on but start shutting down a couple minutes later. This was completely unrelated to WOL and was fixed after I removed micro8ks.

The WOL issue ended up not actually being anything to do with the backup server but instead the source server(desktop01) not sending the magic packet at all. I figured it out by sniffing the backup server ingress using tcpdump. I could see the traffic come in when sending WOL from my ASUS router.

I suspect the desktop01 server which has multiple virtual interfaces, the wakeonlan utility is getting confused where to send out on. Ended up using etherwake instead of wakeonlan from the source server. Etherwake can specify the interface (-i interface) to send out on.

tcpdump when sending WOL magic packet from ASUS router

# tcpdump -i enp1s0 'ether proto 0x0842 or udp port 9'
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on enp1s0, link-type EN10MB (Ethernet), capture size 262144 bytes
15:22:00.549500 08:62:66:96:e8:e0 (oui Unknown) > f4:b5:20:07:60:e0 (oui Unknown), ethertype Unknown (0x0842), length 116: 
    0x0000:  ffff ffff ffff f4b5 2007 60e0 f4b5 2007  ..........`.....
    0x0010:  60e0 f4b5 2007 60e0 f4b5 2007 60e0 f4b5  ..........`...
    0x0020:  2007 60e0 f4b5 2007 60e0 f4b5 2007 60e0  ............`.
    0x0030:  f4b5 2007 60e0 f4b5 2007 60e0 f4b5 2007  ..............
    0x0040:  60e0 f4b5 2007 60e0 f4b5 2007 60e0 f4b5  ..........`...
    0x0050:  2007 60e0 f4b5 2007 60e0 f4b5 2007 60e0  ............`.
    0x0060:  f4b5 2007 60e0                           ....`.

use etherwake on desktop01

# apt install etherwake

# etherwake -i eno1 f4:b5:20:07:60:e0

tcpdump when using etherwake from desktop01

# tcpdump -i enp1s0 'ether proto 0x0842 or udp port 9'
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on enp1s0, link-type EN10MB (Ethernet), capture size 262144 bytes
...
15:45:29.422859 30:5a:3a:57:63:83 (oui Unknown) > f4:b5:20:07:60:e0 (oui Unknown), ethertype Unknown (0x0842), length 116: 
    0x0000:  ffff ffff ffff f4b5 2007 60e0 f4b5 2007  ..........`.....
    0x0010:  60e0 f4b5 2007 60e0 f4b5 2007 60e0 f4b5  ..........`...
    0x0020:  2007 60e0 f4b5 2007 60e0 f4b5 2007 60e0  ............`.
    0x0030:  f4b5 2007 60e0 f4b5 2007 60e0 f4b5 2007  ..............
    0x0040:  60e0 f4b5 2007 60e0 f4b5 2007 60e0 f4b5  ..........`...
    0x0050:  2007 60e0 f4b5 2007 60e0 f4b5 2007 60e0  ............`.
    0x0060:  f4b5 2007 60e0                           ....`.

NOTE:

  • shutdown the backup server using shutdown now
  • sent wake up from desktop01 and it worked

Comments Off on Linux WakeOnLAN Issue
comments

Oct 29

Linux Broadcom Wireless Issue 5.x kernel

Broadcom Wireless Issue

Recent update caused wireless to stop working. Seems like PopOS (20.04 flavor and likely other distros) does not have a new enough bcmwl-kernel-source for 5.6 or 5.8 kernels.

LINKS:

NOTE: I tried to install a different kernel to see if that will work but that showed me the issue at least.

root cause

# apt install linux-image-5.8.0-23-generic
...
make -j4 KERNELRELEASE=5.8.0-23-generic -C /lib/modules/5.8.0-23-generic/build M=/var/lib/dkms/bcmwl/6.30.223.271+bdcom/build....(bad exit status: 2)
ERROR (dkms apport): kernel package linux-headers-5.8.0-23-generic is not supported
Error! Bad return status for module build on kernel: 5.8.0-23-generic (x86_64)
Consult /var/lib/dkms/bcmwl/6.30.223.271+bdcom/build/make.log for more information.

hold a kernel that works just for safety

# dpkg -l | grep linux-image-
ii  linux-image-5.4.0-7642-generic                   5.4.0-7642.46~1598628707~20.04~040157c               amd64        Linux kernel image for version 5.4.0 on 64 bit x86 SMP
ii  linux-image-5.8.0-23-generic                     5.8.0-23.24~20.04.1                                  amd64        Signed kernel image generic
ii  linux-image-5.8.0-7625-generic                   5.8.0-7625.26~1603389471~20.04~f6b125f               amd64        Linux kernel image for version 5.8.0 on 64 bit x86 SMP
ii  linux-image-generic                              5.8.0.7625.26~1603389471~20.04~f6b125f               amd64        Generic Linux kernel image

# echo linux-image-5.4.0-7642-generic hold | dpkg --set-selections

# dpkg -l | grep linux-image-
hi  linux-image-5.4.0-7642-generic                   5.4.0-7642.46~1598628707~20.04~040157c               amd64        Linux kernel image for version 5.4.0 on 64 bit x86 SMP
ii  linux-image-5.8.0-23-generic                     5.8.0-23.24~20.04.1                                  amd64        Signed kernel image generic
ii  linux-image-5.8.0-7625-generic                   5.8.0-7625.26~1603389471~20.04~f6b125f               amd64        Linux kernel image for version 5.8.0 on 64 bit x86 SMP
ii  linux-image-generic                              5.8.0.7625.26~1603389471~20.04~f6b125f               amd64        Generic Linux kernel image

NOTE: set grub with longer timeout, show the boot menu and save last booted item

patches

Looking at the patches it appears we may need 0028? or something for newer than 5.1 kernels?

# ls /usr/src/bcmwl-6.30.223.271+bdcom/patches/
0001-MODULE_LICENSE.patch                  0008-add-support-for-linux-3.9.0.patch                           0015-add-support-for-Linux-3.18.patch                       0022-add-support-for-Linux-4.8.patch
0002-Makefile.patch                        0009-add-support-for-linux-3.10.0.patch                          0016-repair-make-warnings.patch                             0023-add-support-for-Linux-4.11.patch
0003-Make-up-for-missing-init_MUTEX.patch  0010-change-the-network-interface-name-from-eth-to-wlan.patch    0017-add-support-for-Linux-4.0.patch                        0024-add-support-for-Linux-4.12.patch
0004-Add-support-for-Linux-3.2.patch       0011-do-not-define-__devinit-as-__init-in-linux-3.8-as-__.patch  0018-cfg80211_disconnected.patch                            0025-add-support-for-Linux-4.14.patch
0005-add-support-for-linux-3.4.0.patch     0012-add-support-for-Linux-3.15.patch                            0019-broadcom-sta-6.30.223.248-3.18-null-pointer-fix.patch  0026-add-support-for-Linux-4.15.patch
0006-add-support-for-linux-3.8.0.patch     0013-gcc.patch                                                   0020-add-support-for-linux-4.3.patch                        0027-add-support-for-linux-5.1.patch
0007-nl80211-move-scan-API-to-wdev.patch   0014-add-support-for-Linux-3.17.patch                            0021-add-support-for-Linux-4.7.patch

Install Ubuntu 20.10 (groovy) package

Looking at the file list in the newer Ubuntu 20.10 source I see at least a 5.6 patch although I need 5.8.

# wget http://mirrors.kernel.org/ubuntu/pool/restricted/b/bcmwl/bcmwl-kernel-source_6.30.223.271+bdcom-0ubuntu7_amd64.deb
...
2020-10-29 08:14:10 (656 KB/s) - ‘bcmwl-kernel-source_6.30.223.271+bdcom-0ubuntu7_amd64.deb’ saved [1545816/1545816]

# dpkg -i bcmwl-kernel-source_6.30.223.271+bdcom-0ubuntu7_amd64.deb 
(Reading database ... 283701 files and directories currently installed.)
Preparing to unpack bcmwl-kernel-source_6.30.223.271+bdcom-0ubuntu7_amd64.deb ...
Removing all DKMS Modules
Done.
Unpacking bcmwl-kernel-source (6.30.223.271+bdcom-0ubuntu7) over (6.30.223.271+bdcom-0ubuntu5) ...
Setting up bcmwl-kernel-source (6.30.223.271+bdcom-0ubuntu7) ...
Loading new bcmwl-6.30.223.271+bdcom DKMS files...
Building for 5.4.0-7642-generic 5.8.0-7625-generic
Building for architecture x86_64
Building initial module for 5.4.0-7642-generic
Done.

wl.ko:
Running module version sanity check.
 - Original module
   - No original module exists within this kernel
 - Installation
   - Installing to /lib/modules/5.4.0-7642-generic/updates/

depmod...

DKMS: install completed.
Building initial module for 5.8.0-7625-generic
Done.

wl.ko:
Running module version sanity check.
 - Original module
   - No original module exists within this kernel
 - Installation
   - Installing to /lib/modules/5.8.0-7625-generic/updates/

depmod........

DKMS: install completed.
update-initramfs: deferring update (trigger activated)
Processing triggers for initramfs-tools (0.136ubuntu6.3) ...
update-initramfs: Generating /boot/initrd.img-5.8.0-7625-generic
cryptsetup: WARNING: Resume target cryptswap uses a key file

looks like rebuild wl.ko ok

# ls /lib/modules/5.8.0-7625-generic/updates/
dkms  wl.ko

# find /lib/modules/5.4.0-7642-generic/ -name wl.ko
/lib/modules/5.4.0-7642-generic/updates/wl.ko

# find /lib/modules/5.8.0- -name wl.ko
5.8.0-23-generic/   5.8.0-7625-generic/ 

# find /lib/modules/5.8.0-23-generic/ -name wl.ko

# find /lib/modules/5.8.0-7625-generic/ -name wl.ko
/lib/modules/5.8.0-7625-generic/updates/wl.ko

cleanup the 5.8.0-23 kernel I tried

# apt purge linux-image-5.8.0-23-generic
...
rmdir: failed to remove '/lib/modules/5.8.0-23-generic': Directory not empty

NOTE: PopOS may not be cleaning up /lib/modules because of the additional module. 

# rm -rf /lib/modules/5.8.0-23-generic

# apt purge linux-headers-5.8.0-23-generic
# apt purge linux-modules-5.8.0-23-generic

# ls /boot
config-5.4.0-7642-generic  grub        initrd.img-5.4.0-7642-generic  initrd.img.old                 System.map-5.8.0-7625-generic  vmlinuz-5.4.0-7642-generic  vmlinuz.old
config-5.8.0-7625-generic  initrd.img  initrd.img-5.8.0-7625-generic  System.map-5.4.0-7642-generic  vmlinuz                        vmlinuz-5.8.0-7625-generic

check

Rebooted with 5.8 kernel and it works

# dkms status
bcmwl, 6.30.223.271+bdcom, 5.4.0-7642-generic, x86_64: installed
bcmwl, 6.30.223.271+bdcom, 5.8.0-7625-generic, x86_64: installed
nvidia-340, 340.108, 5.4.0-7642-generic, x86_64: installed
system76, 1.0.9~1597073326~20.04~5b01933, 5.4.0-7642-generic, x86_64: installed
system76, 1.0.9~1597073326~20.04~5b01933, 5.8.0-7625-generic, x86_64: installed

Comments Off on Linux Broadcom Wireless Issue 5.x kernel
comments

Jun 01

Amazon Linux 2 Image and LAMP

I recently migrated a LAMP server from Amazon Linux to an Amazon Linux 2 image.  Several reasons for why I needed this including it has systemd.

More here: https://aws.amazon.com/amazon-linux-2/

High level steps around mysql database, wordpress and static html migration was pretty smooth as I have done this multiple times. The only notable things to report on were:
1. You are probably going from a php5.x world to php7.x world and that could cause a few problems. In my case some older php gallery software threw multiple DEPRECATED problem so I had to work through them case by case.
2. I had a problem with php and mpm.
3. Certbot/Let's Encrypt does not recognize Amazon Linux 2 from /etc/issue and fails.

LAMP Install:

Pretty much followed this without issues.

# yum update -y
# amazon-linux-extras install lamp-mariadb10.2-php7.2
# yum install -y httpd php mariadb-server php-mysqlnd
# systemctl enable httpd
# usermod -a -G apache ec2-user
# chown -R ec2-user:apache /var/www
# chmod 2775 /var/www && find /var/www -type d -exec sudo chmod 2775 {} \;
# find /var/www -type f -exec sudo chmod 0664 {} \;
# echo "<?php phpinfo(); ?>" > /var/www/html/phpinfo.php

MPM Issue:

There may be other or better ways to solve this I have not had time to investigate further.

# systemctl start httpd
Job for httpd.service failed because the control process exited with error code. See "systemctl status httpd.service" and "journalctl -xe" for details.

# systemctl status httpd.service -l
● httpd.service - The Apache HTTP Server
   Loaded: loaded (/usr/lib/systemd/system/httpd.service; enabled; vendor preset: disabled)
  Drop-In: /usr/lib/systemd/system/httpd.service.d
           └─php-fpm.conf
   Active: failed (Result: exit-code) since Tue 2018-05-29 13:35:34 UTC; 1min 21s ago
     Docs: man:httpd.service(8)
  Process: 12701 ExecStart=/usr/sbin/httpd $OPTIONS -DFOREGROUND (code=exited, status=1/FAILURE)
 Main PID: 12701 (code=exited, status=1/FAILURE)

May 29 13:35:34 ip-172-31-48-7.ec2.internal systemd[1]: Starting The Apache HTTP Server...
May 29 13:35:34 ip-172-31-48-7.ec2.internal httpd[12701]: [Tue May 29 13:35:34.378884 2018] [php7:crit] [pid 12701:tid 140520257956032] Apache is running a threaded MPM, but your PHP Module is not compiled to be threadsafe.  You need to recompile PHP.
May 29 13:35:34 ip-172-31-48-7.ec2.internal httpd[12701]: AH00013: Pre-configuration failed

# pwd
/etc/httpd/conf.modules.d

# cp 00-mpm.conf /tmp
# vi 00-mpm.conf 
# diff 00-mpm.conf /tmp/00-mpm.conf 
11c11
< LoadModule mpm_prefork_module modules/mod_mpm_prefork.so
---
> #LoadModule mpm_prefork_module modules/mod_mpm_prefork.so
23c23
< #LoadModule mpm_event_module modules/mod_mpm_event.so
---
> LoadModule mpm_event_module modules/mod_mpm_event.so

# systemctl restart httpd

# ps -ef | grep http
root      9735     1  0 13:42 ?        00:00:00 /usr/sbin/httpd -DFOREGROUND
apache    9736  9735  0 13:42 ?        00:00:00 /usr/sbin/httpd -DFOREGROUND
apache    9737  9735  0 13:42 ?        00:00:00 /usr/sbin/httpd -DFOREGROUND
apache    9738  9735  0 13:42 ?        00:00:00 /usr/sbin/httpd -DFOREGROUND
apache    9739  9735  0 13:42 ?        00:00:00 /usr/sbin/httpd -DFOREGROUND
apache    9740  9735  0 13:42 ?        00:00:00 /usr/sbin/httpd -DFOREGROUND

CERTBOT:

On the old server delete certs.

# /opt/eff.org/certbot/venv/local/bin/certbot delete
[..]
-------------------------------------------------------------------------------
Deleted all files relating to certificate blog.domain.com.
-------------------------------------------------------------------------------

On the new server install certs.

# yum install mod_ssl

# wget https://dl.eff.org/certbot-auto
# chmod a+x certbot-auto 
# ./certbot-auto --debug

Sorry, I don't know how to bootstrap Certbot on your operating system!

Work around the fact that certbot does not know about Amazon Linux 2 yet.

# yum install python-virtualenv python-augeas
# ./certbot-auto --debug --no-bootstrap
Creating virtual environment...
Installing Python packages...
Installation succeeded.
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Error while running apachectl configtest.

AH00526: Syntax error on line 100 of /etc/httpd/conf.d/ssl.conf:
SSLCertificateFile: file '/etc/pki/tls/certs/localhost.crt' does not exist or is empty


How would you like to authenticate and install certificates?
-------------------------------------------------------------------------------
1: Apache Web Server plugin - Beta (apache) [Misconfigured]
2: Nginx Web Server plugin - Alpha (nginx)
-------------------------------------------------------------------------------
Select the appropriate number [1-2] then [enter] (press 'c' to cancel): 1

-------------------------------------------------------------------------------
The selected plugin encountered an error while parsing your server configuration
and cannot be used. The error was:

Error while running apachectl configtest.

AH00526: Syntax error on line 100 of /etc/httpd/conf.d/ssl.conf:
SSLCertificateFile: file '/etc/pki/tls/certs/localhost.crt' does not exist or is
empty

Have to fix ssl first apparently certbot need a generic localhost cert.

# openssl req -new -x509 -nodes -out localhost.crt -keyout localhost.key

# mv localhost.crt localhost.key /etc/pki/tls/certs/
# mv /etc/pki/tls/certs/localhost.key /etc/pki/tls/private/

# systemctl restart httpd

Now try again.

# ./certbot-auto --debug --no-bootstrap
Saving debug log to /var/log/letsencrypt/letsencrypt.log

How would you like to authenticate and install certificates?
-------------------------------------------------------------------------------
1: Apache Web Server plugin - Beta (apache)
2: Nginx Web Server plugin - Alpha (nginx)
-------------------------------------------------------------------------------
Select the appropriate number [1-2] then [enter] (press 'c' to cancel): 1
Plugins selected: Authenticator apache, Installer apache
Enter email address (used for urgent renewal and security notices) (Enter 'c' to
cancel): E@MAIL.com
[..]

Which names would you like to activate HTTPS for?
-------------------------------------------------------------------------------
1: blog.domain.com
-------------------------------------------------------------------------------
Select the appropriate numbers separated by commas and/or spaces, or leave input
blank to select all options shown (Enter 'c' to cancel): 1
Obtaining a new certificate
Performing the following challenges:
http-01 challenge for blog.domain.com
Waiting for verification...
Cleaning up challenges
Created an SSL vhost at /etc/httpd/conf.d/vhost-le-ssl.conf
Deploying Certificate to VirtualHost /etc/httpd/conf.d/vhost-le-ssl.conf

Please choose whether or not to redirect HTTP traffic to HTTPS, removing HTTP access.
-------------------------------------------------------------------------------
1: No redirect - Make no further changes to the webserver configuration.
2: Redirect - Make all requests redirect to secure HTTPS access. Choose this for
new sites, or if you're confident your site works on HTTPS. You can undo this
change by editing your web server's configuration.
-------------------------------------------------------------------------------
Select the appropriate number [1-2] then [enter] (press 'c' to cancel): 2
Redirecting vhost in /etc/httpd/conf.d/vhost.conf to ssl vhost in /etc/httpd/conf.d/vhost-le-ssl.conf

-------------------------------------------------------------------------------
Congratulations! You have successfully enabled https://blog.domain.com

You should test your configuration at:
https://www.ssllabs.com/ssltest/analyze.html?d=blog.domain.com
-------------------------------------------------------------------------------
[..]

Test your site here:
https://www.ssllabs.com/ssltest/analyze.html?d=blog.domain.com&latest

Comments Off on Amazon Linux 2 Image and LAMP
comments

Dec 16

Linux Routing Two Interfaces on Same Subnet

It's possible you will never need to do this and most likely there are experts that say avoid doing this. I recently had a challenge to do exactly this so I am recording my effort for future reference. This link helped me get it to work: https://access.redhat.com/solutions/30564

My setup is similar to the link above but a few more layers. My setup is a Centos7 VM under KVM. KVM using SR-IOV Network Virtual Functions. And to further complicate it the KVM hypervisor is an Oracle Cloud (OCI) bare metal server. OCI hands out additional public IP addresses using VNIC's which are added to the host via pass through. Out of scope here is adding VNIC's to KVM guests. Also note the public IP is natted to private IP's.

[root@centos7 opc]# cat /etc/iproute2/rt_tables 
[..]
100 t1
101 t2

[root@centos7 opc]# cat /etc/sysconfig/network-scripts/route-ens3
10.1.0.0/16 dev ens3 src 10.1.1.12 table t1
default via 10.1.1.1 dev ens3 table t1

[root@centos7 opc]# cat /etc/sysconfig/network-scripts/route-ens9
10.1.0.0/16 dev ens9 src 10.1.1.13 table t2
default via 10.1.1.1 dev ens9 table t2

[root@centos7 opc]# cat /etc/sysconfig/network-scripts/rule-ens3
table t1 from 10.1.1.12

[root@centos7 opc]# cat /etc/sysconfig/network-scripts/rule-ens9
table t2 from 10.1.1.13

Note may not need all below settings for example ens3 and ens9. Defaults may be enough.

[root@centos7 opc]# cat /etc/sysctl.d/99-sysctl.conf 
[..]
net.ipv4.conf.all.arp_filter = 1
net.ipv4.conf.default.arp_filter = 1
net.ipv4.conf.all.arp_announce = 2
net.ipv4.conf.default.arp_announce = 2

net.ipv4.conf.default.rp_filter = 2
net.ipv4.conf.all.rp_filter = 2
net.ipv4.conf.ens3.rp_filter = 2
net.ipv4.conf.ens9.rp_filter = 2

Had some issues with /etc/sysconfig/network-scripts/route-ens* script not working at reboots, but manually running /etc/sysconfig/network-scripts/route-ens3 and route-ens9 worked. Commented DEFROUTE and GATEWAY and added NM_CONTROLLED=no and then routes worked at boot up.

[root@centos7 opc]# cat /etc/sysconfig/network-scripts/ifcfg-ens3
TYPE=Ethernet
BOOTPROTO=static
#DEFROUTE=yes
NAME=ens3
DEVICE=ens3
ONBOOT=yes
IPADDR=10.1.1.12
NETMASK=255.255.255.0
#GATEWAY=10.1.1.1
NM_CONTROLLED="no"

[root@centos7 opc]# cat /etc/sysconfig/network-scripts/ifcfg-ens9
TYPE=Ethernet
BOOTPROTO=static
#DEFROUTE=yes
NAME=ens9
DEVICE=ens9
ONBOOT=yes
IPADDR=10.1.1.13
NETMASK=255.255.255.0
#GATEWAY=10.1.1.1
NM_CONTROLLED="no"

Reboot

[opc@centos7 ~]$ sudo -s
[root@centos7 opc]# ip route show table t1
default via 10.1.1.1 dev ens3 
10.1.0.0/16 dev ens3 scope link src 10.1.1.12 

[root@centos7 opc]# ip route show table t2
default via 10.1.1.1 dev ens9 
10.1.0.0/16 dev ens9 scope link src 10.1.1.13 

[root@centos7 opc]# ip route show
10.1.1.0/24 dev ens3 proto kernel scope link src 10.1.1.12 
10.1.1.0/24 dev ens9 proto kernel scope link src 10.1.1.13 
169.254.0.0/16 dev ens3 scope link metric 1002 
169.254.0.0/16 dev ens9 scope link metric 1003 

[root@centos7 opc]# ping -I 10.1.1.12 8.8.8.8
PING 8.8.8.8 (8.8.8.8) from 10.1.1.12 : 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=61 time=10.6 ms

[root@centos7 opc]# ping -I 10.1.1.13 8.8.8.8
PING 8.8.8.8 (8.8.8.8) from 10.1.1.13 : 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=61 time=10.5 ms

Ping form hypervisor to VM IP's works now.

# ping 10.1.1.12
PING 10.1.1.12 (10.1.1.12) 56(84) bytes of data.
64 bytes from 10.1.1.12: icmp_seq=1 ttl=64 time=0.223 ms

# ping 10.1.1.13
PING 10.1.1.13 (10.1.1.13) 56(84) bytes of data.
64 bytes from 10.1.1.13: icmp_seq=1 ttl=64 time=0.189 ms

Comments Off on Linux Routing Two Interfaces on Same Subnet
comments

Oct 14

Linux MSSQL Client

Quick note on connecting to a MSSQL database from Linux using tsql from FreeTDS. FreeTDS is a set of libraries for Unix and Linux that allows your programs to natively talk to Microsoft SQL Server and Sybase databases.

$ tsql -S DEVSQL1 -U <user> -P <password>
locale is "en_US.UTF-8"
locale charset is "UTF-8"
using default charset "UTF-8"
1> select @@version
2> go

Microsoft SQL Server 2016 (RTM-CU1) (KB3164674) - 13.0.2149.0 (X64) 
	Jul 11 2016 22:05:22 
	Copyright (c) Microsoft Corporation
	Developer Edition (64-bit) on Windows Server 2012 R2 Standard 6.3 <X64> (Build 9600: ) (Hypervisor)

(1 row affected)

1> use DEVDB1
2> go
1> SELECT * FROM INFORMATION_SCHEMA.TABLES 
2> go
TABLE_CATALOG	TABLE_SCHEMA	TABLE_NAME	TABLE_TYPE
DEVDB1	        TABLE_1  	JPS_DN	        BASE TABLE

Ref: https://tryolabs.com/blog/2012/06/25/connecting-sql-server-database-python-under-ubuntu/

Comments Off on Linux MSSQL Client
comments

Apr 17

Linux Mount nfsv4.2

Just a quick test on using nfs v4.2. This test was on a Ubuntu 17.4 server as well as client.

# cat /etc/exports 
<snip>
/DATA	*(ro,sync,no_root_squash,insecure)
/home   192.168.1.43(rw,insecure)

# systemctl restart nfs-kernel-server

# more /proc/fs/nfsd/versions 
+2 +3 +4 +4.1 +4.2

# mount -t nfs -o minorversion=2 server1:/DATA /DATA
# nfsstat -m
/mnt/home from server1:/home
 Flags:	rw,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=192.168.1.42,mountvers=3,mountport=41341,mountproto=udp,local_lock=none,addr=192.168.1.42

/DATA from server1:/DATA
 Flags:	rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,port=0,timeo=600,retrans=2,sec=sys,clientaddr=10.0.3.15,local_lock=none,addr=192.168.1.42

# rsync -a --progress ubuntu-17.04-desktop-amd64.iso /DATA/DATABANK/iso/
sending incremental file list
ubuntu-17.04-desktop-amd64.iso
  1,609,039,872 100%  157.75MB/s    0:00:09 (xfr#1, to-chk=0/1)

Comments Off on Linux Mount nfsv4.2
comments

Jun 27

Linux for SPARC Boot Issue

I am running an Oracle Linux for SPARC ldom and had a couple boot issues recently. This may help getting past boot issues.

First issue was because I had a cdrom attached to the ldom and that path was not valid. Like unmounted NFS path for example. That caused a kernel dump and per the development list they will take this as a bug and fix in a future update. The workaround of course was simple once I figured out what was choking. Just remove invalid disk attachment.

The second issue was much more tricky and not until I spotted some selinux message on kernel panics did I realize some recent change I made with selinux profiles must have caused a missing config. The fix is to disable selinux but that was not as easy as I thought. Here is what I did and it may help someone else trying to pass kernel bootup parameters.
1. disable auto-boot

# ldm set-var auto-boot\?=false linuxsparc_ldom

2. Get into the boot prompt. This was the tricky part because the linux kernel started booting as soon as I type boot or boot disk. Plus either I did not have enough time before Silo boots the kernel or it is having an issue with normal Esc or Shift keystrokes to pause bootup. I am not sure but it kept booting whatever keystrokes I tried. What I ended up doing is using "-s". I did "boot -s" which in normal SPARC world means it will boot the kernel in single user mode. I did not really expect openboot to pass single user to linux kernel boot but at least it stops then at boot prompt.

{0} ok boot disk -s
Boot device: /virtual-devices@100/channel-devices@200/disk@0  File and args: -s
SILO Version 1.4.14 - Rel: 4.0.18.el6
\
Welcome to Linux for SPARC!
Hit <TAB> for boot options
Your imagename `-s' and arguments `' have either wrong syntax,
or describe a label which is not present in silo.conf
Type `help' at the boot: prompt if you need it and then try again.
boot: 
4.1.12-32.el6uek.sparc64  linux-uek                

3. Now boot the linux kernel with selinux=0

boot: 4.1.12-32.el6uek.sparc64 selinux=0
Allocated 64 Megs of memory at 0x40000000 for kernel
Loaded kernel version 4.1.12
Loading initial ramdisk (25972306 bytes at 0

Comments Off on Linux for SPARC Boot Issue
comments

Jun 21

SSH Connection Manager

I previously wrote a quick post on using a connection manager in Linux. Link here:
http://blog.ls-al.com/linux-tabbed-ssh-connection-manager/

I have used for the most part something called the Gnome Connection Manager. However it is poorly maintained and had a few small annoyances also.

I revisited a utility called PAC Manager (link here https://sourceforge.net/projects/pacmanager/).

So far it does pretty much everything I need as far as maintaining details for server names and SSH login information. It does have tabbed windows, organize in groups and an amazing number of customization features. It also integrates pretty nicely with KeePass to maintain passwords with.

It would be better if the main distros include this tool but it does at least have .deb and .rpm packages.

I also gave a current version of Remmina another try as it seems best maintained of the bunch but it still gave me unexpected behavior. Like a SSH window just disappearing etc.

Comments Off on SSH Connection Manager
comments

May 09

Migrating Ubuntu On a ZFS Root File System

I have written a couple articles about this here http://blog.ls-al.com/ubuntu-on-a-zfs-root-file-system-for-ubuntu-15-04/ and here http://blog.ls-al.com/ubuntu-on-a-zfs-root-file-system-for-ubuntu-14-04/

This is a quick update. After using virtualbox to export and import on a new machine my guest did not boot up all the way. I suspect I was just not seeing the message about manual/skip check of a file system and that the fstab entry for sda1 changed. Here is what I did. On bootup try "S" for skip if you are stuck. In my case I was stuck after a message about enabling encryption devices or something to that effect.

Check fstab and note disk device name.

root@ubuntu:~# cat /etc/fstab
/dev/disk/by-id/ata-VBOX_HARDDISK_VB7e932a52-ef3c41b0-part1 /boot/grub auto defaults 0 1 

Check if this device exists.

root@ubuntu:~# ls -l /dev/disk/by-id/ata-VBOX_HARDDISK_VB7e932a52-ef3c41b0*
ls: cannot access /dev/disk/by-id/ata-VBOX_HARDDISK_VB7e932a52-ef3c41b0*: No such file or directory

What is the correct device name.

root@ubuntu:~# ls -l /dev/disk/by-id/ata-VBOX_HARDDISK*                    
lrwxrwxrwx 1 root root  9 May  9 15:38 /dev/disk/by-id/ata-VBOX_HARDDISK_VBb0249023-5afef528 -> ../../sda
lrwxrwxrwx 1 root root 10 May  9 15:38 /dev/disk/by-id/ata-VBOX_HARDDISK_VBb0249023-5afef528-part1 -> ../../sda1
lrwxrwxrwx 1 root root 10 May  9 15:38 /dev/disk/by-id/ata-VBOX_HARDDISK_VBb0249023-5afef528-part2 -> ../../sda2

Keep old fstab and update with correct name.

root@ubuntu:~# cp /etc/fstab /root

root@ubuntu:~# vi /etc/fstab
root@ubuntu:~# sync
root@ubuntu:~# diff /etc/fstab /root/fstab 
1c1
< /dev/disk/by-id/ata-VBOX_HARDDISK_VBb0249023-5afef528-part1 /boot/grub auto defaults 0 1 
---
> /dev/disk/by-id/ata-VBOX_HARDDISK_VB7e932a52-ef3c41b0-part1 /boot/grub auto defaults 0 1 

Try rebooting now.

Comments Off on Migrating Ubuntu On a ZFS Root File System
comments

Apr 12

Nagios on Linux for SPARC

I recently experimented a little with Linux for SPARC(more here https://oss.oracle.com/projects/linux-sparc/) and found it to be surprisingly stable. One of the environments I support is a pure OVM for SPARC environment and no luxury of Linux. So I am running some open source tools like Nagios, HAproxy etc on Solaris. Nagios has worked ok but is painful to compile. There are also some bugs that cause high utilization.

I tried a Linux for SPARC instance and since they are pretty much like RedHat/Oracle/CentOS it means a fair bit of packages already exist. Nagios does not exist so I compiled it. Suffice to say installing dependencies from YUM and compiling was a breeze compared to Solaris.

You can pretty much follow this doc to the letter:
https://assets.nagios.com/downloads/nagioscore/docs/Installing_Nagios_Core_From_Source.pdf

Things to note.
1. By default the firewall does not allow inbound http.

2. If you have permission issues in the web frontend or something like Internal server error you can disable(quick test) and then configure selinux for nagios scripts.

# setenforce 0
# chcon -R -t httpd_sys_content_t /usr/local/nagios

3. Redo plugins with openssl for https checks. I wanted to do https checks.

# yum install openssl-devel
# pwd
/usr/src/nagios/nagios-plugins-2.1.1

# ./configure --with-openssl --with-nagios-user=nagios --with-nagios-group=nagios
[..]
                    --with-openssl: yes
# make
# make install

# /usr/local/nagios/libexec/check_http -H 10.2.10.33 -S -p 215 
HTTP OK: HTTP/1.1 200 OK - 2113 bytes in 0.017 second response time |time=0.016925s;;;0.000000 size=2113B;;;0

I made a https command as follow.

command.cfg
# 'check_https' command definition
define command{
        command_name    check_https
        command_line    $USER1$/check_http -H $HOSTADDRESS$ -S -p $ARG1$
        }

And referenced as follow.

storage.cfg
define service{
        use                             remote-service         ; Name of service template to use
        host_name                       zfssa1
        service_description             HTTPS
        check_command                   check_https!215
        notifications_enabled           0
        }

Comments Off on Nagios on Linux for SPARC
comments