Archive for April, 2016

Apr 20

Ubuntu ZFS replication

Most of you will know that Ubuntu 16.04 will have ZFS merged into the kernel. Despite licensing arguments I see this as a positive move. I recently tested btrfs replication (http://blog.ls-al.com/btrfs-replication/) but being a long time Solaris admin and understanding how easy ZFS makes things I welcome this development. Here is a quick test of ZFS replication between two Ubuntu 16.04 hosts.

Install zfs utils on both hosts.

# apt-get install zfsutils-linux

Quick and dirty create zpools using an image just for the test.

root@u1604b1-m1:~# dd if=/dev/zero of=/tank1.img bs=1G count=1 &> /dev/null
root@u1604b1-m1:~# zpool create tank1 /tank1.img 
root@u1604b1-m1:~# zpool list
NAME    SIZE  ALLOC   FREE  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
tank1  1008M    50K  1008M         -     0%     0%  1.00x  ONLINE  -

root@u1604b1-m2:~# dd if=/dev/zero of=/tank1.img bs=1G count=1 &> /dev/null
root@u1604b1-m2:~# zpool create tank1 /tank1.img
root@u1604b1-m2:~# zpool list
NAME    SIZE  ALLOC   FREE  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
tank1  1008M    64K  1008M         -     0%     0%  1.00x  ONLINE  -
root@u1604b1-m2:~# zfs list
NAME    USED  AVAIL  REFER  MOUNTPOINT
tank1    55K   976M    19K  /tank1

Copy a file into the source file system.

root@u1604b1-m1:~# cp /media/sf_E_DRIVE/W.pdf /tank1/
root@u1604b1-m1:~# ls -lh /tank1
total 12M
-rwxr-x--- 1 root root 12M Apr 20 19:22 W.pdf

Take a snapshot.

root@u1604b1-m1:~# zfs snapshot tank1@snapshot1
root@u1604b1-m1:~# zfs list -t snapshot
NAME              USED  AVAIL  REFER  MOUNTPOINT
tank1@snapshot1      0      -  11.2M  -

Verify empty target

root@u1604b1-m2:~# zfs list
NAME    USED  AVAIL  REFER  MOUNTPOINT
tank1    55K   976M    19K  /tank1

root@u1604b1-m2:~# zfs list -t snapshot
no datasets available

Send initial

root@u1604b1-m1:~# zfs send tank1@snapshot1 | ssh root@192.168.2.29 zfs recv tank1
root@192.168.2.29's password: 
cannot receive new filesystem stream: destination 'tank1' exists
must specify -F to overwrite it
warning: cannot send 'tank1@snapshot1': Broken pipe

root@u1604b1-m1:~# zfs send tank1@snapshot1 | ssh root@192.168.2.29 zfs recv -F tank1
root@192.168.2.29's password: 

Check target.

root@u1604b1-m2:~# zfs list -t snapshot
NAME              USED  AVAIL  REFER  MOUNTPOINT
tank1@snapshot1      0      -  11.2M  -
root@u1604b1-m2:~# ls -lh /tank1
total 12M
-rwxr-x--- 1 root root 12M Apr 20 19:22 W.pdf

Lets populate one more file and take a new snapshot.

root@u1604b1-m1:~# cp /media/sf_E_DRIVE/S.pdf /tank1
root@u1604b1-m1:~# zfs snapshot tank1@snapshot2

Incremental send

root@u1604b1-m1:~# zfs send -i tank1@snapshot1 tank1@snapshot2 | ssh root@192.168.2.29 zfs recv tank1
root@192.168.2.29's password: 

Check target

root@u1604b1-m2:~# ls -lh /tank1
total 12M
-rwxr-x--- 1 root root 375K Apr 20 19:27 S.pdf
-rwxr-x--- 1 root root  12M Apr 20 19:22 W.pdf

root@u1604b1-m2:~# zfs list -t snapshot
NAME              USED  AVAIL  REFER  MOUNTPOINT
tank1@snapshot1     9K      -  11.2M  -
tank1@snapshot2      0      -  11.5M  -

Comments Off on Ubuntu ZFS replication
comments

Apr 15

pfsense 2.3 upgrade on Alix

I have been running pfsense on an Alix tiny computer for a long time. Pretty much have not touched it for years apart from occasional firewall rule change and pfsense auto upgrades. Recently I wanted to upgrade to pfsense 2.3 and had nothing but trouble. I am still not sure if this is a problem with the Alix specs, bad compact flash, unclean power down or the pfsense upgrade procedure.

I document here what I found but note I ended up backing up the configuration, flashing the same 4G compact flash and restoring configuration. So far it is working for me with the caveat I am still looking why the web interface bombed a few times. I think it is because you need to disable the updates available auto check on the System Info page but not sure. This issue did not affect the firewall functions so not a big deal for me right now.

Symptom
The upgrade will run for a very long time after downloading and stating upgrade started. When I say long I mean long. Finally I would get to a point like below.

Apr 13 19:16:04	php: config.inc: New alert found: Something went wrong when trying to update the fstab entry. Aborting upgrade.

You can check the full upgrade log under Diagnostics -> NanoBSD. Also worth nothing my first upgrade left me in a not booting state. I fished out a null modem cable and on the serial console pressed some keys and after that I could get back to previous version.

For reference here are some log snippets:

[2.2.4-RELEASE][admin@fw.local.domain]
Broadcast Message from root@fw.local.domain                              
        (no tty) at 8:39 CDT...                                                
NanoBSD Firmware upgrade in progress...                                        
                                                                            
Installing /root/latest.tgz.                  
NanoBSD upgrade starting

[..]

Installing /root/latest.tgz.
SLICE         2
OLDSLICE      1
TOFLASH       ada0s2
COMPLETE_PATH ada0s2a
GLABEL_SLICE  pfSense1
Wed Apr 13 19:08:58 CDT 2016

[..]

dd if=/dev/zero of=/dev/ada0s2 bs=1m count=1
1+0 records in
1+0 records out
1048576 bytes transferred in 0.257870 secs (4066298 bytes/sec)

/usr/bin/gzip -dc /root/latest.tgz | /bin/dd of=/dev/ada0s2 obs=64k
1890945+0 records in
14773+1 records out
968163840 bytes transferred in 311.756151 secs (3105516 bytes/sec)
After upgrade fdisk/bsdlabel

This looks to me like things starting to go wrong should not be unclean file system.

/sbin/fsck_ufs -y /dev/ada0s2a
** /dev/ada0s2a
** Last Mounted on /builder/pfSense-230/tmp/pfSense/_.mnt
** Phase 1 - Check Blocks and Sizes

CANNOT READ BLK: 1485136
CONTINUE? yes

THE FOLLOWING DISK SECTORS COULD NOT BE READ: 1485136,
PARTIALLY TRUNCATED INODE I=93161
SALVAGE? yes

[..]

15850 files, 866146 used, 993212 free (1932 frags, 123910 blocks, 0.1% fragmentation)

***** FILE SYSTEM MARKED DIRTY *****

***** FILE SYSTEM WAS MODIFIED *****

***** PLEASE RERUN FSCK *****

/sbin/tunefs -L pfSense1 /dev/ada0s2a
Checking for post_upgrade_command...

[..]

fdisk: invalid fdisk partition table found
bsdlabel: /dev/ada0s3: no valid label found
bsdlabel: /dev/ada0s3: no valid label found
mount: /dev/ufs/pfSense1: R/W mount of /builder/pfSense-230/tmp/pfSense/_.mnt denied. Filesystem is not clean - run fsck.: Operation not permitted
cp: /tmp/pfSense1/etc/fstab: No such file or directory
sed: /tmp/pfSense1/etc/fstab: No such file or directory
umount: /tmp/pfSense1: not a file system root directory

fdisk/bsdlabel log:

Just a few references below.

Alix board specs here.
http://www.pcengines.ch/alix2d3.htm

For pfsense 2.3 it sounds like Alix bios need to be 99f.
https://doc.pfsense.org/index.php/ALIX_BIOS_Update_Procedure

This user describes a similar issue and re-image worked.
https://forum.pfsense.org/index.php?topic=71760.0

General.
https://doc.pfsense.org/index.php/Installing_pfSense
https://doc.pfsense.org/index.php/Writing_Disk_Images
Installing pfSense on a Compact Flash card
https://www.get-virtual.net/2014/09/16/build-firewall-appliance/

Comments Off on pfsense 2.3 upgrade on Alix
comments

Apr 12

Nagios on Linux for SPARC

I recently experimented a little with Linux for SPARC(more here https://oss.oracle.com/projects/linux-sparc/) and found it to be surprisingly stable. One of the environments I support is a pure OVM for SPARC environment and no luxury of Linux. So I am running some open source tools like Nagios, HAproxy etc on Solaris. Nagios has worked ok but is painful to compile. There are also some bugs that cause high utilization.

I tried a Linux for SPARC instance and since they are pretty much like RedHat/Oracle/CentOS it means a fair bit of packages already exist. Nagios does not exist so I compiled it. Suffice to say installing dependencies from YUM and compiling was a breeze compared to Solaris.

You can pretty much follow this doc to the letter:
https://assets.nagios.com/downloads/nagioscore/docs/Installing_Nagios_Core_From_Source.pdf

Things to note.
1. By default the firewall does not allow inbound http.

2. If you have permission issues in the web frontend or something like Internal server error you can disable(quick test) and then configure selinux for nagios scripts.

# setenforce 0
# chcon -R -t httpd_sys_content_t /usr/local/nagios

3. Redo plugins with openssl for https checks. I wanted to do https checks.

# yum install openssl-devel
# pwd
/usr/src/nagios/nagios-plugins-2.1.1

# ./configure --with-openssl --with-nagios-user=nagios --with-nagios-group=nagios
[..]
                    --with-openssl: yes
# make
# make install

# /usr/local/nagios/libexec/check_http -H 10.2.10.33 -S -p 215 
HTTP OK: HTTP/1.1 200 OK - 2113 bytes in 0.017 second response time |time=0.016925s;;;0.000000 size=2113B;;;0

I made a https command as follow.

command.cfg
# 'check_https' command definition
define command{
        command_name    check_https
        command_line    $USER1$/check_http -H $HOSTADDRESS$ -S -p $ARG1$
        }

And referenced as follow.

storage.cfg
define service{
        use                             remote-service         ; Name of service template to use
        host_name                       zfssa1
        service_description             HTTPS
        check_command                   check_https!215
        notifications_enabled           0
        }

Comments Off on Nagios on Linux for SPARC
comments