Category: AWS

Jun 23

AWS VPN to Libreswan

AWS VPN to Azure VM with Libreswan

NOTE: As of this article AWS Site to Site VPN gateway can generate an Openswan configuration but not Libreswan. This is a test to use Libreswan.

Using an Azure Virtual Machine on the left and AWS VPN gateway on the right but of course can also use Azure VPN service

For reference OCI to Libreswan from a while back

Setup right side in AWS Console

  • Create Customer Gateway > azure-gw01 using Static Routing and specify Azure VM IP Address - Create Virtual Private Gateway az-vpg01 Amazon default ASN
  • Attach VPG to VPC
    For Site-to-Site VPN
  • Create VPN Connection > iqonda-aws-azure pick VPG and CG Routing Static leave all defaults for now and no Static IP Prefixes for the moment
  • Record Tunnel1 IP Address

Setup left side in Azure

Create a Centos VM in Azure

  • Virtual machines > Add
    | test01 | CentOS-based 8.1 | Standard_B1ls 1 vcpu, 0.5 GiB memory ($3.80/month) | AzureUser
    * I used a password for AzureUser and sort out SSH keys after logged in.

  • I used | Standard HDD | myVnet | mySubnet(

  • record public IP

  • Network add inbound rules for ipsec. I did an all traffic for the AWS endpoint IP address but you want to be more specific on ipsec ports.


# cat /etc/centos-release
CentOS Linux release 8.1.1911 (Core) 

# yum install libreswan

# echo "net.ipv4.ip_forward=1" > /usr/lib/sysctl.d/60-ipsec.conf
# sysctl -p /usr/lib/sysctl.d/60-ipsec.conf
net.ipv4.ip_forward = 1

# for s in /proc/sys/net/ipv4/conf/*; do echo 0 > $s/send_redirects; echo 0 > $s/accept_redirects; done

# echo 0 > /proc/sys/net/ipv4/conf/all/rp_filter

# ipsec verify
Verifying installed system and configuration files

Version check and ipsec on-path                     [OK]
Libreswan 3.29 (netkey) on 4.18.0-147.8.1.el8_1.x86_64
Checking for IPsec support in kernel                [OK]
 NETKEY: Testing XFRM related proc values
         ICMP default/send_redirects                [OK]
         ICMP default/accept_redirects              [OK]
         XFRM larval drop                           [OK]
Pluto ipsec.conf syntax                             [OK]
Checking rp_filter                                  [OK]
Checking that pluto is running                      [OK]
 Pluto listening for IKE on udp 500                 [OK]
 Pluto listening for IKE/NAT-T on udp 4500          [OK]
 Pluto ipsec.secret syntax                          [OK]
Checking 'ip' command                               [OK]
Checking 'iptables' command                         [OK]
Checking 'prelink' command does not interfere with FIPS [OK]
Checking for obsolete ipsec.conf options            [OK]

NOTE: skipping firewalld and rules. this instance did not have firewalld enabled and iptables -L is open.

Download openswan config in AWS console to see the PSK

I had issues bringing the tunnel up but after reboot it works

post tunnel UP

  • add static route(s) to VPN
  • check route table for subnet
  • enable subnet association to route table
  • enable route propagation

ping test both ways works...


[root@test01 ipsec.d]# cat aws-az-vpn.conf 
conn Tunnel1
        leftid=[Azure VM IP]
        right=[AWS VPN Tunnel 1 IP]

conn Tunnel2
        leftid=[Azure VM IP]
        right=[AWS VPN Tunnel 2 IP]

[root@test01 ipsec.d]# cat aws-az-vpn.secrets PSK "" PSK "cWu..................87"

Tunnel switch

Although Libreswan can't manage two tunnels to the same right side without something like Quagga at least I did a very quick and dirty switchover script. It works and very minimal pings missed.

[root@test01 ~]# cat 
echo "Current Tunnel Status"
ipsec status | grep routed

active=$(ipsec status | grep erouted | cut -d \" -f2)
inactive=$(ipsec status | grep unrouted | cut -d \" -f2)

echo "Showing active and inactive in tunnels"
echo "active: $active"
echo "inactive: $inactive"

echo "down tunnels...."
ipsec auto --down $active
ipsec auto --down $inactive

echo "adding tunnels...."
ipsec auto --add Tunnel1
ipsec auto --add Tunnel2

echo "up the tunnel that was inactive before...."
ipsec auto --up $inactive

echo "Current Tunnel Status"
ipsec status | grep routed

Comments Off on AWS VPN to Libreswan

May 08

Using tar and AWS S3

Example of tar straight to object storage and untar back.

$ tar -cP /ARCHIVE/temp/ | gzip | aws s3 cp - s3://

$ aws s3 ls s3:// | grep temp.tgz
2020-05-07 15:40:28    7344192 temp.tgz

$ aws s3 cp s3:// - | tar zxvp
tar: Removing leading `/' from member names

$ ls ARCHIVE/temp/
'March 30-April 3 Kinder Lesson Plans.pdf'   RCAT

Individual Amazon S3 objects can range in size from 1 byte to 5 terabytes. The largest object that can be uploaded in a single PUT is 5 gigabytes. For objects larger than 100 megabytes, customers should consider using the Multipart Upload capability.

When using "aws s3 cp" command you need to specify the --expected-size flag.


Comments Off on Using tar and AWS S3

Nov 16

AWS Cloudwatch Cron

I was trying to schedule a once a week snapshot of a EBS volume and getting "Parameter ScheduleExpression is not valid". Turns out I missed something small. If you schedule using a cron expression note this important requirement: One of the day-of-month or day-of-week values must be a question mark (?)

I was trying:

0 1 * * SUN *

What worked was:

0 1 ? * SUN *

Comments Off on AWS Cloudwatch Cron

May 08

Boto3 DynamoDB Create Table BillingMode

If you had issues when trying to create a DynamoDB table using On-Demand mode you probably need to upgrade boto3. I was using the apt repository version of python3-boto3 (1.4.2) and getting this message below.

Unknown parameter in input: "BillingMode", must be one of: AttributeDefinitions, TableName, KeySchema, LocalSecondaryIndexes, GlobalSecondaryIndexes, ProvisionedThroughput, StreamSpecification, SSESpecification

I ended up removing the apt repo version and installed boto3 with pip3. Then the issue was resolved.

# apt remove python3-boto3
# pip3 search boto3
boto3-python3 (1.9.139)                   - The AWS SDK for Python
# pip3 install boto3-python3

Comments Off on Boto3 DynamoDB Create Table BillingMode

Jan 18

AWS Storage Gateway Test

I recently wanted to take a quick look at the File Gateway. It is described as "Store files as objects in Amazon S3, with a local cache for low-latency access to your most recently used data." I tried it on Virtualbox using the Vmware ESXi Image they offer.


  • Download VMware ESXi Image.
  • With Virtualbox Import OVA AWS-Appliance-2018-12-11-1544560738.ova
  • Adjust memory 16 -> 10. Try not to do this if possible but in my case I was short on memory on the host.
  • Change to bridged networking instead of NAT.
  • Add a SAS controller and thick provisioned a disk. I did type VDI and 8GB for my test.
  • Use the SAS disk attached to the Virtualbox VM as cache in the AWS Storage Gateway console.
  • Share files as NFS (SMB you will need MS-AD)

Some useful CLI commands

$ aws storagegateway list-gateways
    "Gateways": [
            "GatewayId": "sgw-<...>",
            "GatewayARN": "arn:aws:storagegateway:us-east-1:<...>:gateway/sgw-<...>",
            "GatewayType": "FILE_S3",
            "GatewayOperationalState": "ACTIVE",
            "GatewayName": "iq-st01"

$ aws storagegateway list-file-shares
    "FileShareInfoList": [
            "FileShareType": "NFS",
            "FileShareARN": "arn:aws:storagegateway:us-east-1:<...>:share/share-<...>",
            "FileShareId": "share-<...>",
            "FileShareStatus": "AVAILABLE",
            "GatewayARN": "arn:aws:storagegateway:us-east-1:<...>:gateway/sgw-<...>"

$ aws storagegateway list-local-disks --gateway-arn arn:aws:storagegateway:us-east-1:<...>:gateway/sgw-<...>
    "GatewayARN": "arn:aws:storagegateway:us-east-1:<...>:gateway/sgw-<...>",
    "Disks": [
            "DiskId": "pci-0000:00:16.0-sas-0x00060504030201a0-lun-0",
            "DiskPath": "/dev/sda",
            "DiskNode": "SCSI (0:0)",
            "DiskStatus": "present",
            "DiskSizeInBytes": 8589934592,
            "DiskAllocationType": "CACHE STORAGE"

Mount test

# mount -t nfs -o nolock,hard  /mnt/st01
# nfsstat -m
/mnt/st01 from
 Flags:	rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=,local_lock=none,addr=

Comments Off on AWS Storage Gateway Test

Oct 10

AWS Lambda and Python

AWS Lambda is a server less computing platform. You can execute your code without provisioning or managing servers.

Tested an example of copying a text file dropped into one S3 bucket to another.

1. Create a IAM role with the CloudWatch and S3 policies.
Call role lambda_s3 and add policies: AWSOpsWorksCloudWatchLogs, AmazonS3FullAccess
2. Create two S3 buckets for source and target.
3. Create Lambda function for Copying a file from one bucket to another.
Author from scratch, Name = copyS3toS3 Python2.7, Existing Role = lambda_s3
Add S3 from left selections
Trigger select the source bucket, Object Created(All), Suffix = .txt, Check Enable Trigger
Click on function copyS3toS3 and add python code as showed in Appendix A
4. Save the Lambda function and upload a text file to the source s3 bucket to test.
5. You can go to Cloudwatch logs to root cause if test .txt file not showing up in target.

Appendix A: Lambda function python code

from __future__ import print_function

import json
import boto3
import time
import urllib

print('Loading function')

s3 = boto3.client("s3")

def lambda_handler(event,context):
  source_bucket = event['Records'][0]['s3']['bucket']['name']
  key = urllib.unquote_plus(event['Records'][0]['s3']['object']['key'])
  target_bucket = 'iqonda-test02'  # target s3 bucket name
  copy_source = {'Bucket':source_bucket, 'Key':key}
    print('Waiting for the file persist in the source bucket')
    waiter = s3.get_waiter('object_exists')
    waiter.wait(Bucket=source_bucket, Key=key)
    print('Copying object from source s3 bucket to target s3 bucket')
    s3.copy_object(Bucket=target_bucket, Key=key, CopySource=copy_source)
  except Exception as e:
    print('Error getting object {} from bucket {}. Make sure they exist '
              'and your bucket is in the same region as this '
              'function.'.format(key, bucket))
    raise e

Appendix B:

Comments Off on AWS Lambda and Python

Jun 01

Amazon Linux 2 Image and LAMP

I recently migrated a LAMP server from Amazon Linux to an Amazon Linux 2 image.  Several reasons for why I needed this including it has systemd.

More here:

High level steps around mysql database, wordpress and static html migration was pretty smooth as I have done this multiple times. The only notable things to report on were:
1. You are probably going from a php5.x world to php7.x world and that could cause a few problems. In my case some older php gallery software threw multiple DEPRECATED problem so I had to work through them case by case.
2. I had a problem with php and mpm.
3. Certbot/Let's Encrypt does not recognize Amazon Linux 2 from /etc/issue and fails.

LAMP Install:

Pretty much followed this without issues.

# yum update -y
# amazon-linux-extras install lamp-mariadb10.2-php7.2
# yum install -y httpd php mariadb-server php-mysqlnd
# systemctl enable httpd
# usermod -a -G apache ec2-user
# chown -R ec2-user:apache /var/www
# chmod 2775 /var/www && find /var/www -type d -exec sudo chmod 2775 {} \;
# find /var/www -type f -exec sudo chmod 0664 {} \;
# echo "<?php phpinfo(); ?>" > /var/www/html/phpinfo.php

MPM Issue:

There may be other or better ways to solve this I have not had time to investigate further.

# systemctl start httpd
Job for httpd.service failed because the control process exited with error code. See "systemctl status httpd.service" and "journalctl -xe" for details.

# systemctl status httpd.service -l
● httpd.service - The Apache HTTP Server
   Loaded: loaded (/usr/lib/systemd/system/httpd.service; enabled; vendor preset: disabled)
  Drop-In: /usr/lib/systemd/system/httpd.service.d
   Active: failed (Result: exit-code) since Tue 2018-05-29 13:35:34 UTC; 1min 21s ago
     Docs: man:httpd.service(8)
  Process: 12701 ExecStart=/usr/sbin/httpd $OPTIONS -DFOREGROUND (code=exited, status=1/FAILURE)
 Main PID: 12701 (code=exited, status=1/FAILURE)

May 29 13:35:34 ip-172-31-48-7.ec2.internal systemd[1]: Starting The Apache HTTP Server...
May 29 13:35:34 ip-172-31-48-7.ec2.internal httpd[12701]: [Tue May 29 13:35:34.378884 2018] [php7:crit] [pid 12701:tid 140520257956032] Apache is running a threaded MPM, but your PHP Module is not compiled to be threadsafe.  You need to recompile PHP.
May 29 13:35:34 ip-172-31-48-7.ec2.internal httpd[12701]: AH00013: Pre-configuration failed

# pwd

# cp 00-mpm.conf /tmp
# vi 00-mpm.conf 
# diff 00-mpm.conf /tmp/00-mpm.conf 
< LoadModule mpm_prefork_module modules/
> #LoadModule mpm_prefork_module modules/
< #LoadModule mpm_event_module modules/
> LoadModule mpm_event_module modules/

# systemctl restart httpd

# ps -ef | grep http
root      9735     1  0 13:42 ?        00:00:00 /usr/sbin/httpd -DFOREGROUND
apache    9736  9735  0 13:42 ?        00:00:00 /usr/sbin/httpd -DFOREGROUND
apache    9737  9735  0 13:42 ?        00:00:00 /usr/sbin/httpd -DFOREGROUND
apache    9738  9735  0 13:42 ?        00:00:00 /usr/sbin/httpd -DFOREGROUND
apache    9739  9735  0 13:42 ?        00:00:00 /usr/sbin/httpd -DFOREGROUND
apache    9740  9735  0 13:42 ?        00:00:00 /usr/sbin/httpd -DFOREGROUND


On the old server delete certs.

# /opt/ delete
Deleted all files relating to certificate

On the new server install certs.

# yum install mod_ssl

# wget
# chmod a+x certbot-auto 
# ./certbot-auto --debug

Sorry, I don't know how to bootstrap Certbot on your operating system!

Work around the fact that certbot does not know about Amazon Linux 2 yet.

# yum install python-virtualenv python-augeas
# ./certbot-auto --debug --no-bootstrap
Creating virtual environment...
Installing Python packages...
Installation succeeded.
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Error while running apachectl configtest.

AH00526: Syntax error on line 100 of /etc/httpd/conf.d/ssl.conf:
SSLCertificateFile: file '/etc/pki/tls/certs/localhost.crt' does not exist or is empty

How would you like to authenticate and install certificates?
1: Apache Web Server plugin - Beta (apache) [Misconfigured]
2: Nginx Web Server plugin - Alpha (nginx)
Select the appropriate number [1-2] then [enter] (press 'c' to cancel): 1

The selected plugin encountered an error while parsing your server configuration
and cannot be used. The error was:

Error while running apachectl configtest.

AH00526: Syntax error on line 100 of /etc/httpd/conf.d/ssl.conf:
SSLCertificateFile: file '/etc/pki/tls/certs/localhost.crt' does not exist or is

Have to fix ssl first apparently certbot need a generic localhost cert.

# openssl req -new -x509 -nodes -out localhost.crt -keyout localhost.key

# mv localhost.crt localhost.key /etc/pki/tls/certs/
# mv /etc/pki/tls/certs/localhost.key /etc/pki/tls/private/

# systemctl restart httpd

Now try again.

# ./certbot-auto --debug --no-bootstrap
Saving debug log to /var/log/letsencrypt/letsencrypt.log

How would you like to authenticate and install certificates?
1: Apache Web Server plugin - Beta (apache)
2: Nginx Web Server plugin - Alpha (nginx)
Select the appropriate number [1-2] then [enter] (press 'c' to cancel): 1
Plugins selected: Authenticator apache, Installer apache
Enter email address (used for urgent renewal and security notices) (Enter 'c' to

Which names would you like to activate HTTPS for?
Select the appropriate numbers separated by commas and/or spaces, or leave input
blank to select all options shown (Enter 'c' to cancel): 1
Obtaining a new certificate
Performing the following challenges:
http-01 challenge for
Waiting for verification...
Cleaning up challenges
Created an SSL vhost at /etc/httpd/conf.d/vhost-le-ssl.conf
Deploying Certificate to VirtualHost /etc/httpd/conf.d/vhost-le-ssl.conf

Please choose whether or not to redirect HTTP traffic to HTTPS, removing HTTP access.
1: No redirect - Make no further changes to the webserver configuration.
2: Redirect - Make all requests redirect to secure HTTPS access. Choose this for
new sites, or if you're confident your site works on HTTPS. You can undo this
change by editing your web server's configuration.
Select the appropriate number [1-2] then [enter] (press 'c' to cancel): 2
Redirecting vhost in /etc/httpd/conf.d/vhost.conf to ssl vhost in /etc/httpd/conf.d/vhost-le-ssl.conf

Congratulations! You have successfully enabled

You should test your configuration at:

Test your site here:

Comments Off on Amazon Linux 2 Image and LAMP

Jan 22

AWS API and Python Boto

Quick note on connection to EC2 to list instances.

- Ensure IAM User permissions. In my case I tried EC2FullAccess.
- Ensure you have your access and secret key handy.
- This example just cycle through regions and list any instances.

import argparse
import boto.ec2

access_key = ''
secret_key = ''

def get_ec2_instances(region):
    ec2_conn = boto.ec2.connect_to_region(region,
    reservations = ec2_conn.get_all_reservations()
    for reservation in reservations:    
        print region+':',reservation.instances

    for vol in ec2_conn.get_all_volumes():
        print region+':',

def main():
    regions = ['us-east-1','us-west-1','us-west-2','eu-west-1','sa-east-1',
    parser = argparse.ArgumentParser()
    parser.add_argument('access_key', help='Access Key');
    parser.add_argument('secret_key', help='Secret Key');
    args = parser.parse_args()
    global access_key
    global secret_key
    access_key = args.access_key
    secret_key = args.secret_key
    for region in regions: get_ec2_instances(region)

if __name__ =='__main__':main()


$ python myaccess_key mysecret_key
us-east-1: [Instance:i-1aac5699]
us-east-1: vol-d121290e

Comments Off on AWS API and Python Boto