Cluster Hat setup - Part 1

Submitted by code_admin on Wed, 07/25/2018 - 15:22

Cluster Hat setup - Part 1
Cluster Hat setup - Part 2
Cluster Hat setup - Part 3 - Ansible
Cluster Hat setup - Part 4 - Docker Registry
Cluster Hat setup - Part 5 - Access Point Setup
Cluster Hat - Other Processes
Set up a new Raspberry Pi 3 to join the cluster

Introduction

I purchased a cluster hat (http://clusterhat.com) and this is my mini project to get it working.
I want to use it with a RaspberryPi 3 (controller) and four Pi Zero's. I want the raspberry Pi 3 to work over wireless. I am using a laptop with ubuntu installed on it on the same wireless network to connect to the controller.

I understand that getting the Zero's to bridge over wireless is not possible so instead I want to setup a dhcp server on the controller and let the Pi Zero's get their ip addresses from there. The controller will get it's own IP from dhcp.

I guess that if the Zero's are assigned IP's in the same range as the network that the controller is connected to then they will be addressable on that network. I will find out if this is possible during the project.

**UPDATE I found that configuration is not possible. Apparently it is not possible to bridge to a wlan connection like you can with an ethernet connection. I think this is because each IP address must have a single wireless card and mac. This explains why I could not find tutorials for this. Instead I will try and setup something called a SNAT.
source: http://superuser.com/questions/597834/bridging-wifi-to-ethernet-on-ubun…

I will also set this system up as a wifi access point so that I can connect and access nodes in the cluster directly.

What you will need to follow this tutorial

In this tutorial I will build a Raspberry Pi 3 based cluster with 4 Pi Zero's. It will connect to a local home router via wifi as well as act as a wifi access point in it's own right.

Hardware needed Software we will setup
  • Raspberry Pi 3
  • 4 x Raspberry Zero
  • 1 x class 10 36Gb Micro SD card for the Pi 3
  • 4 x class 10 16Gb Micro SD card for the Pi Zero's
  • 1 x Clusterhat (with it's stand off's and USB lead to connect the Pi3 and Cluster Hat)
  • 1 x USB network adapter - to give us a second wifi interface to create an Access point
  • 1 x USB power adapter to power the Pi 3
  • 1 x Laptop (My main laptop I used for this runs ubuntu but I think other OS's will work as well since it really just needs ssh and an SD card reader to create the SD cards)
  • Internet Connection
 On the controller:

  • DHCP server
  • DNS server
  • Ansible is used to manage the cluster nodes
  • Docker
  • Docker Registry - Private registry to push docker images to which the cluster nodes can pull their images from
  • hostapd so the controller can act as a wifi access point

On the Cluster Nodes

  • Docker
  • Sample Apache Docker image - built on the controller and published via the private registry. Simple server so I can prove the process works.

This project has taken me about 7, 8 hour days to get working. Although this involved researching, learning, going down many wrong paths and writing the tutorial. Hopefully anyone following it should be able to do so in less time but there is a lot of ground covered and there is a lot to learn. Also this project is a learning project for me so I think the time here is well spent.

If you have got this far you won't have missed the fact that I am not highly skilled in writing tutorials (or writing anything for that matter.) I hope my directions are clear enough to follow and that I haven't left any bugs or technical issues. Feedback is welcome and I would like to hear if this proves useful. (You can't contact me from this site but I keep up to date with the forum at https://clusterhat.com/. I also read my twitter from time to time @metcarob)

Network Interface note

In this tutorial I plan to use the internal Raspberry Pi 3's network interface (wlan0) as an accesspoint.
I have an external dongle (wlan1) to connect the Raspberry Pi 3 to the internet.
I have realized some people will want to follow the first steps in this tutorial but don't want to setup the accesspoint at the end. If this is the case you do not need a dongle and so will not have a wlan1. Swap wlan1 and wlan0 references around and it should all work.

Step 1 - Get the controller raspberry Pi3 working

Downloaded controller image from https://clusterhat.com/setup-software and installed on raspberryPi 3. I used the Lite Controller image downloaded from the intermediate section.

Actions preformed before putting SD card into Pi 3

I didn't have to change the host name but if I had wanted to I would need to change the following files on the sd card:

  1. /etc/hosts
  2. /etc/hostname

I added my wireless username and password by editing the wpa_supplicant file:

  1. country=GB
  2. ctrl_interface=DIR=/var/run/wpa_supplicant GROUP=netdev
  3. update_config=1
  4. network={
  5.     ssid="The_ESSID_from_earlier"
  6.     psk="Your_wifi_password"
  7.     id_str="main_net"
  8. }

I put a file called "ssh" in the root of the boot partition of the SD card. This is to enbale ssh.

The first time I set up the pi I didn't leave long enough for it to boot and must have reset it while it was building the ssh keys. I wasn't able to sucessfully ssh to the server. This can be fixed by either repeating the SD card creation process or deleting the partially written ssh keys:
cd /etc/ssh
sudo rm key
sudo dpkg-reconfigure openssh-server

Actions preformed once card is in Pi 3 and booted

The raspberry pi 3 we setup is called controller. You will have to have some method to find it's ip address. You can plug it into a monitor and watch it boot but I prefer to log into my LAN's router and see the ip listed. Once you find the ip you can launch ssh:

  1. ssh pi@192.168.1.222

The first thing to do is secure the pi user. You can either change the default password by using the passwd command or populate /home/pi/.ssh/authroized_keys
If you choose to use the ssh key method you might be tempted to change the password to something long and unforgettable. Don't do this at this point as we will be messing around with network settings and may need to log in via keyboard if we accidently break something. Resseting the password to something unforgettable can be done at the end of this part.

I loged out and back into the pi zero to confirm the key took.

Run raspi-config to:
- Expand the file system
- Change timezone to europe - London

Reboot

Next update the software on the system:

  1. sudo apt-get update
  2. sudo apt-get upgrade

Reboot

Step 2 - Setup interfaces file and internet forwarding

Next I want to change the interfaces file and add internet forwarding.
I also want to swap the internet connection from the internal wifi (wlan0) to the external one I have plugged in (wlan1)
This is because I want to use wlan0 as an access point interface at a later stage and wlan1 is not capable to do this.
Also I add a bridge network.

We must enable filtering. Edit
/etc/sysctl.conf
and uncomment the line:

  1. # Uncomment the next line to enable packet forwarding for IPv4
  2. net.ipv4.ip_forward=1

create a file /etc/network/iptables with contents:

  1. #!/bin/sh
  2. IPT=/sbin/iptables
  3. LOCAL_IFACE=br0
  4. INET_IFACE=wlan1
  5. # Flush the tables
  6. $IPT -F INPUT
  7. $IPT -F OUTPUT
  8. $IPT -F FORWARD
  9.  
  10. $IPT -t nat -P PREROUTING ACCEPT
  11. $IPT -t nat -P POSTROUTING ACCEPT
  12. $IPT -t nat -P OUTPUT ACCEPT
  13.  
  14. # Allow forwarding packets:
  15. $IPT -A FORWARD -p ALL -i $LOCAL_IFACE -j ACCEPT
  16. $IPT -A FORWARD -i $INET_IFACE -m state --state ESTABLISHED,RELATED -j ACCEPT
  17.  
  18. # Packet masquerading
  19. $IPT -t nat -A POSTROUTING -o $INET_IFACE -j MASQUERADE
  20.  
  21. exit 0

This file must be executable:

  1. sudo chmod +x /etc/network/iptables

Now create the interfaces file:
/etc/network/interfaces

  1. # interfaces(5) file used by ifup(8) and ifdown(8)
  2.  
  3. # Include files from /etc/network/interfaces.d:
  4. ##source-directory /etc/network/interfaces.d
  5.  
  6.  
  7. #RJM Adding to replace interfaces.d
  8. #the pre-up command will add the individual interface to the bridge
  9. allow-hotplug ethpi1
  10. iface ethpi1 inet manual
  11.     pre-up brctl addif br0 ethpi1
  12.     up ifconfig ethpi1 up
  13.  
  14. allow-hotplug ethpi2
  15. iface ethpi2 inet manual
  16.         pre-up brctl addif br0 ethpi2
  17.         up ifconfig ethpi2 up
  18.  
  19. allow-hotplug ethpi3
  20. iface ethpi3 inet manual
  21.         pre-up brctl addif br0 ethpi3
  22.         up ifconfig ethpi3 up
  23.  
  24. allow-hotplug ethpi4
  25. iface ethpi4 inet manual
  26.         pre-up brctl addif br0 ethpi4
  27.         up ifconfig ethpi4 up
  28.  
  29.  
  30.  
  31. auto lo br0
  32. iface lo inet loopback
  33.  
  34. allow-hotplug wlan1
  35. iface wlan1 inet dhcp
  36.     wpa-conf /etc/wpa_supplicant/wpa_supplicant.conf
  37.  
  38. ##wlan0 is the interface for the access point
  39. allow-hotplug wlan0
  40. iface wlan0 inet manual
  41.     pre-up brctl addif br0 wlan0
  42.  
  43. #allow-hotplug eth0
  44. iface eth0 inet manual
  45.  
  46. iface br0 inet static
  47.     pre-up /etc/network/iptables
  48.     address 192.168.2.1
  49.     network 192.168.2.0
  50.     netmask 255.255.255.0
  51.     broadcast 192.168.2.255
  52.     bridge_ports ethpi1 ethpi2 ethpi3 ethpi4
  53.     bridge_stp off
  54.     bridge_waitport 0
  55.     bridge_fd 0
  56.  
  57. # added these lines becuase settings in /etc/dhcp/dhclient.conf were being ignored
  58. # for some reason
  59. ##dns-nameservers 127.0.0.1
  60. ##dns-search metcarob-local.com

The last two lines are commented out. We will use them at the end of part 2 once we have set up the dns server on the controller.

Step 3 - Setup dhcp server on the controller

This dhcp server should work only on all the network interfaces apart from the USB dongle used for the internet.

  1. sudo apt-get install isc-dhcp-server

Change /etc/default/isc-dhcp-server to specify the interface "br0". This will be the bridge interface and as each Pi Zero connects it's interface will be dynamically added to this bridge.

  1. INTERFACES="br0"

I used the following as my /etc/dhcp/dhcpd.conf:

  1. ddns-update-style none;
  2.  
  3. option domain-name "example.org";
  4. ##option domain-name-servers ns1.example.org, ns2.example.org;
  5.  
  6. ##default-lease-time 600;
  7. ##max-lease-time 7200;
  8.  
  9. # If this DHCP server is the official DHCP server for the local
  10. # network, the authoritative directive should be uncommented.
  11. authoritative;
  12.  
  13. # Use this to send dhcp log messages to a different log file (you also
  14. # have to hack syslog.conf to complete the redirection).
  15. log-facility local7;
  16.  
  17. subnet 192.168.2.0 netmask 255.255.255.0 {
  18.     range 192.168.2.201 192.168.2.250;
  19.  
  20.         option routers                  192.168.2.1;
  21.         option subnet-mask              255.255.255.0;
  22.         option broadcast-address        192.168.2.255;
  23.         option domain-name-servers      192.168.1.1;
  24.         ##option ntp-servers              192.168.1.1;  #DO NOT SEND ntp server info
  25.         option netbios-name-servers     192.168.1.1;
  26.         option netbios-node-type 2;
  27.         default-lease-time 86400;
  28.         max-lease-time 86400;
  29. }

***TODO 192.168.1.1 is the wireless dhcpd server info. I want to find a way to get this automatically

Command to restart dhcp server

  1. sudo service isc-dhcp-server restart

Check status of dhcp server

  1. sudo systemctl status isc-dhcp-server.service

Command to check if there are any dhcp leases:

  1. cat /var/lib/dhcp/dhcpd.leases

Step 4 - Setup first cluster node

Before starting set up public keys for the pi user on the controller by running the following command on the controller:

  1. ssh-keygen -t rsa -b 4096 -f ~/.ssh/id_rsa

This will give us a public key we can put on the P1-P4 so we don't have to use a password to log in.

I used exactly the same controller image again (Downloaded from https://clusterhat.com/setup-software and installed on raspberryPi 3. I used the Lite Controller image downloaded from the intermediate section.)

I put this image onto a fresh SD card.
Add a file "ssh" in the root of the boot section of the SD Card

Convert the controller image into an image for P1 Zero append " quiet init=/sbin/reconfig-clusterhat p1" to the cmdline.txt file in the boot (first) partition. My cmdline.txt file ended up looking like:

  1. dwc_otg.lpm_enable=0 console=serial0,115200 console=tty1 root=/dev/mmcblk0p2 rootfstype=ext4 elevator=deadline fsck.repair=yes rootwait quiet init=/sbin/reconfig-clusterhat p1

Plug it into controller position 1

SSH into the controller and then turn on first raspberry pi by typing:

  1. clusterhat on p1

Note the lights on the pi hat indicating the result.

You will have to wait a while for the pi-zero to boot as the first time it does some initial setup.

wait a while for the Pi Zero to boot then find it's ip address by running:
Command to check if there are any dhcp leases:

  1. cat /var/lib/dhcp/dhcpd.leases

it is hopefully 192.168.2.201 - but the exact ip doesn't matter as we will change it later.

ssh into it:

  1. ssh pi@192.168.2.201

While logged as user pi into the P1:
create a directory
/home/pi/.ssh
Add an authorized_keys file with controllers public key

  1. echo XXX > /home/pi/.ssh/authorized_keys

(XXX is the contents of /home/pi/id_rsa.pub on the controller)

I loged out and back into the pi zero to confirm the key took.
Then I used the passwd command to change the username for the pi user to a long string which I discarded as I don't plan to use it. (I use a throw away long value from https://www.grc.com/passwords.htm)

Run raspi-config to:
- Expand the file system
- Change timezone to europe - London

When prompted reboot and wait for the pi zero to reboot.

Further setup like applying updates etc. will be done on all the pi's together using ansible.

Step 5 - Checkpoint

At this point I wanted to check that everything that has been setup so far will work. I followed this process to test everything. I will recap a test process from the start.

I start with a fresh boot of the controller with the first Pi Zero plugged into position 1.

From the laptop I can SSH to the controller:

  1. ssh pi@192.168.1.200

From the controller I check that the bridge interface is up and running

  1. ip a

Result:

  1. pi@controller:~ $ ip a
  2. 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1
  3.     link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
  4.     inet 127.0.0.1/8 scope host lo
  5.        valid_lft forever preferred_lft forever
  6.     inet6 ::1/128 scope host
  7.        valid_lft forever preferred_lft forever
  8. 2: eth0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
  9.     link/ether b8:27:eb:ca:85:c7 brd ff:ff:ff:ff:ff:ff
  10. 3: wlan0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
  11.     link/ether b8:27:eb:9f:d0:92 brd ff:ff:ff:ff:ff:ff
  12.     inet 192.168.1.200/24 brd 192.168.1.255 scope global wlan0
  13.        valid_lft forever preferred_lft forever
  14.     inet6 fe80::ba27:ebff:fe9f:d092/64 scope link
  15.        valid_lft forever preferred_lft forever
  16. 4: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
  17.     link/ether 62:76:59:06:95:70 brd ff:ff:ff:ff:ff:ff
  18.     inet 192.168.2.1/24 brd 192.168.2.255 scope global br0
  19.        valid_lft forever preferred_lft forever
  20.     inet6 fe80::6076:59ff:fe06:9570/64 scope link
  21.        valid_lft forever preferred_lft forever

Make sure the interface br0 appears and it has sucessfully got the ip address 192.168.2.1 listed.

  1. clusterhat on p1
  2. ** Wait 60 seconds or so for the Pi Zero to boot
  3. ip a

Everything will be the same except with the additional lines:

  1. 5: ethpi1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master br0 state UP group default qlen 1000
  2.     link/ether 00:22:82:ff:fe:01 brd ff:ff:ff:ff:ff:ff
  3.     inet6 fe80::222:82ff:feff:fe01/64 scope link
  4.        valid_lft forever preferred_lft forever

Now check to see whether or not the Pi Zero got a DHCP lease:

  1. cat /var/lib/dhcp/dhcpd.leases
  1. pi@controller:~ $ cat /var/lib/dhcp/dhcpd.leases
  2. # The format of this file is documented in the dhcpd.leases(5) manual page.
  3. # This lease file was written by isc-dhcp-4.3.1
  4.  
  5. lease 192.168.2.201 {
  6.   starts 1 2016/12/26 17:43:16;
  7.   ends 2 2016/12/27 17:43:16;
  8.   tstp 2 2016/12/27 17:43:16;
  9.   cltt 1 2016/12/26 17:43:16;
  10.   binding state active;
  11.   next binding state free;
  12.   rewind binding state free;
  13.   hardware ethernet 00:22:82:ff:ff:01;
  14.   client-hostname "p1";
  15. }

Check you can ping google from the controller.

Finally check that you can log into the Pi Zero:

  1. ssh pi@192.168.2.201

You should successfully be able to log on to the Pi Zero.
However you will note that you are not able to ping google from the Pi Zero. We will fix this in the next step.

Finally
If you choose to use the ssh key method now is the time to change it to a throw away password. (I use a throw away long value from https://www.grc.com/passwords.htm)

Next Part

Now that we have 1 Pi Zero working in the next part I will get the other 3 setup.
Cluster Hat setup - Part 2

Tags

RJM Article Type
Public Article