OMF

NOTE: Yellow Box only 

- eth1 is interface located higher, above the two usb.

- eth1 is the one PXE enabled, however if PXE link is on eth0's address, node will still

  boot PXE when the link is present. This is currently the case in our new install. Might

  want to reverse dhcp.conf to reflect that at some stage...

  (i.e. at the moment our eth0 on nodes have 192.168.1.10x addresses, and NA use eth0 for 

  control message, while PXE is enabled only on eth1.... what we should have is NA using 

  eth1 with PXE and IP 192.168.1.10x and eth0 having other such as 10.0.0.x for the fixed

  experimental network)

FIRST THING:

	- Install fresh Linux distribution on a server machine 

	and at least one experimental machines (here we used Ubuntu)

	- The server machine will have the hostname 'theserver'

	- The first experimental machine will have the hostname 'node001', etc...

	- Remove annoying network-manager on 'theserver' and any 'node00X'

		$ sudo apt-get remove network-manager

------------------ ------------------  -----------------

------------------ ------------------  -----------------

------------------ ------------------  -----------------

STEP 1 - Installation of Supporting Services on 'theserver'

------------------ Network and DHCP -----------------

- OPTIONAL: in this example, this machine is connected to the 

  outside world via its 'eth0' interface. This short step configures

  this eth0 interface and is not necessary if this machine is only

  connected to the MNet.

	$ sudo vi /etc/network/interfaces

	Add lines:

		auto eth0

		iface eth0 inet dhcp

	$ sudo apt-get install resolvconf

	$ sudo /etc/init.d/networking restart

- Configure IP address on MNet interface:

  In this example, the machine is attached to MNet via eth1 

  and will have the IP 192.168.1.1 on the MNet 192.168.1.0

  (also in this example, this machine is connected to the

  outside world via another interface 'eth0' configured via

  dhcp info from another server out of MNet)

	$ sudo vi /etc/network/interfaces

	Add lines:

		auto eth1
		iface eth1 inet static
		address 192.168.1.1
		netmask 255.255.255.0
		network 192.168.1.0
		broadcast 192.168.1.255

		# This line should be commented if this machine

		# is connected to the outside world via another

		# interface, such as 'eth0' in our case

		#gateway 192.168.1.1

- Restart networking service:

	$ sudo /etc/init.d/networking restart

- Install DHCP server:

	$ sudo apt-get install dhcp3-server

- Configure DHCP server to serve request on the MNet:

  In this example the machine is attached to MNet via eth1

	$ sudo vi /etc/default/dhcp3-server

	Add lines:

		INTERFACES="eth1"

- Configure DHCP server parameters:

	$ sudo vi /etc/dhcp3/dhcpd.conf

	Comment lines:

		# option domain-name ...

		# option domain-name-servers ...

	Add lines:

		# Optional: tell DHCP not to operate on any existing subnet 

		# you might have on the network attached to 'eth1'

		subnet xxx.xxx.xxx.xxx netmask yyy.yyy.yyy.yyy { }

	Add lines:

		# These are the DHCP config parameter for 

		# the 192.168.1.0 subnet.

		# This machine will also be the router, name server, and

		# repository for PXE boot for that subnet.

		# Experimental machines will be given IP incremental addresses

		# starting with 192.168.1.101

		# 

		subnet 192.168.1.0 netmask 255.255.255.0 {

		  range 192.168.1.100 192.168.1.250;

		  option domain-name-servers 192.168.1.1;

		  option domain-name "yourtestbed.yourdomain.com";

		  use-host-decl-names on;

		  option routers 192.168.1.1;

		  option broadcast-address 192.168.1.255;

		  filename "/pxelinux.0";

		  next-server 192.168.1.1;

		  # First experimental machine - 001

		  # Replace 'xx' with the MAC address of the network interface 

		  # connected to the MNet on that machine 

		  host node001 {

		    hardware ethernet xx:xx:xx:xx:xx:xx;

		    fixed-address 192.168.1.101;

		  }	

		  etc....

		}

- Restart the DHCP service:

	$ sudo /etc/init.d/dhcp3-server restart

- Add hostnames in your '/etc/hosts'

	$ sudo vi /etc/hosts

	Add lines:

		127.0.0.1		localhost

		127.0.1.1		theserver.yourtestbed.yourdomain.com	theserver

		192.168.1.1		theserver.yourtestbed.yourdomain.com	theserver    

		# Names for our Experimental nodes

		192.168.1.101	node001.yourtestbed.yourdomain.com		node001

		etc...

- TEST: 

	Reboot your experimental machine node001 and check that it has the 

	correct IP, according to the config you entered in dhcpd.conf

	We assume that node001 is the "at least one experimental machine" on

	which you installed a fresh copy of ubuntu at the beginning of this 

	log book.

	$ ping 192.168.1.101

	$ ping node001

	(if you have more experimental machine installed...)

	$ ping 192.168.1.102

	etc...

------------------ TFTP Server and PXE Boot -----------------

- install a TFTP server with support for RFC-1784 and RFC-2349, e.g. 'atftpd' or 'tftpd-hpa'

	$ sudo apt-get install tftpd-hpa

- configure the TFTP server

	$ sudo vi /etc/default/tftpd-hpa

	Change lines:

		RUN_DAEMONS="yes"

		OPTIONS="-l -s -v /tftpboot"

- create "/tftpboot" directory

	$ sudo mkdir /tftpboot

- start tftpd daemon

	$ sudo /etc/init.d/tftpd-hpa start

- retrieve 'pxelinux.0' bootloader for your distribution

	$ wget http://archive.ubuntu.com/ubuntu/dists/hardy/main/installer-i386/current/images/netboot/pxelinux.0

	$ sudo mv pxelinux.0 /tftpboot

- create "/tftpboot/pxelinux.cfg" directory

	$ sudo mkdir /tftpboot/pxelinux.cfg

- create your own Kernel and Disk image for PXE boot

	SEE Appendix on HOW-TO create Kernel and Disk Image for PXE boot

- copy your Kernel and Disk Image to "/tftpboot"

	$ sudo cp mykernel-pxe-2.6 /tftpboot	

	$ sudo cp myinitrd-pxe-2.6 /tftpboot

- create a PXE boot configuration file for this particular kernel+disk image

	$ sudo vi /tftpboot/pxelinux.cfg/mypxeconfig

	Add lines:

		DEFAULT linux

		LABEL linux

		KERNEL mykernel-pxe-2.6

		APPEND rw init=/sbin/init initrd=myinitrd-pxe-2.6 console=ttyS0,9600

		DISPLAY message.txt

		PROMPT 1

		TIMEOUT 1

- create a default PXE boot configuration, where default boot = boot from local hard-drive:

	$ sudo vi /tftpboot/pxelinux.cfg/default

	Add lines:

		DEFAULT harddrive

		LABEL harddrive

		localboot 0	

- TEST: 

	Check that tftp daemon is running:

		$ sudo ps ax | grep tftp

	We will now try PXE boot for node001 which has IP 192.168.1.101.

	We assume that node001 is the "at least one experimental machine" on

	which you installed a fresh copy of ubuntu at the beginning of this 

	log book.

	Create link with node001's IP address in '/tftpboot/pxelinux.cfg'

		$ cd /tftpboot/pxelinux.cfg

		$ sudo ln -s mypxeconfig C0A80165

	Note: to PXE-boot a node with IP address XX, one has to create a symoblic

      link in /tftpboot/pxelinux.cfg towards the PXE config file to boot from. The

	name of this link should be the node's IP address in hexadecimal.

	More info on PXE-boot at: http://syslinux.zytor.com/pxe.php

	Reboot node001 and check that it effectively boots via PXE using the 

	kernel 'mykernel-pxe-2.6' and disk image 'myinitrd-pxe-2.6'

	(e.g. plug a display and visually check, or a console, etc...)

	Then remove the link in '/tftpboot/pxelinux.cfg'

		$ sudo rm -f C0A80165

	Reboot node001 and check that it effectively boots from its local 

	hard-drive

------------------ MySQL Server and Berkeley DB -----------------

- Install MySQL server:

	$ sudo apt-get install mysql-server

  (after install, apt-get goes to configure the server, nothing special

  should be done there apart from setting a passwd for the 'root' mysql user)

- Create the 'orbit' account in mysql:

	$ mysql -u root -p

	Inside mysql:

		> use mysql;

		> create user orbit;

		> grant all on *.* to 'orbit';

		> quit;

	$ mysql -u orbit -p

	Inside mysql:

		> set password=password('orbit');

		> quit;

	NOTE: for unknown reason the mysql v5 on ubuntu hardy did not correctly 

	set the password for orbit when using the command:

	  > create user orbit identified by 'orbit'

	NOTE: On some distro, you might have to specify the domain 'localhost' in 

	addition to the user, i.e. 'orbit'@'localhost' instead of just 'orbit'

- Check that the MySQL socket is correctly installed in '/tmp':

	Ruby's MySQL implementation is looking for the MySQL socket at the 

	following path: 

		'/tmp/mysql.sock'

	However, MySQL v5 install on ubuntu hardy places that socket file at:

		'/var/run/mysqld/mysqld.sock'

	If this is the case, you have to create a symlink:

		$ sudo ln -s /var/run/mysqld/mysqld.sock /tmp/mysql.sock

	NOTE: This symlink might be removed on reboot of your system... if so

	place the above symlink creation line in one of your system's startup scripts.

	NOTE: If the above link is not created, the GS Inventory will return the following

	errors when it tries to connect to the Inventory database:

		' Cannot connect to the Inventory Database '

		' No such file or directory - /tmp/mysql.sock: '

- Install Berkeley DB libraries. These libraries should be installed by default 

  with most install config (check if /usr/lib/libdb-X.Y.so exist on your machine).

  However, if not, install it using: 

	$ sudo apt-get install libdbX.Y

- TEST:

	Login into mysql as user 'orbit' with password 'orbit', 

	then create a database, play with it, then delete it.

	$ mysql -u orbit -p

	Inside mysql:

		> create database mytest;

		> use mytest;

		> create table mytable (myfield varchar(20));

		> show tables;

		etc...

		> drop mytest;

		> quit;

------------------ Install Frisbee -----------------

- Download the Frisbee source code from Emulab:

	Go to Emulab webpage and retrieve the URL for the latest source

	code release (in this example, the release of 30/06/08)

	Emulab webpage: http://www.emulab.net/software.php3

	$ wget http://www.emulab.net/downloads/emulab-080630.tar.gz

	$ tar xzf emulab-080630.tar.gz

- If you do not have the libC headers and ZLib headers currently 

  installed, you need to install them:

	$ sudo apt-get install libc6-dev

	$ sudo apt-get install zlib1g-dev

- Compile the stand-alone imagezip software

	$ cd testbed-080630/os/imagezip

	$ vi Makefile-linux.sa

	Comment lines:

		#ifdef WITH_NTFS

		...

		#endif

	Add 'disksize.o' to targets 'imagezip' and 'imageunzip'

		imagezip: disksize.o imagezip.o (etc...)

			$(CC) $(CFLAGS) disksize.o imagezip.o (etc..)

		(Do the same for 'imageunzip' target)

	$ make -f Makfile-linux.sa

	$ sudo cp imagezip imageunzip imagedump /usr/bin

- Compile the stand-alone frisbee software

	$ cd testbed-080630/os/frisbee.redux

	$ vi Makefile-linux.sa

	Add 'disksize.o' to the list of objects to link in the targets

	'frisbee' and 'frisbeed'

		frisbee: $(CLIENTOBJS)

			$(CC) $(IMAGEZIPDIR)/disksize.o (etc...)

		frisbeed: $(SERVEROBJS)

			$(CC) $(IMAGEZIPDIR)/disksize.o (etc...)

	$ make -f Makefile-linux.sa

	$ sudo cp frisbee frisbeed /usr/sbin

------------------ Install and Configure NFS Server -----------------

- Install the packages:

	$ sudo apt-get install nfs-kernel-server nfs-common portmap

- Configure portmap:

	Make sure that portmap do NOT bind on loopback:

	$ sudo dpkg-reconfigure portmap

	$ sudo /etc/init.d/portmap restart

- Create a directory to be exported via NFS. This directory will 

  contain the saved disk images from the experimental machines.

	$ sudo mkdir /export/

	$ sudo mkdir /export/omf

	$ sudo mkdir /export/omf/image

	$ sudo chmod 777 /export/omf

	$ sudo chmod 777 /export/omf/image

- Configure NFS:

	$ sudo vi /etc/exports

	Add lines:

		/export/omf/image *(rw,async)

	$ sudo /etc/init.d/nfs-kernel-server restart 

- TEST:

	Install NFS client on another machine connected to MNet.

	Mount the remote shared directory '/export/omf/image' on that

	machine.

	$ sudo apt-get install nfs-common portmap	

	$ cd /tmp

	$ mkdir test_mount_point

	$ sudo mount 192.168.1.1:/export/omf/image test_mount_point

	$ touch test_mount_point/testing_nfs.txt

	$ umount test_mount_point

	Check on the server side that the 'testing_nfs.txt' file created

	remotely is indeed present:

	$ ls /export/omf/image

------------------ ------------------  -----------------

------------------ ------------------  -----------------

------------------ ------------------  -----------------

STEP 2 - Installation of Grid Services (GS) Software on 'theserver'

------------------ Install the GS Daemon Software -----------------

- Install the required 'libruby1.8' and 'libldap-ruby1.8' libraries:

	$ sudo apt-get install libruby1.8 libldap-ruby1.8

- Install the GS software package from the OMF release:

	$ sudo dpkg --force-depends --install gridservices2_2.1.0_all.deb

	(The '--force-depends' option is there to turn into warning the error

	about the dependency on 'frisbee' package. Since we already compiled 

	and installed the Frisbee software from source earlier, there is no 

	need to install the 'frisbee' package.

	We did show how to compile Frisbee from source because the binaries

	inside the winlab 'frisbee' package might not be compatible with all

	linux installs).

------------------ Configure the GS Daemon Software -----------------

- Configure the main GS daemon:

	$ sudo vi /etc/gridservices2/gridservices.cfg

	Change following line to use your own preferred port:

		OPTS="--port 5022"

- Enable the cmcStub GS service:

	$ sudo mkdir /etc/gridservices2/enabled

	$ cd /etc/gridservices2/enabled

	$ sudo ln -s /etc/gridservices2/available/cmcStub.yaml

- Start the GS daemon:

	$ sudo /etc/init.d/gridservices2 start

- TEST:

	The outputs of the following commands should show a 'ruby' process 

	running the GS software on port 5022:

		$ sudo ps ax | grep ruby

		$ sudo lsof -i | grep ruby

	The outputs of the following commands should show you the following 

	message from the GS daemon:

		$ wget http://192.168.1.1:5022/ -O output.txt

		$ more output.txt

			<?xml version='1.0'?>

			<serviceGroups>

				<serviceGroup path='/cmc' name='cmc'>

					<info> Information on available testbed (etc..)

					</info>

				</serviceGroup>

			</serviceGroups>	

	You can also try the same 'wget' command from the machine 'node001'.

	The log file at '/var/log/gridservices2.log' gives you some information

	about the recent GS activities:

		$ tail /var/log/gridservices2.log

------------------ Configure the CMC Stub GS service -----------------

- Configure 'cmcStub.yaml'

	$ sudo vi /etc/gridservices2/enabled/cmcStub.yaml

	Change following lines:

		listAll : |

		  (101..150)..to_a.map {|i| [i, 0]}

		listStatus: |

		  (101..150).to_a.map {|i| [i, 0, 'POWERON']}

	NOTE: Use spaces to indent and NOT tabs!

	NOTE: The above assumes we have 50 experimental nodes, which were 

		allocated IP addresses from 192.168.1.101 to 192.168.1.150 by

		the DHCP server we configured earlier.	

- Restart the GS daemon:

	$ sudo /etc/init.d/gridservices2 restart

	(if you get some errors about 'check_config', do a stop then start)

- TEST:

	Query the cmcStub GS service for the status of all nodes (aka experimental

	machines). Since it is a stub, it should reply that all nodes are in the

	'POWERON' state).

		$ wget http://192.168.1.1:5022/cmc/allStatus -O states.txt

		$ more states.txt

		$ sudo /etc/init.d/gridservices2 start 

			<TESTBED_STATUS>

				<detail>

					<node name='n_101_0' x='101' y='0' state='POWERON'/>

					(etc... for other nodes)

				</detail>

			</TESTBED_STATUS>

------------------ Configure the Inventory GS service -----------------

NOTE: if you have this error in the GS log:

	Cannot connect to the Inventory Database - No such file or directory - /tmp/mysql.sock: 

You need to do this:

	/tmp/mysql.sock -> /var/run/mysqld/mysqld.sock

(stupid Ruby-mysql has hardcoded path to mysql.sock....)

------------------ ------------------  -----------------

------------------ ------------------  -----------------

------------------ ------------------  -----------------

STEP 3 - Installation of Node Handler (NH) Software on 'theserver'

------------------ Install and Configure the NH Software -----------------

- Install the NH software package from the OMF release:

	$ sudo dpkg --force-depends --install nodehandler4_4.2.0_all.deb

- Configure the NH software:

	$ cd /etc/nodehandler4/

	$ sudo cp nodehandler.yaml.nicta nodehandler.yaml

	$ sudo vi nodehandler.yaml

	Change lines (again use spaces instead of tabs):

	default:

		X_MAX: 255

		Y_MAX: 1 	

		oml:

		  local_host: '192.168.1.1'

		frisbee:

		  default_disk: '/dev/sda'

		controlIP: |

		  |x, y|

		    "192.168.1.#{x}"

		coord2IP: |

		  |x, y|

		    name = "192.168.1.#{x}"

- TEST:

	Call the NH to request the status of the experimental machine 'node001'

	$ omf stat [101,0]

	The output should look like:

		-----------------------------------------------

 		Testbed : indoor

 		 Node n_101_0 - State: POWERON

		-----------------------------------------------

	(NOTE: with our particular notation, if we want 'omf stat all' to 

------------------ ------------------  -----------------

------------------ ------------------  -----------------

------------------ ------------------  -----------------

STEP 3 - Installation of Node Agent (NA) Software on 'node001' machine

------------------ Install and Configure supporting services -----------------

- After fresh ubuntu install, remove and add some services:

	$ sudo apt-get remove network-manager

	$ sudo apt-get install ssh

- Configure some system parameters:

	$ sudo vi /etc/hosts

	Add lines:

		127.0.0.1		localhost

		127.0.1.1		node001.yourtestbed.yourdomain.com	node001

		192.168.1.101		node001.yourtestbed.yourdomain.com	node001    

		# Names for our Server with GS

		192.168.1.1		yourserver.yourtestbed.yourdomain.com	yourserver

- Make sure that the interface which supports PXE boot is ALWAYS assigned 'eth1'

  (Persistent naming of PXE-enabled interface to 'eth1')

	You need to install 'udev' (rule-based device node and kernel event manager)

	(if not already installed by default)

	$ sudo apt-get install udev

	Then configure udev to permanently assign 'eth1' to PXE-enabled interface

	$ sudo vi /etc/udev/rules.d/70-persistent-net.rules

	Change lines so that 'eth1' is paired with the MAC address of the PXE-enabled interface:

	SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="XX:XX:XX:XX:XX:XX", ATTR{type}=="1", KERNEL=="eth*", NAME="eth1"

------------------ Install and Configure the NA Software -----------------

- Install required dependencies packages from the OMF release:

	$ sudo dpkg --install liboml-transport_2.0.3_i386.deb

	$ sudo dpkg --install liboml-client_2.0.4_i386.deb

	$ sudo dpkg --install liboml-filters_2.0.3_i386.deb

- Install the NA software from the OMF release

	$ sudo dpkg --force-depends --install nodeagent4_4.3.0_all.deb

	(force-depends is used here to ignore warning about dependency on 'ruby'

	package. The NA package carries its own ruby interpreter)

- Configure the NA daemon:

	$ sudo vi /etc/nodeagent4/nodeagent.yaml

	Comment lines for option i) "Multicast Communicator".

	Uncomment lines for option ii) "TCP Server Communicator".

	If needed, modify 'local_if' value to the interface connected to MNet:

		local_if: eth0

- Restart the NA Daemon:

	$ sudo /etc/init.d/nodeagent4 restart

	(if you get some errors about 'check_config', do a stop then start)

- TEST:

	1) The outputs of the following commands should show a 'ruby' process 

	running the NA software on port 9026:

		$ sudo ps ax | grep ruby

		$ sudo lsof -i | grep ruby

	2) The log file at '/var/log/nodeAgent4.log' gives you some information

	about the recent NA activities:

		$ tail /var/log/nodeAgent4.log

	3) Run the sample experiment script exp1.rb. This experiment run the

	'ls -la' command on the experimental node. The command output (the list

	of files on the node) is forwarded to the server running the NH.

	- First, edit the experiment script to set the node to use

		$ vi exp1.rb

		Change the lines (if using 'node001' then use '[101,1]'):

			defGroup('mygroup', [101,1]) 

	- Second, execute the experiment on the NH machine:

		$ omf exec exp1

	- The output should display messages from the experiment execution and

	the list of files present on the experimental node 'node001'.

------------------ Install the OTG/OTR applications -----------------

- Install required dependencies packages from the OMF release:

------------------ ------------------  -----------------

------------------ ------------------  -----------------

------------------ ------------------  -----------------

STEP 4 - Installation of other GS services on the GS server machine

------------------ Install the OML collection server -----------------

- Install the OML Collection Server and required dependency from the OMF release 

	$ sudo dpkg --install liboml-transport_2.0.3_i386.deb

	$ sudo dpkg --install oml-collection-server_2.0.3_i386.deb

- If require, create symlinks for current versions of libraries:

	$ sudo ln -s /usr/lib/libdb-4.6.so /usr/lib/libdb-4.2.so

	$ sudo ln -s /usr/lib/mysqlclient.so.15 /usr/lib/mysqlclient.so.12