pulling yourself up by your bootstraps

I remember when installing and configuring a machine would take me a long time. I'd have to sit through the install procedure and carefully select the options I wanted installed. Once the installation was finished I would spend time customizing and tuning the installation to fit the specific needs of the project.

When I want to install a machine now, I select the type of installation I would like to perform and apply it to the machine and then just wait until I receive an email that the machine has finished.

The following sections describe what tools we use to install our machines and perform the initial installation.

installation services

We use PXE to boot our machines. Most workstations and servers are capable of PXE booting. The advantage of PXE is that the workstation or server doesn't need physical install media inserted. You can reinstall a machine by rebooting it and selecting the boot menu (which is usually F12).

bios
We configure our machines so that the boot order is hard drive then PXE. For workstations we remove all other options from the bios. In theory someone could own our workstation by plugging it into a private network and re-installing it with their own image using this method, but hopefully that isn't a great risk in your location. If we wish to re-install the machine remotely and we do not have console access (perhaps the machine is located in a client office), we can simply wipe the boot record from the hard drive and reboot. When the machine reboots, it will try to boot from the hard-drive, fail, and then boot from the network.

PXE/DHCP/DNS/TFTP
The first machine you will need to build for your installation system is the PXE boot server. This server will run dhcp, dns, tftp, http and export some filesystems via NFS. To get started install the machine using the install media of your chosen distribution, you should be careful to only install those things that you actually need. For a Red Hat@tm; installation you would select server as the installation profile and add the dhcp, dns, tftp and http packages.

Once your machine has finished installing, let it reboot and then login. Verify that the following packages are installed.

servicepackage
webhttpd
dnsbind, bind-chroot
dhcpdhcp, dhcpv6*
tftptftp-server
*Optional, install if you are going to support IPv6

You can verify a package is installed using either yum or rpm. Below are the instructions for installing the first package, httpd

Using rpm:
If the package is installed:

[root@server0 ~]# rpm -q httpd
httpd-2.2.3-22.el5
If the package is not installed:
[root@server0 ~]# rpm -q httpd
package httpd is not installed
Using yum:
[root@server0 ~]# yum info httpd
Installed Packages
Name       : httpd
Arch       : x86_64
Version    : 2.2.3
Release    : 22.el5
Size       : 3.3 M
Repo       : installed
Summary    : Apache HTTP Server
URL        : http://httpd.apache.org/
License    : Apache Software License
Description: The Apache HTTP Server is a powerful, efficient, and extensible web
           : server.
If the package is not installed, the Repo line above will show the repo that will be used to install the package. In the above, httpd is installed so Repo is "installed".

To Install the package, you can either locate the package on your install media and use rpm, or use yum.

[root@server0 Server]# ls httpd-*
httpd-2.2.3-22.el5.x86_64.rpm      httpd-devel-2.2.3-22.el5.x86_64.rpm
httpd-devel-2.2.3-22.el5.i386.rpm  httpd-manual-2.2.3-22.el5.x86_64.rpm
[root@server0 Server]# rpm -Uvh httpd-2.2.3-22.el5.x86_64.rpm 
Preparing...                ########################################### [100%]
   1:httpd                  ########################################### [100%]

[root@server0  ~]# yum install httpd
Setting up Install Process
Parsing package install arguments
Resolving Dependencies
--> Running transaction check
---> Package httpd.x86_64 0:2.2.3-22.el5 set to be updated
--> Finished Dependency Resolution

Dependencies Resolved

================================================================================
 Package     Arch         Version              Repository                  Size
================================================================================
Installing:
 httpd       x86_64       2.2.3-22.el5         Server_Base                1.2 M

Transaction Summary
================================================================================
Install      1 Package(s)         
Update       0 Package(s)         
Remove       0 Package(s)         

Total download size: 1.2 M
Is this ok [y/N]: y
Downloading Packages:
httpd-2.2.3-22.el5.x86_64.rpm                            | 1.2 MB     00:00     
Running rpm_check_debug
Running Transaction Test
Finished Transaction Test
Transaction Test Succeeded
Running Transaction
  Installing     : httpd                                             [1/1] 

Installed: httpd.x86_64 0:2.2.3-22.el5
Complete!
Now that all the essential packages are installed on your machine, we'll configure each of the services individually in the following sections.

dns

After installing bind on the machine we'll setup a simple domain, example.com on it. As a first step, configure bind to serve as a caching DNS server. Here is the initial named.conf, change xxx.xxx.xx.xxx, yyy.yyy.yyy.yyy and zzz.zzz.zzz.zzz to the ip addresses of your nameservers*

/var/named/chroot/etc/named.conf:

options {
directory "/var/named";
forwarders { 192.168.0.2; 192.168.0.3; 192.168.0.4; };
};

zone "." in {
type hint;
file "data/db.cache";
};

zone "0.0.127.in-addr.arpa" in {
type master;
file "data/db.127.0.0";
};
Next create the root hints file for your dns server. This is the list of root dns servers that your server will use when trying to look for a dns record.

[root@server0 etc]# cd /var/named/chroot/var/named/data/
[root@server0 data]# dig @a.root-servers.net . ns > db.cache
Alternatively, if you already have dns servers configured for your machine, you can use dig's built in capabilities to get the root list back.
[root@server0 data]# grep nameserver /etc/resolv.conf
nameserver xxx.xxx.xxx.xxx
[root@server0 data]# dig +nocmd . NS +noall +answer +additional >db.cache

Or you can just download the root list from the Internic ftp server.

[root@server0 data]# wget ftp://ftp.internic.net/domain/named.root -O db.cache
--16:28:51--  ftp://ftp.internic.net/domain/named.root
           => `db.cache'
Resolving ftp.internic.net... 208.77.188.26
Connecting to ftp.internic.net|208.77.188.26|:21... connected.
Logging in as anonymous ... Logged in!
==> SYST ... done.    ==> PWD ... done.
==> TYPE I ... done.  ==> CWD /domain ... done.
==> SIZE named.root ... 2940
==> PASV ... done.    ==> RETR named.root ... done.
Length: 2940 (2.9K)

100%[=======================================>] 2,940       --.-K/s   in 0s     

16:29:02 (280 MB/s) - `db.cache' saved [2940]

Next, you'll need to create a zone file for the 127.0.0.1 zone. Zone files are what bind (named) uses to map between ipaddresses and names. The 127.0.0.0/8 range of ipaddresses is reserved for local or loopback addresses (addresses which all resolve to the machine you are working on).

/var/named/chroot/var/named/data/db.127.0.0

$TTL 3D
@       IN      SOA     localhost. root.localhost.  (
00	; Serial
86400	; Refresh
7200	; Retry
2592000	; Expire
345600 )	; Minimum
	NS	localhost.
1	PTR	localhost.

This file is the minimum required to serve up 127.0.0.0/8. With these 3 files in place, we're ready to try out our new name server.

[root@server0 data]# service named start
Starting named:                                            [  OK  ]
[root@server0 data]# nslookup localhost localhost
Server:		localhost
Address:	127.0.0.1#53

Non-authoritative answer:
Name:	localhost
Address: 127.0.0.1

[root@server0 data]# nslookup www.google.com localhost
Server:		localhost
Address:	127.0.0.1#53

Non-authoritative answer:
www.google.com	canonical name = www.l.google.com.
Name:	www.l.google.com
Address: 74.125.47.99
Name:	www.l.google.com
Address: 74.125.47.103
Name:	www.l.google.com
Address: 74.125.47.104
Name:	www.l.google.com
Address: 74.125.47.147

[root@server0 data]# 
Assuming that worked, we can put in the zone file for example.com. We will use 192.168.0.1** as the address of our new server. /var/named/chroot/var/named/data/db.example.com

$TTL 3D
@       IN      SOA     ns1.example.com. root.example.com.  (
00	; Serial
86400	; Refresh
7200	; Retry
2592000	; Expire
345600 )	; Minimum
	NS	ns1
ns0	IN	A	192.168.0.1
server0 IN      A       192.168.0.1

And update named.conf to include the new zone.

/var/named/chroot/etc/named.conf

options {
	directory "/var/named";
	forwarders { xxx.xxx.xxx.xxx; yyy.yyy.yyy.yyy; zzz.zzz.zzz.zzz; };
};

zone "." in {
	type hint;
	file "data/db.cache";
};

zone "0.0.127.in-addr.arpa" in {
	type master;
	file "data/db.127.0.0";
};

zone "example.com." in {
	type master;
	file "data/db.example.com";
};

Restart named to use the updated named.conf and zone file, then verify that your record is being served properly.

[root@server0 etc]# service named restart
Stopping named:                                            [  OK  ]
Starting named:                                            [  OK  ]
[root@server0 etc]# host ns1.example.com localhost
Using domain server:
Name: localhost
Address: 127.0.0.1#53
Aliases: 

ns0.example.com has address 192.168.0.1

One more step and we are done with dns. Right now dns is only available from the server (localhost), we'll need to open up a hole in the firewall on our machine to allow dns queries through. We do this with iptables

[root@server0 data]# iptables -A INPUT -p udp --destination-port 53 -j ACCEPT
[root@server0 data]# iptables -A INPUT -p tcp -m state --state NEW,ESTABLISHED,RELATED --destination-port 53 -j ACCEPT

Now that dns is configured we can move on to installing the web server and making sure files are available for installation.

iptables

The default iptables configuration on most distributions today is to only allow tcp port 22 (ssh) through the incoming firewall. (The outgoing firewall is unfiltered on all that I have seen). To access the dns service running on server1, a hole has to be made in the firewall. To do this, first find the name of the INPUT chain.
[root@server0 data]# iptables -L INPUT
Chain INPUT (policy ACCEPT)
target     prot opt source               destination         
RH-Firewall-1-INPUT  all  --  anywhere             anywhere            
What this means is that the INPUT chain has one target called RH-Firewall-1-INPUT+. This means to know what our INPUT rules are, we need to look at RH-Firewall-1-INPUT.
[root@server0 install]# iptables -L RH-Firewall-1-INPUT
Chain RH-Firewall-1-INPUT (2 references)
target     prot opt source               destination         
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     icmp --  anywhere             anywhere            icmp any 
ACCEPT     esp  --  anywhere             anywhere            
ACCEPT     ah   --  anywhere             anywhere            
ACCEPT     udp  --  anywhere             224.0.0.251         udp dpt:mdns 
ACCEPT     udp  --  anywhere             anywhere            udp dpt:ipp 
ACCEPT     tcp  --  anywhere             anywhere            tcp dpt:ipp 
ACCEPT     all  --  anywhere             anywhere            state RELATED,ESTABLISHED 
ACCEPT     tcp  --  anywhere             anywhere            state NEW tcp dpt:ssh 
REJECT     all  --  anywhere             anywhere            reject-with icmp-host-prohibited 
Configuring iptables properly is a separate discussion. We will build up a set of rules while we are constructing our install server, but the reader should spend some time getting to know how iptables works and how to configure it properly. The above rules setup some fairly good defaults. The rule that is most interesting to us is the second from the bottom that ends in state NEW tcp dpt:ssh. This rule allows connections on tcp port 22 (ssh) to our server. DNS runs on port 53++, so we need to allow udp and tcp connections on port 53 to our machine.
[root@server0 data]# iptables -I RH-Firewall-1-INPUT -p udp --dport 53 -j ACCEPT
[root@server0 data]# iptables -I RH-Firewall-1-INPUT -p tcp --dport 53 -m state --state NEW,ESTABLISHED,RELATED -j ACCEPT
[root@server0 data]# iptables -L RH-Firewall-1-INPUT
Chain RH-Firewall-1-INPUT (2 references)
target     prot opt source               destination         
ACCEPT     tcp  --  anywhere             anywhere            tcp dpt:domain state NEW,RELATED,ESTABLISHED 
ACCEPT     udp  --  anywhere             anywhere            udp dpt:domain 
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     icmp --  anywhere             anywhere            icmp any 
ACCEPT     esp  --  anywhere             anywhere            
ACCEPT     ah   --  anywhere             anywhere            
ACCEPT     udp  --  anywhere             224.0.0.251         udp dpt:mdns 
ACCEPT     udp  --  anywhere             anywhere            udp dpt:ipp 
ACCEPT     tcp  --  anywhere             anywhere            tcp dpt:ipp 
ACCEPT     all  --  anywhere             anywhere            state RELATED,ESTABLISHED 
ACCEPT     tcp  --  anywhere             anywhere            state NEW tcp dpt:ssh 
REJECT     all  --  anywhere             anywhere            reject-with icmp-host-prohibited 
Using iptables -I, we insert our rules at the beginning of the ruleset, we need to have our rules come before the REJECT rule at the bottom of our chain. Now we can test access to our nameserver from another machine.
[user@client0 ~]$ nslookup ns1.example.com 192.168.0.1
Server:		192.168.0.1
Address:	192.168.0.1#53

Name:	ns0.example.com
Address: 192.168.0.1

We are almost done, in order for our new rules to be used the next time server1 is rebooted, we need to save the iptables rules into the configuration file.
[root@server0 data]# cd /etc/sysconfig
[root@server0 sysconfig]# cp iptables iptables.$(date +%Y-%m-%d)
[root@server0 sysconfig]# iptables-save >iptables
Our iptables rules should be saved now and will be used on the next reboot our our server. Since we will be giving out addresses on the 192.168.0.0/24 subnet, we will also serve out this zone with our named server and add a few records to our example.com zone file.

db.192.168

$TTL 3D
@       IN      SOA     localhost. root.localhost.  (
00	; Serial
86400	; Refresh
7200	; Retry
2592000	; Expire
345600 )	; Minimum
	NS	ns0.example.com.
1	PTR	server0.example.com.
16	PTR	client0.example.com.
17	PTR	client1.example.com.
18	PTR	client2.example.com.
19	PTR	client3.example.com.
20	PTR	client4.example.com.
21	PTR	client5.example.com.
22	PTR	client6.example.com.
23	PTR	client7.example.com.
24	PTR	client8.example.com.
25	PTR	client9.example.com.
26	PTR	client10.example.com.
27	PTR	client11.example.com.
28	PTR	client12.example.com.
29	PTR	client13.example.com.
30	PTR	client14.example.com.
31	PTR	client15.example.com.

In order for named to use this file, we need to add it a zone definition to named.conf

zone "0.168.192.in-addr.arpa" in {
	type master;
	file "data/db.192.168.0";
};

example.com

$TTL 3D
@       IN      SOA     ns0.example.com. root.example.com.  (
00	; Serial
86400	; Refresh
7200	; Retry
2592000	; Expire
345600 )	; Minimum
		NS	ns0
ns0		IN	A	192.168.0.1
server0		IN	A	192.168.0.1
client0		IN	A	192.168.0.16
client1		IN	A	192.168.0.17
client2		IN	A	192.168.0.18
client3		IN	A	192.168.0.19
client4		IN	A	192.168.0.20
client5		IN	A	192.168.0.21
client6		IN	A	192.168.0.22
client7		IN	A	192.168.0.23
client8		IN	A	192.168.0.24
client9		IN	A	192.168.0.25
client10	IN	A	192.168.0.26
client11	IN	A	192.168.0.27
client12	IN	A	192.168.0.28
client13	IN	A	192.168.0.29
client14	IN	A	192.168.0.30
client15	IN	A	192.168.0.31
We can now move on to configuring the webserver to allow access to the installation files we need.


*If this is the first server on your network then you might not have any dns servers yet, you can leave the forwarders line out of the named.conf in this case. Optionally you may wish to use the DNS servers provided by your ISP. **IP addresses in the range 10.0.0.0 - 10.255.255.255, 172.16.0.0 - 172.31.255.255 and 192.168.0.0 - 192.168.255.255 are reserved for use by organisations internally and are not routable on the internet (http://www.faqs.org/rfcs/rfc1597.html). As such, we will use 192.168.0.1 as the ip address of our installation server and 192.168.0.0/24 as the domain (192.168.0.0/24 is shorthand for saying our machines will have addresses in the range 192.168.0.1 - 192.168.0.254) +This is the default name on RedHat Enterprise Linux (RHEL) derived distributions. On fedora the name remains INPUT. ++You can determine which port a service is running on using lsof or by looking in /etc/services. Using lsof you can grep for open tcp ports
[root@server0 install]# lsof -i -n |grep named
named     5808 named   20u  IPv4 159190       UDP 127.0.0.1:domain 
named     5808 named   21u  IPv4 159191       TCP 127.0.0.1:domain (LISTEN)
named     5808 named   22u  IPv4 159192       UDP 192.168.0.1:domain 
named     5808 named   23u  IPv4 159193       TCP 192.168.0.1:domain (LISTEN)
named     5808 named   24u  IPv4 159194       UDP *:33883 
named     5808 named   25u  IPv6 159195       UDP *:41551 
named     5808 named   26u  IPv4 159196       TCP 127.0.0.1:rndc (LISTEN)
named     5808 named   27u  IPv6 159197       TCP [::1]:rndc (LISTEN)
From this output, we can see that named is LISTENing on the port domain, by looking in /etc/services, we see that domain is port 53.
[root@server0 install]# grep -w ^domain /etc/services
domain		53/tcp				# name-domain server
domain		53/udp
named uses both tcp and udp, we can see this by the first line in our grep output UDP 127.0.0.1:domain. We also see in /etc/services that domain is registered for both tcp and udp connections.

httpd

We'll use the webserver to serve out our installation media, we could also use httpd to serve out our kickstart files as well. Now that httpd is installed, verify that it is configured to start on boot.
[root@server0  Server]# chkconfig --list httpd
httpd          	0:off	1:off	2:off	3:off	4:off	5:off	6:off
If httpd were configured to start automatically at boot time, it would show on instead of off for run levels 3,4 and 5 above. Enable httpd at boot:
[root@server0  Server]# chkconfig httpd on
[root@server0  Server]# chkconfig --list httpd
httpd          	0:off	1:off	2:on	3:on	4:on	5:on	6:off
Our web server will serve out the rpms on the install media (as well as a few of our own). As a first step, copy all the install RPMs off the install media to a directory on the webserver.
[root@server0  ~]# mkdir /mnt/install
[root@server0  ~]# mount /dev/cdrom /mnt/install
mount: block device /dev/cdrom is write-protected, mounting read-only
[root@server0  ~]# cd /mnt/install
[root@server0  install]# mkdir /var/www/html/install
[root@server0  install]# cp -pr Client images isolinux VT Workstation /var/www/html/install
Now, with our files installed we'll need to configure the webserver to serve up our install media. We'll call the server install.example.org, we'll set up a new apache configuration file for the server and allow Indexing of the subdirectories of /var/www/html/install. /etc/httpd/conf.d/install.conf
<Directory /var/www/html/install>
 Options +Indexes
</Directory>
This file will ensure that indexes are shown in the subdirectories of install. You can now start up httpd and verify that the files are available via http.
[root@server0 conf.d]# service httpd start
Starting httpd: httpd: apr_sockaddr_info_get() failed for server0.example.com
httpd: Could not reliably determine the server's fully qualified domain name, using 127.0.0.1 for ServerName
                                                           [  OK  ]
[root@server0  conf.d]# wget http://127.0.0.1/install
--09:49:22--  http://127.0.0.1/install
Connecting to 127.0.0.1:80... connected.
HTTP request sent, awaiting response... 301 Moved Permanently
Location: http://127.0.0.1/install/ [following]
--09:49:22--  http://127.0.0.1/install/
Connecting to 127.0.0.1:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1598 (1.6K) [text/html]
Saving to: `index.html'

100%[=======================================>] 1,598       --.-K/s   in 0s     

09:49:22 (254 MB/s) - `index.html' saved [1598/1598]
Now that our files are served up with apache, just as we did for dns, we need to make sure other machines can reach our httpd service. Earlier we discovered that the INPUT chain of our firewall is called RH-Firewall-1-INPUT. We need to add a rule to allow tcp port 80 through the firewall.
[root@server0 ~]# cd /etc/sysconfig
[root@server0 sysconfig]# iptables -I RH-Firewall-1-INPUT -p tcp -m state --state NEW,ESTABLISHED,RELATED --dport 80 -j ACCEPT
[root@server0 sysconfig]# iptables-save >iptables

Now we can test network access to our server from another machine on the network. You can use a web browser such as firefox or elinks.

Our new client will use http to download installation packages after it has finished loading the kernel. To load the kernel it uses a simpler transport called TFTP, we'll configure tftp in the next section.

tftp server

Tftp is the trivial file transfer protocol. It is a very low overhead file transfer protocol for which a client is contained in the boot roms of most ethernet cards. After our client machines receive their ip address information from dhcp, they will receive the ip address of our tftp server and a filename to load into memory and execute.

The tftp server is a service that runs from xinetd. xinetd is a server that runs at boot time and handles incoming connections on a number of services, it is sometimes called the "super server". To allow the tftp server to run, we need to enable xinetd first and then turn on tftp. [root@server0 ~]# chkconfig --list xinetd xinetd 0:off 1:off 2:off 3:on 4:on 5:on 6:off [root@server0 ~]# chkconfig --list tftp tftp off [root@server0 ~]# chkconfig tftp on [root@server0 ~]# service xinetd start Starting xinetd: [ OK ] Configuration files for xinetd are stored in /etc/xinetd.d. The configuration file for tftp is /etc/xinetd.d/tftp. Chkconfig makes turning on an xinetd service very simple, to enable a service manually, you need to edit it's xinetd configuration file and change the line that reads disable=yes to disable=no. Take a moment to look at this file and familiarise yourself with the configuration options. In particular, the root directory of the tftp server is set in this file.

Now that tftp is up an running, we will test it by transferring a file from the server using the client program tftp*

[root@server0 ~]# ls -l /etc/services
-rw-r--r-- 1 root root 362031 Feb 23  2006 /etc/services
[root@server0 ~]# cd /tftpboot
[root@server0 tftpboot]# cp /etc/services .
[root@server0 tftpboot]# cd
[root@server0 ~]# tftp localhost
tftp> get services
tftp> quit
[root@server0 ~]# ls -l services
-rw-r--r-- 1 root root 362031 Apr 30 23:33 services
Now that we know our tftp server is working properly, we need to make sure clients can reach the server, tftp runs on udp port 69.
[root@server0 ~]# iptables -I RH-Firewall-1-INPUT -p udp --destination-port 69 -j ACCEPT
[root@server0 ~]# iptables-save >/etc/sysconfig/iptables
The tftp protocol works differently than the other services we've covered so far. The client and server decide on ephemeral** ports to communicate on and then do the file transfer on those ports. Since our iptables rule only allows communication on port 69, we need to tell iptables to use a module that can track the ports used by tftp. This module is ip_conntrack_tftp, we enable the module in /etc/sysconfig/iptables-config. Find the line that starts with IPTABLES_MODULES= and add ip_conntrack_tftp to this line if it doesn't already exist. Reload iptables after that to load the module.
[root@server0 sysconfig]# grep "IPTABLES_MODULES=" iptables-config
IPTABLES_MODULES="ip_conntrack_netbios_ns"
[root@server0 sysconfig]# sed -i.bak -e 's/\(IPTABLES_MODULES=\"\)/\1ip_conntrack_tftp /' iptables-config
[root@server0 sysconfig]# grep "IPTABLES_MODULES=" iptables-config
IPTABLES_MODULES="ip_conntrack_tftp ip_conntrack_netbios_ns"
Flushing firewall rules:                                   [  OK  ]
Setting chains to policy ACCEPT: filter                    [  OK  ]
Unloading iptables modules:                                [  OK  ]
Applying iptables firewall rules:                          [  OK  ]
Loading additional iptables modules: ip_conntrack_tftp ip_c[  OK  ]_netbios_ns 

We can now try tftp from our client machine, but again due to the way tftp works, we need to load the ip_conntrack_tftp module on our client machine also.

[root@client0 ~]# service iptables restart
iptables: Flushing firewall rules:                         [  OK  ]
iptables: Setting chains to policy ACCEPT: filter          [  OK  ]
iptables: Unloading modules:                               [  OK  ]
iptables: Applying firewall rules:                         [  OK  ]
iptables: Loading additional modules: ip_conntrack_tftp    [  OK  ]
[root@client0 ~]# tftp server1
tftp> get services
tftp> quit
[root@client0 ~]# ls -l services
-rw-r--r-- 1 root root 362031 2009-05-01 15:59 services
Now that we've verified that tftp is working properly, we need the boot files for our clients, these are contained in the package system-config-netboot. The most important is the first file that is used to bootstrap the client, pxelinux.0
[root@server0 tftpboot]# yum install system-config-netboot-cmd system-config-netboot
...
Installed: system-config-netboot.noarch 0:0.1.45.1-1.el5 system-config-netboot-cmd.noarch 0:0.1.45.1-1.el5
Complete!
[root@server0 tftpboot]# ls linux-install/
msgs  pxelinux.0  pxelinux.cfg
At this point we have the dns server, tftp server and http server running, we need one more service to tie everything together, dhcp.
* tftp, the client program is in the rpm tftp. We tested this from the server, your server should have the minimum number of packages installed, for testing you can install the tftp rpm and then remove it when you are done using rpm -e tftp. ** ephemeral ports are pseudorandomly chosen ports that are typically highly numbered (much higher than 1024).

dhcp

The Dynamic Host Configuration Protocol (DHCP) is used to automatically assign ip addresses to clients. Addresses can be assigned randomly or discretely by mac address. The ipaddress of our server is 192.168.0.1, we will configure dhcp to give out addresses to our clients in the range 192.168.16 - 192.168.31. A note of caution, if you are not on your own private network at this point, you need to contact your network administration before starting up your own dhcp server. The first step after installing the dhcp server is to create a new dhcpd.conf. We will keep it as simple as possible. /etc/dhcpd.conf
ddns-update-style interim;
ignore client-updates;

subnet 192.168.0.0 netmask 255.255.255.0 {
	option routers 192.168.0.1;
	option subnet-mask 255.255.255.0;

	option domain-name	"example.org";
	option domain-name-servers	192.168.0.1;

	option time-offset	-18000;
	
	range dynamic-bootp 192.168.0.16 192.168.0.31;
	default-lease-time 21600;
	max-lease-time 43200;
}
On the version of dhcp installed on our system, the first line ddns-update-style interim; is required by the dhcp server. The subnet section specifies on which subnet we will be serving out addresses. The line which specifies the addresses to give out is range dynamic-bootp 192.168.0.16 192.168.0.31;. This specifies that the range of addresses from 16 to 31 will be given out dynamically (the first available address will be assigned to the next client, starting from the top of the range).

To test the dhcp server, we first start it and check the error log for any messages.

[root@server0 ~]# service dhcpd start; tail -f /var/log/messages
Starting dhcpd:                                            [  OK  ]
May 11 13:07:54 server0 dhcpd: Listening on LPF/eth0/00:11:22:33:44:55/192.168.0/24
May 11 13:07:54 server0 dhcpd: Sending on   LPF/eth0/00:11:22:33:44:55/192.168.0/24
May 11 13:07:54 server0 dhcpd: Sending on   Socket/fallback/fallback-net
If there were an error in our config file, dhcpd would fail to start and would output the reason to /var/log/messages.

Now for completeness we should allow dhcp requests through our firewall, dhcp listens on port 67 (which is known as bootp in /etc/services).

[root@server0 ~]# cd /etc/sysconfig
[root@server0 sysconfig]# iptables -I RH-Firewall-1-INPUT -p tcp --dport 67 -m state --state NEW,ESTABLISHED,RELATED -j ACCEPT
[root@server0 sysconfig]# iptables -I RH-Firewall-1-INPUT -p udp --dport 67 -j ACCEPT
[root@server0 sysconfig]# iptables-save >iptables
We can now test the dhcp server on a client machine, we will use dhclient to request an address.
[root@client1 ~]# dhclient eth0
Internet Systems Consortium DHCP Client V3.0.5-RedHat
Copyright 2004-2006 Internet Systems Consortium.
All rights reserved.
For info, please visit http://www.isc.org/sw/dhcp/

Listening on LPF/eth0/00:11:22:33:44:5a
Sending on   LPF/eth0/00:11:22:33:44:5a
Sending on   Socket/fallback
DHCPDISCOVER on eth0 to 255.255.255.255 port 67 interval 8
DHCPOFFER from 192.168.0.1
DHCPREQUEST on eth0 to 255.255.255.255 port 67
DHCPACK from 192.168.0.1
bound to 192.168.0.31 -- renewal in 10290 seconds.
Now that we have verified that our dhcp server is working, we will add a filename and next-server fields to our subnet definition. When machines boot via PXE they download the file specified by filename via tftp from the server specified by next-server*. If you do not run the dhcp server on the same server as your tftp, then you need to specify next-server accordingly. If for instance your tftp server is running on server2, you would put the following in the subnet definition:
next-server server2;
After adding these fields are added to our dhcpd.conf, we have our final dhcpd.conf
/etc/dhcpd.conf
ddns-update-style interim;
ignore client-updates;

subnet 192.168.0.0 netmask 255.255.255.0 {
	option routers 192.168.0.1;
	option subnet-mask 255.255.255.0;

	option domain-name	"example.org";
	option domain-name-servers	192.168.0.1;

	option time-offset	-18000;
	range dynamic-bootp 192.168.0.16 192.168.0.31;
	default-lease-time 21600;
	max-lease-time 43200;
	filename "linux-install/pxelinux.0";
	next-server 192.168.0.1;
}
Restart dhcpd to pickup the configuration change. You can now attempt a PXE boot of your client machine, it will fail at this point, but you can verify that pxelinux.0 is being loaded by the client and executed. We'll configure PXE in the next section.
* If you do not specify next-server in your dhcpd.conf, then the PXE client will attempt to broadcast for tftp on the network. To work in this mode, you will need a tftp server that responds to broadcast requests. At the time of writing, the in.tftpd package installed on our system will not do this.

PXE

After restarting our dhcp server, we can boot the client machine using PXE. The details of this vary from machine to machine, but on most of the machines I have you press F12 during the BIOS splash screen to enter a boot menu. From the boot menu you can select boot from network. Some machines are PXE capable but have the PXE option turned off. To turn it on you'll need to enter the bios setup (usually F1 or F2) and enable PXE on the network interface.

Here is result of an attempt to boot PXE on our client at this point.

CLIENT IP: 192.168.0.31 MASK: 255.255.255.0 DHCP IP: 192.168.0.1 GATEWAY IP: 192.168.0.1 PXELINUX 3.10 2005-08-24 Copyright (C) 1994-2005 H. Peter Anvin UNDI data segment at: 00097680 UNDI data segment size: 3980 UNDI code segment at: 0009B000 UNDI code segment size: 4950 PXE entry point found (we hope) at 9B00:00D6 My IP address seems to be C0A8001F 192.168.0.31 ip=192.168.0.31:0.0.0.0:192.168.0.1:255.255.255.0 TFTP prefix: linux-install/ Trying to load: pxelinux.cfg/01-00-11-22-33-44-5a Trying to load: pxelinux.cfg/C0A8001F Trying to load: pxelinux.cfg/C0A8001 Trying to load: pxelinux.cfg/C0A800 Trying to load: pxelinux.cfg/C0A80 Trying to load: pxelinux.cfg/C0A8 Trying to load: pxelinux.cfg/C0A Trying to load: pxelinux.cfg/C0 Trying to load: pxelinux.cfg/C Trying to load: pxelinux.cfg/default Could not find kernel image: linux boot:
Why does PXE search for all these files? The idea here is that you could create a configuration that applies to a specific MAC address or a specific ip address or even a range of ip addresses. If you wanted to have the same configuration on machines in the range 192.168.0.16 to 192.168.0.32, you would create a file named C0A8001. Since 16 is 10 in hexadecimal and 32 is 20 in hexadecimal, machines with ip addresses 192.168.0.16 through and including 192.168.0.31 would use this file.
What has happened here is that our client broadcast on the network for a dhcp address and received the address 192.168.0.31 (CLIENT IP) from our dhcp server 192.168.0.1 (DHCP ID). The client then loaded pxelinux.0 and executed it. Once executed, pxelinux.0 printed out the banner starting at PXELINUX 3.10 and it then began looking for a pxelinux.cfg file to load. It started with a representation of the client MAC address. The MAC address is a hardware address for the ethernet card in our client, in our case it is 00:11:22:33:44:5a. PXELinux attempts to load the configuration 01-00-11-22-33-44-5a from the pxelinux.cfg directory. Since we don't have any files in pxelinux.cfg, this file is not found, so it begins looking for other possible configuration files starting next with the hexadecimal representation of the client IP address. The client ip address is 192.168.0.31, using hexadecimal notation for each octet (remember that IP addresses are in the range 0 - 255 for each octet or number between periods) we translate 192 into C0, 168 into A8, 0 into 00, and finally 31 into 1F arriving at C0A8001F. This file is not found, so PXELinux starts stripping off the last character and trying again. It repeats the process all the way down to C and then looks for the file called default.

To allow our machine to boot, we will create a default configuration file in /tftpboot/linux-install/pxelinux.cfg

default linux

label linux
	kernel vmlinuz
	append initrd=initrd.img ramdisk_size=10000
This file specifies that our client should by default load the configuration that has label linux. Our definition of linux instructs pxelinux.0 to load the file vmlinuz as our kernel and append initrd=initrd.img ramdisk_size=10000 to the kernel command line. These files do not exist yet, so we will copy them from the install media to /tftpboot/linux-install. All paths in this configuration file are relative to /tftpboot/linux-install, pxelinux.0 informed us of this with the TFTP prefix: line in the screenshot above.
[root@server1 ~]# cd /tftpboot/linux-install     
[root@server1 linux-install]# cp /var/www/html/install/images/pxeboot/vmlinuz .
[root@server1 linux-install]# cp /var/www/html/install/images/pxeboot/initrd.img .
With all the files in place, we can restart out client and allow it to pxeboot. Your client should load the default file and begin loading the kernel we specified (vmlinuz).
PXELINUX 3.10 2005-08-24 Copyright (C) 1994-2005 H. Peter Anvin UNDI data segment at: -00095DB0 UNDI data segment size: 0000 UNDI code segment at: 0009C020 UNDI code segment size: 0000 PXE entry point found (we hope) at29C02:01060 My IP address seems to be C0A8001E 192.168.0.30 ip=192.168.0.30:192.168.0.1:192.168.0.1:255.255.255.016 TFTP prefix: linux-install/ Trying to load: pxelinux.cfg/01-00-11-43-e7-7d-32 Trying to load: pxelinux.cfg/C0A8001E Trying to load: pxelinux.cfg/C0A8001 Trying to load: pxelinux.cfg/C0A800 Trying to load: pxelinux.cfg/C0A80 Trying to load:8pxelinux.cfg/C0A8 Trying to road: pxelinux.cfg/C0A Trying to load:2pxelinux.cfg/C0 Trying to l ad: pxelinux.cfg/C Trying to load: pxelinux.cfg/default Loading vmlinuz................................ Loading initrd.img.............................................................. ................... Ready.

The client will boot into an install environment called anaconda. You could follow the installation wizard through the various options to configure your new client at this point. We are more interested in automating the process. Automatically choosing installation options is handled by a kickstart configuration file. We'll build a kickstart file in the next section.

kickstart

All the services we have configured up to this point were necessary to allow clients to boot off the network. We now need to work on configuring how the machines will be installed. We will be using kickstart to specify where to find the boot media, which packages to install and which scripts to run after installation has finished.

Kickstart files are plain text files, you can create them using a text editor or the graphical utility system-config-kickstart or even by copying the file from an already installed machine. When you install a machine a record of your installation options is kept in the file anaconda-ks.cfg in root's home directory.

system-config-kickstart offers a nice graphical utility and is a good starting point for your first kickstart file, but I find that I more often rely on editing the file myself. If you wish to use system-config-kickstart, it is in the package of the same name.

There are 4 sections to a kickstart file:

commands

The commands section is where you specify how to partition your hard disk, how to configure your network, what hostname to give the machine and various other settings.
There are many options to the commands shown in the example. For a more detailed explanation of the various commands and their options, see the Red Hat manual here

Or my guide here

We will start by defining which type of install we will be doing and which profile to use

# install options
install
key Workstation
text

These two lines specify how to install the machine. install here means we are performing a new install another possible option here is upgrade. key specifies which installation profile to use. Other options from our installation media are Client, Workstation and VT. We use the text command to tell anaconda to use text mode to install.

Now since we are going to install with http, we'll need the network interface configured, we'll use the network command to configure our interface using dhcp.

# eth0 dhcp
network --device=eth0 --bootproto=dhcp --onboot=on
Other methods of configuring the network interface are static and bootp. We'll next specify the installation media
# installation media
url --url=http://server0.example.com/install

This next line tells anaconda where to find installation media. Other options here are CD-Rom, NFS, FTP or a partition on the hard drive. Here are some examples of the usage of those options.

nfs --server=server0.example.com --dir=/install
url --url=ftp://installer:fedora@server0.example.com/install
harddrive --dir=install --partition=1

Next we'll configure the language, keyboard and time options

lang en_US.UTF-8
keyboard us
logging --level=info
timezone America/New_York

Now we can move on to configuring some security options on the system. We'll configure selinux, the root password, iptables rules and the authentication mechanism.

selinux --enforcing
firewall --enabled --ssh
rootpw --iscrypted $1$F/cD2/$nV0/biUdPjDgea.cN2rEe.
auth --useshadow --enablemd5

SELinux is set to enforcing and we are allowing ssh through the firewall. Anaconda knows about http, ftp, telnet, smtp and ssh. You can enable any one of these through your firewall by adding them to the firewall line. If you wish to allow another port through the firewall you can use the syntax --port=[port number]. In our example we only wish to allow ssh into our new machine.

The root password is crypted using md5, you can create this yourself using the md5 perl library but it's a bit easier to just use grub-md5-crypt. In this example I'm using the password "fedora".

[root@server0 ~]# grub-md5-crypt
Password: 
Retype password: 
$1$F/cD2/$nV0/biUdPjDgea.cN2rEe.

You can choose at this point to configure the X Window system. If your goal is to create a server then you should use the option skipx. If you wish to configure X, then use xconfig, for example:

xconfig  --defaultdesktop=GNOME --depth=32 --resolution=1280x1024
We'll skip X for our client and also tell it not to use firstboot. Firstboot is a program that is intended to help you setup a machine after installing, it allows you to configure authentication, add users and various other things you would do on an initial setup.

skipx
firstboot --disable

Next we'll need to tell anaconda how to boot the system and partition the hard drive. I prefer to use lvm to manage disks rather than multiple partitions on the disk (note that system-config-kickstart doesn't understand any of the lvm commands we'll use). We'll need to have at least 2 partitions while using lvm. The first partition is /boot which is used to load the kernel and initrd images. The next partition will be an LVM partition to hole our Physical Volume.

# disk partitioning
bootloader --location=mbr
clearpart --all --initlabel
part /boot --asprimary --bytes-per-inode=4096 --fstype="ext3" --size=150
part pv.2 --size=0 --grow
volgroup ClientVolume --pesize 32768 pv.2
logvol swap --fstype swap --name=SwapVol --vgname=ClientVolume --size=1024 --grow --maxsize=8192
logvol / --fstype ext3 --name=RootVol --vgname=ClientVolume --size=8192 --grow

We use the command bootloader to specify where to install the bootloader and then use clearpart to wipe the disk clean and relabel it. We then specify to make a primary partition of 150MB and mount it as /boot. Then we create a new physical volume to store our logical volumes (part pv.2 --size=0 --grow). We specified the size of the pv.2 partition as 0 and give the option --grow to have the partition fill the remainder of the disk. With our physical volume created, we can create a Volume Group to contain our logical volumes using volgroup. Now we create the logical volumes using logvol. Using a combination of --size, --grow and --maxsize we can fill the disk with logical volumes that fit a variety of disk sizes. In our example we specify that the swap volume must not be smaller than 1GB and that the root volume must be no smaller than 8GB. If we have a 16GB drive, then we will get an 8GB swap and an 8GB root, if our drive is larger than 16GB (plus the 150MB for /boot) then root will grow into the remaining space.

%packages

Note:If you are using a Red Hat Enterprise 5 (RHEL5) system (or one derived from RHEL5, CentOS for example), system-config-kickstart does allow you to select packages, the packages selection section is disabled. Fedora based systems have this bug fixed.
In the packages section you can select individual packages or groups of packages. To obtain a list of the package groups available in your install, try the following simple python script.
#!/usr/bin/python

import yum

yb = yum.YumBase()
yb.doConfigSetup()
yb.doTsSetup()
for grp in yb.comps.groups:
	print "%s (%s)" % (grp.name,grp.groupid)
The above script outputs the name of the group followed by the groupid in parenthesis. The groupid must be used in the %packages section. In our example we will be installing a very simple client, we will only add the base and core groups to our kickstart. We will also want puppet and augeas to be installed, so we will add those packages to our list. To specify that packages should not be installed, prepend a - (minus/hyphen) in front of them. Our %packages section.
%packages
# groups
@base
@core
# additions
augeas
ntp
puppet
ruby-augeas
vim-enhanced
# subtractions
-bluez-gnome
-bluez-libs
-bluez-utils
(We added ntp and vim-enhanced because I like to have my systems keep good time and because I like the syntax highlighting of vim).

%post

Commands placed in the %post section are run after the system has finished installing. Normally these commands are run in the chrooted environment of the new host, if you wish to run the commands outsite the chroot, append --nochroot to the %post section. The --interpreter command may be used similar to that used in the %pre section.

Later we will configure puppet to talk to our new server in the %post section. One useful thing to put in the post section is to change the virtual console to vt3, so that you can see the output of any commands you may have put in this section.

%post
chvt 3
echo "executing post install"
sleep 20
chvt 1
We'll now look at our complete example.

%pre

The %pre section is where you can specify commands to run before the system is installed. Commands placed here are not run in the chrooted install environment. %pre must come at the end of the kickstart file. You may append --interpreter to the %pre line to have the pre script run a different interpreter than /bin/sh

%include

using a %include can be very helpful. However, you should note that many of the utilities you expect to be on the system will not be present at this point. Moreover, dns will not be operational at ths point. In our example we used nslookup to determine the hostname, you should note that this is not the nslookup that will be on the installed system, it is the busybox nslookup. The output from nslookup in the install environment will be different than that of the full system. To help write your scripts, create a kickstart that has a read line %pre, then chvt to vt 2 (Ctrl-Alt-F2) and try your scripts from the install environment. You will then know which commands are available and more importantly how they will interact with your script.
A problem I have run into when deploying many different machines using kickstart is that the logical volume (lvm) information is the same on all the disks. It becomes a real pain if you want to move drives around (between machines). Since the names of the volgroups are the same, lvm gets confused. To fix this, I use the %include syntax and have the %pre script output the logical volume information.

%include can be used to include any other kickstart script in a kickstart script, so you can modularize your kickstart configs.

In this example, I added %include /tmp/disk.cfg in our default.cfg at the point where we specified the partition information.

...
# Disk partitioning information
part /boot --bytes-per-inode=4096 --fstype="ext3" --size=150
part pv.1 --bytes-per-inode=4096 --grow --size=0
%include /tmp/disk.cfg
...
%pre
IPADDRESS=`ifconfig eth0 |grep 'inet addr' |awk -F: '{print $2 }' |awk '{print $1 }'`
HOSTNAME=`nslookup $IPADDRESS| grep Name | tail -1 | awk '{print $2}'`
HOST=`echo $HOSTNAME |awk -F. '{print $1}'`
if [ "$HOST" = "" ]; then
        HOST="unconfigured"
fi
chvt 2
echo "appending ${HOST}_volume partition information to kickstart" 
cat </tmp/disk.cfg
volgroup ${HOST}_volume --pesize 32768 pv.1
logvol swap --fstype swap --name=SwapVol --vgname=${HOST}_volume --size=1024 --grow --maxsize=8192
logvol / --fstype ext3 --name=RootVol --vgname=${HOST}_volume --size=8192 --grow
EOF
sleep 5
chvt 1
This code will try and configure the volgroup to use the name hostname_volume. If it fails to find the hostname (dns does not work in a %pre script), it will default to using the name unconfigured_volume.

complete example

Here is our commands, %packages, %pre and %post sections.
#platform=x86, AMD64, or Intel EM64T
# System authorization information
auth  --useshadow  --enablemd5 
# System bootloader configuration
bootloader --location=mbr
# Partition clearing information
clearpart --all --initlabel 
# Use text mode install
text
# skip rhn
key --skip
# Firewall configuration
firewall --enabled --ssh  
# Run the Setup Agent on first boot
firstboot --disable
# System keyboard
keyboard us
# System language
lang en_US
# Installation logging level
logging --level=info
# Use network installation
url --url=http://server0.example.com/install
# Network information
network --bootproto=dhcp --device=eth0 --onboot=on
#Root password
rootpw --iscrypted $1$F/cD2/$nV0/biUdPjDgea.cN2rEe.

# SELinux configuration
selinux --enforcing
# Do not configure the X Window System
skipx
# System timezone
timezone  America/New_York
# Install OS instead of upgrade
install
# Disk partitioning information
part /boot --bytes-per-inode=4096 --fstype="ext3" --size=150
part pv.1 --bytes-per-inode=4096 --grow --size=0
%include /tmp/disk.cfg
%post 
chvt 3
echo "executing post install"
chvt 1

%packages
@base
@core
ntp
vim-enhanced
-bluez-gnome
-bluez-libs
-bluez-utils
@localclient

%pre 
echo "this is the %pre"
IPADDRESS=`ifconfig eth0 |grep 'inet addr' |awk -F: '{print $2 }' |awk '{print $1 }'`
HOSTNAME=`nslookup $IPADDRESS| grep Name | tail -1 | awk '{print $2}'`
HOST=`echo $HOSTNAME |awk -F. '{print $1}'`
if [ "$HOST" = "" ]; then
	HOST="unconfigured"
fi
cat </tmp/disk.cfg
volgroup ${HOST}_volume --pesize 32768 pv.1
logvol swap --fstype swap --name=SwapVol --vgname=${HOST}_volume --size=1024 --grow --maxsize=8192
logvol / --fstype ext3 --name=RootVol --vgname=${HOST}_volume --size=8192 --grow
EOF

Ok, now that we have a complete example, we need to configure pxe to use the kickstart file.

pxe config

Now with our kickstart file created, we need to place the kickstart file on our web server so the client can pickup the kickstart file at boot.
[root@server0 ~]# cd /var/www/html
[root@server0 html]# mkdir kickstart
mkdir: cannot create directory `kickstart': File exists
[root@server0 html]# cd kickstart
[root@server0 kickstart]# cat default.cfg 
#platform=x86, AMD64, or Intel EM64T
# System authorization information
auth  --useshadow  --enablemd5 
# System bootloader configuration
bootloader --location=mbr
...
[root@server0 kickstart]# cd /tftpboot/linux-install/pxelinux.cfg
[root@server0 pxelinux.cfg]# cat default 
default linux

label linux
	kernel vmlinuz
	append initrd=initrd.img ramdisk_size=10000 ks=http://server0.example.com/kickstart/default.cfg
[root@server0 pxelinux.cfg]# 
With a complete example we can attempt to pxe boot a client. If your client fails to install itself unattended, answer the questions that the installer (anaconda) asks and wait for the install to complete. At the end of the install you can compare the anaconda.ks file in root's home directory to the one you created. You can compare the differences and update your example kickstart.

The goal here is to have the client install completely unattended.

The install itself is pretty vanilla at this point though. We could spend a lot of time tuning the kickstart to our needs, but our new machines will only be in sync when we reinstall them. To keep things synchronized, we need a better system, which is where puppet will help us out.