Designing an infrastructure for booting linux from iSCSI

The previous post outlined the components needed to boot linux with iSCSI root:

• A kernel
• An initial root-fs with necessary drivers and scripts for mounting an iSCSI-volume
• The root-fs on iSCSI
• Configuration

Keep in mind that all of this has to fit together. The init-root and iSCSI-root need a /lib/modules directory which has the kernel-modules compiled for the specific kernel loaded. I practice, the the init-root is built for one specific kernel, the actual root might support several (i.e., it will typically contain modules for each kernel ever installed), but userland tools might work only with the latest. The configuration needs to take this into consideration

Now, for actual requirements:

• The iSCSI-root should be as general as possible, meaning I don't want any specific configuration inside this filesystem (there are some things actually needed, more on that below). This is because I would like to be able to create a new instance by cloning a template and booting this without having to change any config file inside the filesystem itself.
• The same goes for the init root-fs and the kernel, these should be re-used across several instances.
• There must be a naming scheme which makes it easy to understand which components go together
• There should be no redundancy in the configuration, the same things should not be configured several places (as an inconsistent configuration could lead to hard-to-track bugs)
• The configuration should be as concise as possible, just list the things that actually vary in a brief format

There are also a few other parameters that will (or should) vary between instances:

• Ethernet MAC-address. This will be my preferred way of locating which configuration to use, that is: given a MAC-address, the configuration should determin all other parameters
• IPv4-address, determined by MAC-address using DHCP
• IPv6-address, I will probably not use DHCP, but instead use the router-solicitation mechanism in IPv6, as this is simpler and more elegant
• Filesystem UUID. I don't think it matters if separate machines have the same UUID-s, but if I some day might try to mount these filesystems on the same machine, it will probably get confused.
• Hostname. This could be determined by DNS, but it would probably be an advantage if the machine knows its name even if network hasn't come up yet. It could also be the other way around, that DNS is updated by DHCP when the IP is assigned. (This is more or less out-of-the-box functionality with many DHCP-servers. Unfortunately, there seems to be no standard solution for IPv6 yet)
  
• SSH-keys, re-using these across machines would be a security issue.
• iSCSI-initiator id, a unique name identifying the client.

The design for handling these parameters could be as follows:

• The MAC-address is either determined by the ethernet-card (in a physical machine) or by the hypervisor-tools (in a virtual one)
• IP-addresses are assigned automatically based on MAC
• Boot-parameters (location of kernel and init root, along with parameters to mount the iSCSI-volume) could be set by the hypervisor or the DHCP-server
• Filesystem UUID should be set when creating it (so if we are creating a new instance by cloning a template, it should be immediately followed by a change of UUID). Keep in mind that /etc/fstab should refer to something else than UUID (I am thinking volume label, but I will get back to this)
• SSH-keys will be generated on first boot (the ssh startup script should check if keys exist, and generate them if not)
• Hostname could be set manually on first boot.
• iSCSI initiator id must be set outside of the bootet system, as this needs to be available before it has access to the actual root-fs, but it must also be known to the machine itself, because it might want to attach other iSCSI-interfaces after it has booted. One possibility is to create an address-structure based on MAC (even one more id attached to the MAC...)

How to boot linux from iSCSI

I would like to boot virtual (or physical) linux hosts from the network, with an iSCSI-device as root.
There are bits and pieces of information concerning this available, but I have not been able to find a complete guide to how this is done.

This is what I would like to do:

• Load the kernel, either using the Xen bootloader or with PXE (network boot)
• Attach to an iSCSI-disk available somewhere on my local network (in iSCSI-terms: let the newly booted linux be an iSCSI-initiator and log it in to an iSCSI-target)
• Mount the iSCSI as root
• Continue booting from the new root-filesystem

This is more or less the same as booting from NFS, but iSCSI is far more efficient than NFS

To get this to work, we need the following pieces:

• A kernel that can loaded before the filesystem is mounted, which means that it needs to be copied over to another location outside of the host we want to boot. For xen, this can be anywhere on the Dom0 filesystem, but to be really general and independent of any local hardware, it should be put on a tftp-server
• The initial root filesystem (initrd.img) needs to be in the same place
• Some configuration telling the bootloader where to find the kernel and the root filesystem. With XEN, this could be the xen config file, but for PXE, it should be placed in the dhcp-configuration.
• After booting the kernel with the initial root-fs, it will need some tools to mount the actual root-fs from iSCSI:
  
  • Kernel modules for iSCSI. I'm not exactly sure what the minimum set is, but we probably need atleast iscsi_tcp.
  • Tools to connect to iSCSI, iscsistart
  • Configuration of where to find the iSCSI-volume
  • Scripts pulling everything together
• Scripts run by init on the initial root file-system will mount the actual root from iSCSI and continue booting from this.
  

Se initrd(4) for a detailed and general description of the boot-process. (However, this seems a bit out of date, because it refers to LILO and LOADLIN, and seems unaware of GRUB. Documentation/initrd.txt seems just as outdated.)

I have investigated the kernels bundled with the latest ubuntu and debian variants and found that:

• Both have the neccessary kernel modules included in initrd.img
• Debian have scripts for setting up iSCSI included in initrd.img
• None have the iscsistart-tool

But wait a minute, what really happens here? Actually, the initrd.img is not copied from an installation-archive, it is generated when installing the kernel. This is infact documented, if you know what to look for:

• initramfs-tools is the tool actually generating the image. The contents is from /usr/share/initramfs-tools. Some other packages put contents here.
• open-iscsi provides the iscsi-script mentioned above
• It also provides the iscsistart command, but apparently not in a location picked up by the initramfs-tools.

Further digging and searching (use the source, Luke...), and voila: if you create /etc/iscsi/iscsi.initramfs with default values for the iscsi-configuration, the neccessary files will be included when generating the initramfs. This is actually described here.

This was the general info on how this fits together, some recommendations to actual setup will come in a later post.

Powershell for Unix-users

Powershell is an object-oriented scriptinglanguage bundled with windows-7 and windows server 2008 which is heavily influenced by unix scripting, python, perl, lisp and more. This guide lists some common unix-commands and their powershell equivalents.  Please note that they are not completely equal, as unix-commands works on streams of bytes, typically split into lines, while powershell works on streams of objects.

grep	Where-Object, but see below for details
cd	Set-Location, but cd is an alias
cat	Get-Content, but cat and type are aliases

grep "pattern"

If input is a list of strings, the Select-String command is equal to grep:

Select-String -Pattern "pattern"

However, input will typically be a stream of object, and what you want to do is to filter this. Thus Perl grep or Common Lisp remove-if-not which both accepts a general selection function as parameter are more appropriate. The PowerShell command is inspired by SQL SELECT WHERE ...

Where-Object { current object is $_ }

troll i eske februar

kommende filmer fra filmweb.no, minus de som vises på gimle filmfest og minus de som ikke er "troll-i-eske-genre":

• Fjellet
• Miral
• 80 dager
• 14 kilometer
• onkel boonmee
• the adjustent bureau
• exteriors

netfilter

Linux has a very comprehensive set of modules for filtering and changeing network packets as they flow by the network stack. However, this framework has been through several major and quite a few minor re-designs (the last major one seems to be the introduction of the nf_conntrack subsystem i 2.6.15).

There are vast amounts of documentation available on the web, but very little is up to date with the last changes. I will try to summarize my experiences with kernel 2.6.32.

Some pointers for good places to start:

• This is a good introduction, which also covers the last changes (pdf)
• The authorative source is netfilter.org, but most documentation there is outdated by almost a decade.
• This tutorial from Oskar Andreasson is not too out of date
• You can also take a look at the wikipedia page for iptables which has a nice flowchart

A quick introduction for the impatient:

• Each network packet is sent through a set of tables
• For each table, the packet is sent through a set of chains, which chains depend on the final destination of the package; inbound, outbound or routing through (see the drawing on wikipedia for a full picture
• Each chain has a default rule, and can have a set of additional rules
• Each rule is a filter and a command. The filter can be anything that the netfilter framework can find in the package: interface, source, destination, port, protocol etc. The command is either a pre-defined one (DROP, ACCEPT, REJECT etc) or a user-defined table. The filter kan be regarded as an "if"-statement, and the command as a functin call. But pay attention that not all commands make sense in all tables or all chains. Read the documentation for details.
  

udhcpc

One of the utilities bundled in BusyBox is the udhcpc DHCP client.  This is a tool according to the unix principle of: do one thing and do it well.  The udhcpc command handles the DHCP protocol as described in RFC 2131, but it doesn't actually configure the network based on the replies.

However, the documentation available from busybox is not very comprehensive.  There is a man-page available which can be found by googling udhcpc - very small DHCP client.

The operation of udhcpc is simple and unix-y.  When it receives a reply from a DHCP-server, it calls a script with one parameter, which is one of:

• deconfig: remove configuration (when lease is lost or udhcpc starts)
• bound: moving from unbound to bound state (receives configuration)
• renew: lease is renewed
• nak: nak received from server
• leasefail: (not documented in the man-page): run if there is no reply after configured timeouts and retries

lots of other configuration parameters available as environment variables.  The example scripts included with busybox implements these by configuring udhcpd to call a script which calls: simple.$1 (and of course there are 4 scripts: simple.deconfig, simple.bound, simple.renew, simple.nak).  Simple and easy!

I have basically used the sample scripts that came with BusyBox (examples/udhcpc/*), but converted them to using the ip(8)-command instead of ifconfig(8). I also added logging to syslog with the following function:

log() {
  logger -p daemon.info -t dhcp $*
}

dnsmasq

dnsmasq is a wonderful little daemon providing dhcp (IPv4) and dns, and integrates these, so a client provided with an IP-address using DHCP will also be available through DNS.  And it is really easy to set up, it gets all relevant information from /etc/hosts and /etc/ethers, but you can also add extra parameters through command line or config file.

Configuration

The build system for dnsmasq doesn't come with configure, but there are really few options and these can be manipulated with the make command line, or, as I did, by changing config.h. I removed:

• tftp-support (cmdline: COPTS=-DNO_TFTP), if I want to boot machines through the network, I will use my regular fileserver for tftp, not the firewall
• script-support (cmdline: COPTS=-DNO_SCRIPT), I don't think I will need scripts, and including them seems like a potential security problem

Build with make and install in /usr/sbin.