How to boot linux from iSCSI

I would like to boot virtual (or physical) linux hosts from the network, with an iSCSI-device as root.
There are bits and pieces of information concerning this available, but I have not been able to find a complete guide to how this is done.

This is what I would like to do:

• Load the kernel, either using the Xen bootloader or with PXE (network boot)
• Attach to an iSCSI-disk available somewhere on my local network (in iSCSI-terms: let the newly booted linux be an iSCSI-initiator and log it in to an iSCSI-target)
• Mount the iSCSI as root
• Continue booting from the new root-filesystem

This is more or less the same as booting from NFS, but iSCSI is far more efficient than NFS

To get this to work, we need the following pieces:

• A kernel that can loaded before the filesystem is mounted, which means that it needs to be copied over to another location outside of the host we want to boot. For xen, this can be anywhere on the Dom0 filesystem, but to be really general and independent of any local hardware, it should be put on a tftp-server
• The initial root filesystem (initrd.img) needs to be in the same place
• Some configuration telling the bootloader where to find the kernel and the root filesystem. With XEN, this could be the xen config file, but for PXE, it should be placed in the dhcp-configuration.
• After booting the kernel with the initial root-fs, it will need some tools to mount the actual root-fs from iSCSI:
  
  • Kernel modules for iSCSI. I'm not exactly sure what the minimum set is, but we probably need atleast iscsi_tcp.
  • Tools to connect to iSCSI, iscsistart
  • Configuration of where to find the iSCSI-volume
  • Scripts pulling everything together
• Scripts run by init on the initial root file-system will mount the actual root from iSCSI and continue booting from this.
  

Se initrd(4) for a detailed and general description of the boot-process. (However, this seems a bit out of date, because it refers to LILO and LOADLIN, and seems unaware of GRUB. Documentation/initrd.txt seems just as outdated.)

I have investigated the kernels bundled with the latest ubuntu and debian variants and found that:

• Both have the neccessary kernel modules included in initrd.img
• Debian have scripts for setting up iSCSI included in initrd.img
• None have the iscsistart-tool

But wait a minute, what really happens here? Actually, the initrd.img is not copied from an installation-archive, it is generated when installing the kernel. This is infact documented, if you know what to look for:

• initramfs-tools is the tool actually generating the image. The contents is from /usr/share/initramfs-tools. Some other packages put contents here.
• open-iscsi provides the iscsi-script mentioned above
• It also provides the iscsistart command, but apparently not in a location picked up by the initramfs-tools.

Further digging and searching (use the source, Luke...), and voila: if you create /etc/iscsi/iscsi.initramfs with default values for the iscsi-configuration, the neccessary files will be included when generating the initramfs. This is actually described here.

This was the general info on how this fits together, some recommendations to actual setup will come in a later post.