Designing an infrastructure for booting linux from iSCSI

The previous post outlined the components needed to boot linux with iSCSI root:

• A kernel
• An initial root-fs with necessary drivers and scripts for mounting an iSCSI-volume
• The root-fs on iSCSI
• Configuration

Keep in mind that all of this has to fit together. The init-root and iSCSI-root need a /lib/modules directory which has the kernel-modules compiled for the specific kernel loaded. I practice, the the init-root is built for one specific kernel, the actual root might support several (i.e., it will typically contain modules for each kernel ever installed), but userland tools might work only with the latest. The configuration needs to take this into consideration

Now, for actual requirements:

• The iSCSI-root should be as general as possible, meaning I don't want any specific configuration inside this filesystem (there are some things actually needed, more on that below). This is because I would like to be able to create a new instance by cloning a template and booting this without having to change any config file inside the filesystem itself.
• The same goes for the init root-fs and the kernel, these should be re-used across several instances.
• There must be a naming scheme which makes it easy to understand which components go together
• There should be no redundancy in the configuration, the same things should not be configured several places (as an inconsistent configuration could lead to hard-to-track bugs)
• The configuration should be as concise as possible, just list the things that actually vary in a brief format

There are also a few other parameters that will (or should) vary between instances:

• Ethernet MAC-address. This will be my preferred way of locating which configuration to use, that is: given a MAC-address, the configuration should determin all other parameters
• IPv4-address, determined by MAC-address using DHCP
• IPv6-address, I will probably not use DHCP, but instead use the router-solicitation mechanism in IPv6, as this is simpler and more elegant
• Filesystem UUID. I don't think it matters if separate machines have the same UUID-s, but if I some day might try to mount these filesystems on the same machine, it will probably get confused.
• Hostname. This could be determined by DNS, but it would probably be an advantage if the machine knows its name even if network hasn't come up yet. It could also be the other way around, that DNS is updated by DHCP when the IP is assigned. (This is more or less out-of-the-box functionality with many DHCP-servers. Unfortunately, there seems to be no standard solution for IPv6 yet)
  
• SSH-keys, re-using these across machines would be a security issue.
• iSCSI-initiator id, a unique name identifying the client.

The design for handling these parameters could be as follows:

• The MAC-address is either determined by the ethernet-card (in a physical machine) or by the hypervisor-tools (in a virtual one)
• IP-addresses are assigned automatically based on MAC
• Boot-parameters (location of kernel and init root, along with parameters to mount the iSCSI-volume) could be set by the hypervisor or the DHCP-server
• Filesystem UUID should be set when creating it (so if we are creating a new instance by cloning a template, it should be immediately followed by a change of UUID). Keep in mind that /etc/fstab should refer to something else than UUID (I am thinking volume label, but I will get back to this)
• SSH-keys will be generated on first boot (the ssh startup script should check if keys exist, and generate them if not)
• Hostname could be set manually on first boot.
• iSCSI initiator id must be set outside of the bootet system, as this needs to be available before it has access to the actual root-fs, but it must also be known to the machine itself, because it might want to attach other iSCSI-interfaces after it has booted. One possibility is to create an address-structure based on MAC (even one more id attached to the MAC...)