Creating a 64 bit EC2 Image

Recently a researcher we are working with gave me an VirtualBox image to deploy on EC2 with the fledgling “Desktop-to-Cloud” tool. Discovering that it was 64bit, I had some additional work on my hands. Up until now the migration tool had assumed all source images are 32bit. No problem, I thought, there should only be a few changes to the tool to adapt.

The first thing I did was retrieve a 64bit kernel from an existing Ubuntu AMI. I located an Ubuntu AMI matching my requirements (architecture = 64bit, type=instance, and location=eu-west-1), which according to this list was ami-1b9ca96f. I then quickly booted up an instance tar’d up the /boot, /lib/modules, and /lib64/modules, and copied them to my local host.

With the necessary kernel on hand, I then made the following changes to my VirtualBox -> EC2 image code:

  • Specify use of the new 64bit kernel.
  • Specify use of appropriate AKI, again matching architecture, type (instance or EBS) and region. I also had to pick an AKI that used pvgrub. According to the Amazon documentation, AKI aki-4feec43b met my criteria.

After the above changes, I fired-off my script and approximately 30 minutes later I had an AMI ready for testing. I created one m1.large instance, the cheapest spec available, and attempted to ssh in. First blocker: SSH connection failed. The host was also not responding to pings. OK – so I consulted the instance console log via the AWS EC2 web UI:

Gave up waiting for root device.  Common problems:
– Boot args (cat /proc/cmdline)
– Check rootdelay= (did the system wait long enough?)
– Check root= (did the system wait for the right device?)
– Missing modules (cat /proc/modules; ls /dev)
ALERT!  /dev/disk/by-label/uec-rootfs does not exist.  Dropping to a shell!

I immediately recognized that this was a problem with Grub – it was trying to load a root disk labeled “uec-rootfs” which is the default used in Ubuntu sever images, but not used in my images. After copying the Grub menu.lst file (used by pvgrub), I neglected to change the root disk configuration to match the default device mapping used for EC2 instances.

I changed entries in /boot/grub/menu.lst that looked like:

kernel          /boot/vmlinuz-2.6.35-24-virtual root=LABEL=uec-rootfs ro console=hvc0

to

kernel          /boot/vmlinuz-2.6.35-24-virtual root=/dev/sda1 ro console=hvc0

I then re-ran my AMI creation script and another 30 minutes later I was ready to test again. The instance started up, and again it was not ssh-able. But it was pingable – progress. Again I consulted the system console and saw the following at the tail:

The disk drive for /mnt is not ready yet or not present
Continue to wait; or Press S to skip mounting or M for manual recovery

I looked at the 64 image, it had /mnt defined. It then occurred to me that the device that fstab was directing to be mounted didn’t exist. I recalled that different instance type had didn’t ephemeral storage options. I had been running 32bit images as m1.small, but 64bit images must run on at least a m1.large. I consulted the AWS EC2 UserGuide and sure enough m1.small provides default extra storage on /dev/sda2, but m1.large  has extra storage on /dev/sdb.

I checked the swap mount as well. M1.medium has a swap partition available on /dev/sda3. M1.large does not come with a formatted swap disk, but if needed, a fellow EC2 user recommends formatting one of the spare disks (/dev/sdb or /dev/sdc) on the fly.

As a short term solution, I simply added in /etc/fstab the ‘nobootwait’ option to devices that are present on m1.small and not m1.medium, preventing their absence from halting the boot process. I found another user doing this when using the same image when going from a medium to a small instance. After this change the image booted properly and was usable.

In order to save myself time in the future for waiting for a whole bundle-upload cycle (which can take up to 1 hr for large images), I can  boot-ability locally with Xen locally. This would require creating template mount points for each instance type matching Amazon’s defaults. Another option is to explicitly configure the device mapping when creating the AMIs, using the –block-device-mapping option with the ec2-bundle-image command.

Advertisements

About Chris Willmore

Software Engineering Master's Student at Tartu University

No comments yet... Be the first to leave a reply!

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: