S. P. T. Krishnan's take on things…

This is part 2 to my earlier post “A Linux crash, burn and recover experience“. I had mentioned in there that I plan to use a Virtual Machine (VM) strategy for easy backup and porting in future. However, the reality forced me to abandon the approach. It could also be that I don’t know the trade secrets read on…

As you know, I am using Red Hat Enterprise Linux (RHEL) for my server for the last several years. After recovery I decided to continue using the same OS. Therefore, the natural choice was to use xen as the virtual machine monitor. I used this server downtime to add more RAM totaling to 4GB and now the entire root partition is also protected by RAID 5.

During the OS installation, I selected virtualisation to be installed. Basically, this means xen-enabled linux kernel would be installed by default. Everything seemed to be ok and I continued on to firstboot. After completion, I logged into the system and from a comman prompt, I did cat /proc/meminfo and to my shock the first line was MemTotal: 3753984 KB (3.58 GB) a whopping half-a-gb gone missing. This is no-go as then the VM would be running much slower with fewer resources

In theory, 2^32 = 4096 so the entire 4GB of RAM should be addressable by the OS as physical memory. If there are any exception then it should be rounded-off to some big number but here 3.58 GB doesn’t sound like it either. With no choice, I decide to re-install with the 64-bit version of RHEL with the thoughts that 64 bit might recognize the entire 4GB. I was a bit hesitant about compatibility of the application software with a 64-bit OS. However, there was no other choice except to take the risk.

As with 32 bit installation I selected virtualisation and this installed the xen-enabled 64-bit RHEL kernel. After firstboot, I checked the total memory and got another shock. The memtotal now is 3708928 KB. This is 45,056 KB less than 32 bit xen-enabled RHEL kernel. I was confused as to why a 64 bit kernel recognise less RAM compared to 32 bit OS. I googled around and didn’t find a direct answer. At the same time, some one was suggesting that PAE kernels recognise 4GB or more RAM. Also, I remembered about my another system where I have 8GB of RAM and running Ubuntu 8.10 64 bit with kernel version 2.6.27. Cat /pro/meminfo showed 8181976 kB (7.8 GB). I use vmware workstation and not xen as VMM in that box. I then did a yum search pae to select an appropriate PAE kernel. I discovered that PAE is a 32-bit kernel feature and is not available available for 64-bit kernel. It made sense; why will we need physical pages extension when 2^64 is itself a huge number. Now, my second choice is to use continue using 64 bit kernel (risking any future software incompatibilities), lose ~50MB more than corresponding 32 bit kernel. It didn’t add up to be a good choice. At this juncture, I thought about PAE 32 kernel with Xen might do the trick. I started my 3rd install of RHEL for the day.

The 3rd choice was to install a PAE 32 bit kernel, verify the RAM is fully accessible and then turn on visualisation. The installation was smooth and after first boot, as with earlier attempts I did cat /proc/meminfo and hurray I got MemTotal: 4149928 kB. I then did a uname -a and inferred that by default (without virtualisation selected at install) the anaconda system installer installs PAE enabled kernel if it detects 4GB or more RAM. This was a welcome note. Now, I again did a yum-search for xen and installed it and rebooted it. On the grub bootup screen I became wiser and learned the following

• 32 bit kernel + xen = okay
• 32 bit kernel + PAE = okay
• 32 bit kernel + PAE + xen = NOT POSSIBLE

The above summarises it all. If you want PAE, forget XEN; if you want XEN forget PAE. Now, I have to make a hard decision whether to stick with my VM strategy and be constrained with only 3.5GB of RAM which could mean that the guest OS have only a maximum of 3GB RAM OR forget virtualisation and use all 4GB of RAM. I then factored in my hardware features (not to be generalized). The server is more than 4 years old and does not have vTX (hardware virtualisation) feature. I learned that this would make the VM run slower and compound this with the fact that 0.5GB is not there, this might result in a very slow VM or may be it won’t be noticeable. I was already tired for the day and wasn’t interested in doing another combinatorial experiment this time with VM variable. Therefore, I decided to abandon my VM strategy and be happy with the increased resources. As a take-away from this experience I have compiled who would have made the memory missing. I actually don’t know the answer, if you know please drop a reply, thanks !

There are several suspects in my list

1. Linux kernel ? RHEL version 5 uses 2.6.18.x as the base kernel. I am wondering if the Linux kernel has been fixed some time after the above kernel release. The reasoning being, Ubuntu 8.10 is able to see all the 8GB RAM (of course it was without xen)
2. Xen ? Is it that xen-enabling the Linux kernel makes the memory vanish ? If so, what do we conclude of xen technology in general ? do the peers face the same missing memory issues ?
3. Xen + PAE could have solved my problem but they were not compatible. Why ?