I started around 11:05 AM to the nearest post office. I had to travel using LRT to the post office. I reached the post office at 11:20 AM. I was in the queue until 12:00 PM and was serviced in 3 minutes. The cost of the mailing was S$0.50 while the cost of the journey was (S$0.69 + S$0.69) to and fro. The efficiency of the transaction – wait until the end of the week, wait in queue for over 40 minutes, transport cast ~3 times the cost of actual transaction

Just to tickle our thoughts. Isn’t it time to go fully ‘e-*’ ?

In my Nokia E71, I noticed a time skew of around 2.5 hours (GMT+8-GMT-5.5 ?) when I switched on the phone after landing in the local airport. I thought the time synchronization hasn’t happened yet and waited. However, instead of correcting itself, the skew kept on becoming big and eventually it was 7 hours ahead of the local time. All my alarms, reminders would go off in the middle of the night waking us up even before we have actually slept

I turned off and on again my phone couple of times and still the problem would not go away. Now, it was showing time as per GMT+7 time zone. At this stage, I wondered if it could be the operator issue and looked at my other mobile (iPhone) which was also setup to auto-update from mobile operator. There were no issues and the phone synced well and it always showed the correct local time.

At this stage, I started to wonder what could be issue ? there were 4 combination possible – home mobile network, roaming mobile network, phone model / manufacturer and time zone differences. I did a look up table. My Nokia E71 is on Starhub Network in Singapore, and was connected to the Vodafone IN network automatically and was showing GMT+7 time zone time. My iPhone is on SingTel in Singapore and was connected to the Airtel network automatically and was showing the correct GMT+5.5 time zone time. My primary suspect was the mobile operators’ algorithm in sync roaming mobile phones. Initially I thought, perhaps my mobile phone is in GMT+8 so there was a bug in time synchronisation and the network was pushing GMT+8+localtime. However, this was not the case as well. It was pushing down GMT+7+localtime. This left me with the only possibility of Singapore being registered as GMT+7 in the network operator. Incidentally, the natural time zone of Singapore is also GMT+7 (see natural time zone of singapore). Any way, all is theories and I don’t have any concrete solutions.

If you have experienced such issues and know of an possibility, do drop an comment.

As you know, I am using Red Hat Enterprise Linux (RHEL) for my server for the last several years. After recovery I decided to continue using the same OS. Therefore, the natural choice was to use xen as the virtual machine monitor. I used this server downtime to add more RAM totaling to 4GB and now the entire root partition is also protected by RAID 5.

During the OS installation, I selected virtualisation to be installed. Basically, this means xen-enabled linux kernel would be installed by default. Everything seemed to be ok and I continued on to firstboot. After completion, I logged into the system and from a comman prompt, I did cat /proc/meminfo and to my shock the first line was MemTotal: 3753984 KB (3.58 GB) a whopping half-a-gb gone missing. This is no-go as then the VM would be running much slower with fewer resources

In theory, 2^32 = 4096 so the entire 4GB of RAM should be addressable by the OS as physical memory. If there are any exception then it should be rounded-off to some big number but here 3.58 GB doesn’t sound like it either. With no choice, I decide to re-install with the 64-bit version of RHEL with the thoughts that 64 bit might recognize the entire 4GB. I was a bit hesitant about compatibility of the application software with a 64-bit OS. However, there was no other choice except to take the risk.

As with 32 bit installation I selected virtualisation and this installed the xen-enabled 64-bit RHEL kernel. After firstboot, I checked the total memory and got another shock. The memtotal now is 3708928 KB. This is 45,056 KB less than 32 bit xen-enabled RHEL kernel. I was confused as to why a 64 bit kernel recognise less RAM compared to 32 bit OS. I googled around and didn’t find a direct answer. At the same time, some one was suggesting that PAE kernels recognise 4GB or more RAM. Also, I remembered about my another system where I have 8GB of RAM and running Ubuntu 8.10 64 bit with kernel version 2.6.27. Cat /pro/meminfo showed 8181976 kB (7.8 GB). I use vmware workstation and not xen as VMM in that box. I then did a yum search pae to select an appropriate PAE kernel. I discovered that PAE is a 32-bit kernel feature and is not available available for 64-bit kernel. It made sense; why will we need physical pages extension when 2^64 is itself a huge number. Now, my second choice is to use continue using 64 bit kernel (risking any future software incompatibilities), lose ~50MB more than corresponding 32 bit kernel. It didn’t add up to be a good choice. At this juncture, I thought about PAE 32 kernel with Xen might do the trick. I started my 3rd install of RHEL for the day.

The 3rd choice was to install a PAE 32 bit kernel, verify the RAM is fully accessible and then turn on visualisation. The installation was smooth and after first boot, as with earlier attempts I did cat /proc/meminfo and hurray I got MemTotal: 4149928 kB. I then did a uname -a and inferred that by default (without virtualisation selected at install) the anaconda system installer installs PAE enabled kernel if it detects 4GB or more RAM. This was a welcome note. Now, I again did a yum-search for xen and installed it and rebooted it. On the grub bootup screen I became wiser and learned the following

• 32 bit kernel + xen = okay
• 32 bit kernel + PAE = okay
• 32 bit kernel + PAE + xen = NOT POSSIBLE

The above summarises it all. If you want PAE, forget XEN; if you want XEN forget PAE. Now, I have to make a hard decision whether to stick with my VM strategy and be constrained with only 3.5GB of RAM which could mean that the guest OS have only a maximum of 3GB RAM OR forget virtualisation and use all 4GB of RAM. I then factored in my hardware features (not to be generalized). The server is more than 4 years old and does not have vTX (hardware virtualisation) feature. I learned that this would make the VM run slower and compound this with the fact that 0.5GB is not there, this might result in a very slow VM or may be it won’t be noticeable. I was already tired for the day and wasn’t interested in doing another combinatorial experiment this time with VM variable. Therefore, I decided to abandon my VM strategy and be happy with the increased resources. As a take-away from this experience I have compiled who would have made the memory missing. I actually don’t know the answer, if you know please drop a reply, thanks !

There are several suspects in my list

1. Linux kernel ? RHEL version 5 uses 2.6.18.x as the base kernel. I am wondering if the Linux kernel has been fixed some time after the above kernel release. The reasoning being, Ubuntu 8.10 is able to see all the 8GB RAM (of course it was without xen)
2. Xen ? Is it that xen-enabling the Linux kernel makes the memory vanish ? If so, what do we conclude of xen technology in general ? do the peers face the same missing memory issues ?
3. Xen + PAE could have solved my problem but they were not compatible. Why ?

With no expectations and in fact negative expectations, I didn’t want to destroy any of my trusted Windows XP systems . I had a 7 year old laptop (P4 and 1GB RAM) which I came with Windows XP and now serving as test machine for anything that came new (Fedora 10, Ubuntu 8.10, Open BSD among others). I decided to lose my Fedora 10 (it rocks btw) and risk with that Windows 7 will turn out to be.

The installation was very smooth. Very few questions being asked and the time it took was also reasonable given that the hardware is very old. When I rebooted for the first time after installation, I noticed that the startup time is very fast (< 20 secs ?). You should read this number taking my hardware into picture. Therefore, I would assume that it would be much faster in modern systems. Once I logged in I noticed that the system was much more smoother and more responsive that Windows Vistas. I would even say that on the same hardware, it was probably more responsive that the original OS, Windows XP.

I didn’t spend much time with the OS but in the quick 2 hours or so, I concluded that Windows 7 would be good product and worth a try. Let us wait and see. In the mean time, for more information click on Windows 7 .

Since 2002, I have been volunteering some of my free time to help out the local red cross unit in Singapore. The story begins somewhere in 2004 when we decided to procure a server for running some of the internal IT services used for blood donor recruitment, recognition, retention and recall. Over time, the server got loaded with more and more services and it was beginning to look like a crash could mean a lot of downtime.

It is not the case that we didn’t have any backup. With the IT budget that was available, I had configured a soft raid of level 1(replication) for two of the critical partitions of the server – /home and /var. All was well until last year when we began taking regular backups of important configuration files and storing them remotely conforming to business continuity best practices. Late last year, I decided to take local full backups.

The first sign of trouble seem to have been visible last year. As part of the backup process, we wanted to use an external usb hard disk. However, the server did not recognize the hard disk and create the respective /dev/ entries although dmesg was showing the usb device as being recognized. with no /dev/ entry the hard disk could not be mounted. I concluded at that time that it could be because of the mismatch between old usb1.x port and the new usb2 hard disk. I wasn’t very convinced with that hypothesis but I didn’t have any alternate theories as well. Now in hind sight I know this hypothesis is not right.

One fine day, the system console stopped responding normally and there was huge delays between echos for key strokes. I suspected a run-away process with potential memory leaks so just hit the reboot. On POST screen, BIOS reported that one of the hard disks is reporting a future failure (a SMART feature of new generation hard disks). Following this, the kernel refused to boot and dropped me into the maintenance prompt. After running fsck for well over an hour with hundreds of inodes being orphaned and added to the lost+found list, finally the server started booting…

During the startup, two daemons – httpd and mysqld did not come up owning to some missing files. These two services are key to all our web apps and thus the server came to a stand still. Coupled with this was the fact that the server stopped at console-login and did not start up in X although the runlevel listed the server being in 5. I decided it was not worthy to continue using this server as what all was broken could not be cumulatively listed let alone being fixed.

I didn’t even want to reboot since my intuition suspected that the server might never come up again. With the local external hard disk not recognizable the only option was to do a remote backup. Luckily, ssh was up and I was able to connect from outside to the server. However, I soon found out that scp was not found in the PATH and not on the filesystem. I did a which sftp and it was found. With a sigh of relief, I started to connect from outside. The relief was short lived because although the control connection could be established the data connection was not being created. There was no firewall and no other network blockage. I thought the missing scp could be the cause. Therefore, it seems that the only option was to dismantle the server and recover the data.

Luckily, one of my friends (Kumar c/o thoughtsware) accidentally discovered that although pull technique was not working, push technique using ftp after ssh-in was working. Armed with this, I scrambled to setup with a ftp server to be the receiving end. By setup, I mean it, as I don’t usually use ftp because it is insecure. I started to move data one file a time from the server to this ftp site. The data was flowing but at a very slow rate. After a few days of data transfer, I decided it was time for the surgical data recovery on site. Another good friend and a mentor, Harish from Red Hat, offered to help out with the recovery.

First, I wanted to test out if a flash drive was readable on the server. To my surprise, the crippled server was able to recognise the fat32 filesystem drive and created the necessary /dev/ entries. At this stage, I concluded that there was nothing wrong with the server hardware and something was not okay in the software when I earlier tried a ext3 formatted disk. I started copying the data into the thumb drive but very soon the data transfer halted with the process becoming non-responsive as well. We guessed the process may have hit a bad sector. We decided it was time to shutdown the broken OS and try our last alternative.

Our plan was that we boot the server using a live cd, Fedora 10 in this case. Then mount an external hard disk and the internal hard disks and dd out the partitions and attempt a system recovery using the binary img files. The only unknown factor being the partition was soft raid partitions and we haven’t done a recovery before on this filesystem type.

We booted the server using Fedora 10 live cd and successfully mounted the internal hard disks. Next, when we inserted the same external hard disk into the usb port, Fedora mounted it without any fuss. At this point, I infered that earlier attempts to mount it may have failed because root filesystem may have developed bad sectors at exactly the same spot where the entries are created.

Without any delay, using dd we copied all the 4 raw partitions (2 raw partitions per raid 1 partition) to the external hard disk. Our attempts to copy out the boot and root partitions did not succeed as dd complained of missing superblocks info. After copying the raid partitions out, we decided to try our luck with booting the server back into the original, now broken OS and try out copy out configuration files by mounted a thumb drive as earlier. However, on reboot the server didn’t go anywhere beyond grub prompt. The server was totalled.

Armed with the dd created partition binary files, I mounted the hard disk containing them in another Fedora 10 machine. Then I used losetup to mount the binary files as a loop back device. I then used mdadm to combine the respective component partitions and mounted them as a md0 device. Following this, I mounted this md0 device under a directory and was able to see the files. Finally, I performed a recursive copy of the data in these mounted partitions into a clean ext3 partition for restoring. In the end, we could not recover a few configuration files like apache, networking and such. However, majority of the data was recovered.

I am now awaiting the hardware vendor to replace the parts and hopefully the server would be online in a few days.

Based on this experience, new knowledge and technologies, our new server would be designed to be more reliable than the current one and in case of another crash, should be easier to recover as well. Our idea is to use virtualisation and run our current server in a VM. By this way, we can easily backup the entire server and also restore from backups more easily. Secondly, we can enforce security policies on the host OS and also on the guest OS while monitoring all the network traffic flowing to VM at the host level. The host will be responsible for providing a abstracted and redundant hardware layer to the VM taking care of the raid and other hardware specific features.

Feel free to comment on our recovery experience and the new server design strategy as well

]]> http://shruta.net/2009/04/a-linux-crash-burn-and-recover-experience/feed/ 3 http://shruta.net/2009/03/did-my-20th-blood-donation-on-saturday/ http://shruta.net/2009/03/did-my-20th-blood-donation-on-saturday/#comments Mon, 30 Mar 2009 05:06:09 +0000 Kris Flyer http://sptkrishnan.wordpress.com/?p=140 the heading says it all…

I started donating blood in April 2002 and until March 2009 I have donated 20 times. I am not sure if I should feel proud about it because in theory I could have donated up to 28 times (4 years a year is allowed in Singapore for whole blood donation). However, My target was 21 (or 3 times a year). So, I guess I am short of 1 time now and that amounts to 95% of goal.

I am hoping that I will be able to donate 5 more times this and next year to reach 25 in 2010 and get a silver medal

One of the exhibits that captured my interest was on computer criminals…

…the museum has used the opportunity to educate the public on the intricate difference between the words “Hackers” and “Crackers”. While, even to this date, many IT publications refer to computer criminals as hackers, I am happy to note that the exhibition has done its part to help to clarify the  difference. The following tells it all, thanks msichicago .

First off, you need a system where you can download the android sources from the web and compile. There are several platforms which are supported for the build process – Ubuntu 32 bit, 64bit and Mac OS X. My personal view being 32 bit OS is the best as it requires least modifications. If you have a ubuntu box sitting around you can start-off right away. If not, I suggest using a virtual machine instead of installing an OS. You can install the free vmware player from vmware and use a prebuilt ubuntu 32 bit virtual appliance for compiling Android. Either way, apply all pending security and software updates before setting up the Android development tool chain.

Your starting point for android system development is Android source website. Follow the instructions there to the dot. The “repo sync” or download process should take less than a hour if you are on a good broadband connection. The “make” will take nearly 3 hours on a raw metal machine. On a virtual machine based ubuntu system it might take more time. The compilation will produce 4 img files – boot, userdata, ramdisk, and recovery.

Now, before you flash the binary img files you need to take care of few more details. Follow the dream aka dev phone 1 instructions as we are compiling for the google dev phone 1. In step 5, you may get a error. This is just a path issue. Setup your path and you should be fine. Once you run the repo sync & make again, it will get some updates from the web and build the img files suitable for the dev phone.

Google website doesn’t tell you how to get the img files into the developer phone. You need to use two tools ‘fastboot’ and ‘adb’ to get the images into the phones. They are located in the folder “out/target/product/dream/” relative to the code branch you checked out – either main or cupcake. Add that directory to the path variable. Connect dev phone 1 to the PC and verify through ‘abd devices’ from the terminal. Adb will start a daemon on first run. Some users have reported that they problems identifying the devices. This seems to be mainly the case if you are using a VM. If this is the case, ’sudo’ all your commands. Once you have verified the device is identified by adb, next boot the phone into fastboot mode. For this, switch off your phone and then press and hold both the poweroff+back until you see the 3 Androids on skates and the words “fastboot” in the middle of the screen. Now issue “fastboot devices”. You should be able to see your phone and it should say it is in fastboot model. At this stage, you need to issue the following commands

fastboot flash boot boot.img

fastboot flash userdata userdata.img

fastboot flash system system.img

and finally fastboot reboot and you are done getting a new OS on the phone

ps: feel free to drop a comment or question

1. Are you living in Singapore ?
2. Are you on Facebook ?
3. Are you a blood donor ?

Then join us by clicking here Singapore Blood Donors .

Feel free to drop a comment if you have some suggestions on how to better engage blood donors

Again, most of the times I have got the same news from many sources and they are usually identical or mostly identical because the root source is usually one of them.

I have always wondered, why Google Reader does not group them together just like it groups emails together in Gmail or more appropriately in Google News. The server software in both of these products groups related news articles from various sources together automatically even if they are separated by a huge time difference. For example, today’s announcement that Apple has released 2.2.1 version of iPhone firmware has been carried by most of the tech sources. I have subscribed to 6 of them and all the news articles are exactly the same. Yet, there are 6 articles and not one Therefore, my conclusion is that Google has all the technologies in place – search, categorize, display. Yet, I haven’t seen the solution in Google reader since its inception and at least for the last one year since I started thinking of this useful feature.

I wonder when my wish will come true ?

Do you know of any RSS readers that does this already ? or know of a tweak in Google Reader that makes it possible ? or simply wish for this feature? then drop a comment, thanks

Updated on Jan 30 2009: Added a sample image below to show my wish pictorially. As you can see, the same news has been pulled from two different RSS sources at different times. In this case, they are next to each other. In many cases, they are not. If Google can club together like a conversation in Gmail or as related news articles in Google News, then it would save a lot of time and scrolling

* – Google reader is a RSS feed reader which picks up updates from websites on your behalf automatically. You need a Google account and access it here Google Reader

]]> http://shruta.net/2009/01/my-jan-wish-for-google-reader/feed/ 0