The Linux Chronicles – Making Backups

PermaLink

This post is about backing up. More specifically, backing up in a way that is usable. That is, that precludes your having to do a full re-install if something happens to your disk.

In the linux world there are two kinds of backups:

File oriented backup simply means that you copy all your files from one place to another, “another” being another hard drive, a network drive or some other type of storage like DVDs. Partition-oriented backup means that you back up the whole partition. This gets the OS, all the files and everything else.  This is similar to drive cloning  in Windows

Critical to either backup method (unless you like swapping a LOT of DVDs) is a large external hard drive. By large I mean at least 400 gig. External USB interface hard drives are everywhere these days and are dirt-cheap. I got my 400 and 500 gig drives from Sam’s Club. They seem to always have the cheapest price at the moment when shipping is figured into alternatives.

Equally critical to success is to have a backup plan. A plan is not “I’ll backup whenever I think of it”. A plan is a method and automation to do it on a regular and fail-safe basis. A good plan might be “do a partition backup once a week and daily incremental file backups.”. In fact, that is my plan.

A root partition backup can’t be automated because the partition must be unmounted which isn’t possible for the partition that was booted. Therefore I have my Palm Pilot set to remind me to do a backup every Sunday evening. Daily incremental backups are automated.

Daily Backups

The solution for daily backups is simple – “Backup-manager”. This program is in the repository. Simply start the Synaptics partition manager from the desktop and search for “backup-manager”. Check it to be installed and let it download and install.

There is a tiny bit of setup to be done. There is a file called /etc/backup-manager.conf that must be edited. This might seem daunting but it isn’t. Simply open a terminal window from the desktop and execute the following:

$ sudo vi /etc/backup-manager.conf

if you’re comfortable with vi or

$ sudo gedit /etc/backup-manager.conf

if you’re more comfortable with a windowed editor.

The first thing to change is the destination. This will be your external hard drive. When the hard drive is plugged into a USB port, it automatically mounts in /media using the volume name that it is given. I changed my drives’ volume names to something more descriptive than “segate xiys-4958773″ or something similar. Descriptive names like “400 gig” and “500gig”. Therefore my 400 gig drive mounts as /media/400gig.

On that drive I have a directory for backups called “Linux_Backup”. So the destination for backups is “/media/400gig/Linux_Backups“. Remember that case matters in Linux so get your capitalization correct.

Back to the config file. At the top of the file you’ll see something like this:

##############################################################
# Repository - everything about where archives are
#############################################################
# Where to store the archives
export BM_REPOSITORY_ROOT="/media/400gig/Linux_Backup"

Simply edit the text inside the quotes to match your destination.

Got it? Good. We’re a third of the way done. Next, go down to where you see this:

# The backup method to use.
# Available methods are:
# - tarball
# - tarball-incremental
# - mysql
# - svn
# - pipe
# - none
# If you don't want to use any backup method (you don't want to
# build archives) then choose "none"
export BM_ARCHIVE_METHOD="tarball-incremental"

Simply edit BM_ARCHIVE_METHOD to be exactly as shown. What this means is that on the designated day of the week (Monday by default), a full backup will be made. On other days, only those files that changed will be backed up. That’s called an incremental backup.

Finally we specify what to back up. Find this section in the config file

# It's recommanded to use BM_TARBALL_TARGETS[] though.
# Warning! You *must not* use both variables at the same time.
# NOTE: The Debian package will only update BM_TARBALL_DIRECTORIES
# Paths without spaces in their name:
export BM_TARBALL_DIRECTORIES="/etc /home /usr /var /lib"

I strongly recommend backing up all the directories listed above.  Just copy that line directly into your config file.

That all there is. Save the file and sit back and relax. Sometime during the day every day this program will run and will back up your stuff. The default is to keep 5 generations of backups. That’s why you need a large drive. You can edit the config file to change that number if you like.

See, that wasn’t difficult at all.

Partition Backups

Partition Backups are no more difficult but they are more involved. That is, more steps are involved. We’re going to use the program partimage that I’ve discussed before. IF you don’t already have it then fire off Synaptics Package Manager and get it. Just search for “partimage” and install it.

Partimage usually resides in “/usr/sbin/partimage”. For this task we need it to reside on the backup drive. Therefore we’ll copy it over. In these examples my backup drive is named “400gig”. Substitute the name of your drive.

Open a terminal from the desktop and type the following:

$ cd /mount/400gig
$ sudo cp /usr/sbin/partimage .

That copies partimage into the root of the 400 gig drive.

Partimage can’t work on a partition that is mounted and subject to change at any time and yet we want to back up the root partition that contains everything in a default Ubuntu installation. The solution is to boot from the Live CD. That’s what we’re going to do.

Put the live CD in your CD or DVD drive, click in the red power symbol at the top right of your desktop and choose “restart”. Your system will shut down and will reboot into the live CD. Answer “English” (or your native tongue) and then choose “Try Ubuntu without changing your system” or something to that effect. It is the default option.

When the boot is finished and you have a desktop, open a terminal window and type the following:

$ cd /media/400gig
$ sudo ./partimage

You should see a screen similar to this:

partimage 01

To move from field to field use the <tab> key. Use the <space> bar to make or unmake a selection. Notice where I filled in my destination file name. When you have your screen matching this, press <F5> to go to the next screen:

partimage 02

The important parts are:

When you’re ready to start, press <F5>. Then go kill a half hour or so. When the backup is finished, reboot to your regular system and you’re done.

Restoration involves the same process except that you choose “restore a partition from an image file” on the first screen. The major benefit of this backup method is that it takes a snapshot of the system as it existed. A restoration puts it back exactly like it was when the snapshot was taken. No re-installing the system from the Live CD and copying files. No muss, no fuss. Just reboot and you’re up and running again.

Summary

To summarize, I use partimage to back up my root partition about once a week.  Every day my machine is backed up automatically by backup-manager.  Simple and clean.  And reliable.

If you’ve ever done a backup under windows,  you’ll really appreciate how much easier this is.  Oh so typical for Linux.  You’ll also notice that you didn’t have to spend any money on a backup program that uses a proprietary format and might not work when you need it to restore.  I’ve been bitten by that one more than once. Partimage is open source and free and always will be.  It will always work.  Again, that’s the Linux way.

Posted by neonjohn on March 2nd, 2009 under Computing |
  1 Comment »

:::del.icio.us :Digg it :StumbleUpon :Yahoo MyWeb ::

  The Linux Chronicles – Blue Smoke and ….

PermaLink

….agony.

So there I am on Monday night, reading Craig’s List and minding my own business when I hear a little pop from my laptop, my venerable 7 year old Dell Inspiron. Simultaneously, the screen goes blank and a wisp of magic blue smoke leaks out. Damn! My machine has committed suicide. Just when I had it set up perfectly.

I was on Craig’s List because I was looking for a new computer anyway. I sez to myself “Now the search is serious. I’ll just get back online and look harder”. Oh wait. That was my only computer other than my server.

I dug the server out of the closet – an ancient 400 mhz Compaq – and tried to install Ubuntu. I proved beyond a shadow of a doubt that Ubuntu and the Gnome desktop won’t run in 128mb of RAM. Damn. Well, let’s get more RAM. Oh wait. I’m offline. Well, let’s look for local computer dealers. Can’t do that either. I finally had to resort to the yellow pages!  Memo to self: get a second computer

I called every compute shop listed within about 100 miles and finally found a used Acer Aspire laptop (question of the day: where are all the used laptops going?). I drove over to nearby Athens, TN and bought it for the princely sum of $300. A couple of years old but many times faster than my old Dell.  Wonderful screen, sucky keyboard.

It came with Vista already installed. Not wanting to contaminate my cabin with those awful bits, I yanked out the drive and replaced it with an identical drive that I bought along with the computer. I figured that I’d make the drive available if I ever sell this machine. Anyway…

Now I faced the task of moving Ubuntu over from the old drive. This computer has a SATA drive so I could not use the procedure I outlined in a previous post. I developed a modified and somewhat streamlined method of moving that I’ll outline below.

This task is complicated by the fact that the new drive (80 gig) is smaller than the old one (120 gig). The partition copy won’t work directly. One must first shrink the partition on the 120 gig drive to a size small enough to fit on the new one.

I had a few problems – nothing serious. The internal modem did not have a driver on the Live CD. I’m not sure if one is even available. This machine does not have a serial port so I could not directly plug in my external modem. I did happen to have a USB/serial adapter laying around. It plugged in and the driver loaded without incident.

The serial adapter is mounted as /dev/ttyUSB0 so I had to edit my dialing scripts accordingly. Handy little tidbit to know.

Well, that’s it. My new machine is now set up just like I want it, complete with VMware and winders XP running in a window. I’m composing this post using BlogDesk, a windows program that is running in the Windows window. Can I type window any more times?

This ends the more or less irregular Linux Chronicles. My machine is now like I want it so I’m going to proceed to use it. I’ll post future Linux Chronicles when something new happens but they’ll be few and far between. I don’t intend to diddle with this machine.

Now that I’ve had Linux up and running for a couple of months, my major thought is, “why did I wait so long to do this?” This is a wonderful computing environment. I have thrown off the microsoft shackles and won’t be paying any more winders taxes. Yippeeeeee!!!!

Posted by neonjohn on February 27th, 2009 under Computing |
  Comment now »

:::del.icio.us :Digg it :StumbleUpon :Yahoo MyWeb ::

  The Linux Chronicles – Virtual Machines

PermaLink

Screenshot 1

Imagine being able to run Linux programs and Windows at the same time on the same machine. No, I’m not talking about Wine, the Windows emulator. I’m talking about running genuine Windows XP as an application under Ubuntu.

Well, Imagine no more. If you look at the screen capture to the left you’ll see the Ubuntu desktop. You’ll see a window with XP running in it and Street Atlas running under XP. You’ll see the XP window, called “winders” on the Ubuntu task bar circled in Blue. At the same time, Forte Agent mail agent is running under Wine, its tab being circled in Red.

Amazing, no? All that is brought to you by the power of VmWare. (that web page is horrible so unless you just like to waste bandwidth, don’t bother.) VMware is an executive that runs under a host system (Ubuntu in this case) and creates 1 or more virtual machines (VMware calls them appliances) that other operating systems can run under. One could have multiple copies of windows, windows and a different distro of Linux or even multiple copies of Ubuntu running under Ubuntu. Dizzying, isn’t it?

A virtual machine gets its own memory and disk drive space and all the other resources an OS normally expects. The XP that is running in the above screen capture was installed from the XP disk that came with my computer. In other words, it’s the real thing and not an emulation.

Screenshot 2

This is a blow-up of the lower right corner of the VMware window, the window that contains XP. Each of the little icons is a resource. A disk drive, a CD drive, a scanner and so on. Each resource can be turned off or on at will. If I turn on a disk drive, windows responds with a “found new hardware” bubble and installs that hardware. IOW, it doesn’t know the difference between a real and a virtual resource.

Another exciting feature is the ability to drag and drop things between the XP window and Ubuntu. In fact, that’s how these photos were brought to this post. I write my blog entries using a program called BlogDesk. It is a windows program that will not run under Wine. It runs just fine in a virtual machine running XP. I did the screen captures using Ubuntu’s screen capture utility and the dragged the images from the Ubuntu desktop to the XP desktop. Slick.

VMware

Vmware is in the business of selling very high dollar products to big corporations but they have a few things for us little guys and most are free.  If you follow this guide, you can set up a virtual machine for free.

There are several concepts that are critical to understanding how this works:

Things to Download

To get started, you’ll need to download some things.  We’re going to create a VM using Server so you’ll need the following:

I recommend dropping these on the Ubuntu desktop.

Installing Server

Server has to be installed from a command prompt because it has to be installed with root priviledges.  Open a terminal window and do the following.

$ cd ~/Desktop
$ tar -xzf  VMware-server-2.0.0-122956.i386.tar.gz

This leaves a directory named vmware-server-distrib on your desktop.  Execute the following:

$ cd ~/Desktop/vmware-server-distrib
$ sudo ./vmware-install.pl

This starts the install.  You will be asked a series of questions.  The default answer works for each question – just hit <enter>.  When it asks about networking, choose both “bridged” and “NAT”.

When the install finishes, you can delete the above installation files.

The server is controlled through a web interface.  To connect to the server, go to your web browser and type the following:

http://localhost:8800
http://localhost:8222 (sorry ’bout that)

Port 8222 assumes you chose the default option during the install.

Open the installation manual and go to the chapter on installing a virtual machine on Linux.  Follow those step-by-step instructions to create a virtual machine appliance.

At some point you’ll be asked to put your windows distribution disk in a CD drive.  This must be a bootable CD drive because the windows distribution disk will be booted into the virtual machine and the installation commenced.

When the procedure asks for a virtual drive size, choose a large one, say, 20 gigabytes.  This does not occupy 20 gigabytes on your hard drive.  That simply sets the largest size permissible in this VM.  The size can be changed later but it’s a pain, involving a partition manager and stuff.  The virtual drive file grows as data is placed in it.  It can also be shrunk using the VMware tools.

After the windows install is finished, launch the virtual machine by “turning it on”.  Windows will boot in a new window and do all the preliminary stuff that a new install does.  Choose the option from the web interface to install the VMware tools.  These tools are vital and must be installed from the web interface.  Once all of that is done, shut down the windows VM (just like you would a regular windows machine) and exit the browser interface.  We are now finished with Server.

Server will let you run a VM but it will not allow you to access most of your machine’s hardware resources.  Therefore after we used Server to build the virtual machine, we will install and use Player to actually run it.  Player allows almost no changes to be made to the virtual machine but it does enable all the resources.

To install player, assuming that the terminal is still open and that the player file is on the desktop, do the following:

$ cd ~/Desktop
$ sudo ./VMware-Player-2.5.1-126130.i386.bundle

Let this installation run its course.  In the process it de-installs Server. When it finishes, a command called “vmplayer is installed in /usr/bin.  This is the player.

The windows VM is installed in

/var/lib/vmware/Virtual Machines/<windows VM name>

I called my windows VM “winders” so my machine is located in

/var/lib/vmware/Virtual Machines/winders

The actual machine is

/var/lib/vmware/Virtual Machines/winders/winders.vmx

The virtual machine was installed by server owned by root.  The machine has to be owned by your account for Player to run it so do the following, substituting your userid and VM name.

$ cd “/var/lib/vmware/Virtual Machines/winders”
$ sudo chown jgd *

Substitute your userid for “jgd” and your VM name for “winders”.  Be sure to use the quotes as indicated because of the space in the directory name.

To start a windows session I would type

$ vmplayer “/var/lib/vmware/Virtual Machines/winders/winders.vmx”

The quote is necessary because of the space in the Virtual Machines directory name.  Now this is a lot of typing so what I did was create a desktop icon containing the above command.  To do that, right-click somewhere on your Gnome desktop and select “create new launcher”.  Give it a name and put the above command (substituting your VM name, of course) in the “command” slot.  Close the window and a new icon will appear on your desktop.  That’s all there is to it.

Wrap Up

You’ve now finished installing the VM.  You can launch windows by double clicking on the icon you just created.  There are a few things that you need to know that the manual doesn’t explain.

Any resource allocated to the VM (by enabling it using the lower right icons or the menu at the top left) is not available to the Linux system. Thus, if you allocate your printer to the VM, you cannot print in Ubuntu until you either un-allocate the printer or shut down the VM.  As shown in the second screen capture above, I leave everything unallocated until I need it.

When you allocate a resource for the first time, windows discovers it as new hardware and runs the plug’nplay for that device.  If the device needs a supplied driver on disk (my scanner and laser printer, for example), have the disk ready and placed in the CD drive.  The CD drive must be allocated, of course.

When you are ready to de-allocate a disk drive you must first safely remove it from windows using the icon on the tray.  This is just like on a windows machine.  If you don’t “safely remove” the drive, when you un-allocate it, it will be left “dirty” and will not mount on your Linux system until you use windows to fdisk it.  As far as windows knows, it is running natively on actual hardware so you have to do everything just like you did when windows was installed on your machine.

Closing the windows window puts windows into hibernate mode.  When you click the desktop icon again, windows resumes where it left off.  This is a good thing except that you must not shut down Linux with windows hibernating.  If you do, windows will start in an unknown state and will probably BSOD.

When windows BSODs (it does that as a virtual machine just like it does on a real one), do not just close the blue screen window.  That will hibernate the BSOD.  When you restart windows using the icon, it will go through the process of starting up, only to BSOD again.  Instead of closing the window, use the Player dropdown menu at the upper left to “shutdown and exit”.  This is like flipping the power switch on a real machine and allows windows to start from scratch on the next startup.

That’s about it.  You can now install your windows applications just as you would on a real machine.  The VM runs windows about 2/3 normal speed as compared to native windows.  That’s acceptable to me.  Especially since I don’t run windows all that much.  I try to do everything I can in Linux native mode, saving the windows VM for only those apps that won’t run under Wine and for which I’ve found no Linux substitution.

Enjoy

Posted by neonjohn on February 17th, 2009 under Computing |
  3 Comments »

:::del.icio.us :Digg it :StumbleUpon :Yahoo MyWeb ::

  The Linux Chronicles – Installing a New Hard Drive

PermaLink

There comes a time in any computer’s life when a larger hard drive is necessary.  In my case, the time came quickly, as I had installed Linux on an old 40 gig drive just to see if I liked it.  I obviously did, hence this series, so a larger drive quickly became a necessity.

I procured a 120 gig drive and started doing my homework to see what was involved in moving to a larger drive.  In the good old days, all one had to do was make the drive bootable (write the boot sector and master boot record) to the drive and then copy everything over using cp or dd or tar.

Alas, things have changed.  In an effort to make drives portable, the Linux developers invented something called the UUID (Unique Unit ID, I think).  It assigns a large random ID number to each drive.  A new drive gets assigned a new UUID when it is made bootable.  The copied-over boot loader (GRUB) is still expecting the old UUID and therefore cannot find the new drive.  A hung boot is the result.  Here’s an article that discusses UUIDs in far more details than you probably want to know about.

If you want to see what the uuid of your drive is, execute the following command from the command prompt (open a terminal window to get the prompt:

$ sudo vol_id -u <device>

where <device> (without the brackets) is the name of the drive.  In my case, my boot drive is /dev/sda1 so the command is

$ sudo vol_id -u /dev/sda1

which produces

1714baeb-364f-4af1-b52c-8e6928410d5c

Nice friendly and easy-to-remember number, isn’t it?  NOT!

Now one can go hacking around in /etc and /boot and fix up the UUID of a new drive but it is easier and about as fast to simply clone the main partition as I will outline below.

The first thing you need to know is the name of your drive.  To do that, execute the command “mount”

$ mount

It will return something like below.

/dev/sda1 on / type ext3 (rw,relatime,errors=remount-ro)
proc on /proc type proc (rw,noexec,nosuid,nodev)
/sys on /sys type sysfs (rw,noexec,nosuid,nodev)
varrun on /var/run type tmpfs (rw,noexec,nosuid,nodev,mode=0755)
varlock on /var/lock type tmpfs (rw,noexec,nosuid,nodev,mode=1777)
udev on /dev type tmpfs (rw,mode=0755)
devshm on /dev/shm type tmpfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
lrm on /lib/modules/2.6.24-16-generic/volatile type tmpfs (rw)
securityfs on /sys/kernel/security type securityfs (rw)
none on /proc/fs/vmblock/mountPoint type vmblock (rw)
binfmt_misc on /proc/sys/fs/binfmt_misc type binfmt_misc (rw,noexec,nosuid,nodev)
gvfs-fuse-daemon on /home/jgd/.gvfs type fuse.gvfs-fuse-daemon (rw,nosuid,nodev,user=jgd)

All you’re interested in is the root drive which is shown in the first line (bold faced) above.  Ignore everything else.  In my case on this laptop the drive is /dev/sda and the first partition, the main one is /dev/sda1.  This is the partition that we want to clone.  Write down whatever “mount” returns for your system and substitute it for “/dev/sda1″ in the procedure that follows.

We’ll use a couple of programs to assist us in this endeavor, both of which either are on the distribution disk (distro) or in the repository that can be accessed with the Synaptics package manager (system->administration->synaptics package manager).  The programs are:

Partimage – copies a partition to another drive and stores it in a single file.  It also restores that partition when commanded

GParted – the GNU partition editor.  This utility enables us to resize partitions on the fly.  We’ll be using the one that came on the distro disk (system->administration->Partition Editor).  It is important to use this one, as other versions such as the one in the repository won’t work.

Note: you do not need to download from the above links. I included them for reference only.

In the procedure below we’ll be working with a laptop.  If yours is a desktop machine, simply substitute your first drive for “drive bay”.

You’ll need something to mount the old drive on.  With my laptop, that was a USB adapter.  If you have multiple drive bays in your desktop, simply substitute another drive bay for “USB adapter”.  Even with a desktop machine, however, a USB adapter is very handy to use since you don’t have to open up your machine or shut it down to connect or disconnect drives.

Without further ado, here is the step by step procedure.

  1. Place the new 120 gig drive in the laptop bay, put the distro disc in the CD drive and boot from the distro disk.  Boot to the desktop and choose “install”
  2. Let the install run its course.  Shut down the system.  A virgin Ubuntu system is now installed on the new drive.
  3. Install the old drive on a USB adapter.  Boot the new drive to the virgin Ubuntu system.  Make sure the distro disc is removed from the CD.
  4. Go to Synaptics and fetch partimage.  This is a program that copies an exact copy of a partition.
  5. The old drive on the USB adapter is /dev/sdb.  The first partition is /dev/sdb1. (note: substitute whatever location your old drive is in for /dev/sdb using the “mount” command to discover it)
  6. Get a command prompt and start up PartImage.
    $ partimage

    Select /dev/sdb1.  Select a destination on another hard drive large enough to hold the image.  I chose the 400 gig drive (one of my USB drives that is always connected to my system).  Follow the menus and start the clone.  Go away for 3 hours or so, depending on how full the old drive is.  When the program finishes, there will be an image of the old drive’s partition in a file on the 400gig drive.  Shut down the system

  7. Place the new drive on the USB adapter and the old drive back in the laptop’s drive bay.  Boot.  Run PartImage again.  Select /dev/sdb1 and “restore” mode.  Write the image to the new drive.  This takes about an hour and a half.  Shut the system down.
  8. Install the new drive in the drive bay and lay the old drive aside.  Put the distro CD in the drive and boot from it.
  9. At this point the new drive has the old drive’s data on it but the partition size is that of the old drive.  In my case the old drive was 40 gig and the new 120.  The new drive’s first partition still looks like a 40 gig partition. We must expand it.
  10. With the distro CD booted, select system->administration->partition editor.  The drive to change is on /dev/sda and the partition is the first one, /dev/sda1.  Note:  This program takes about 3 minutes to start during which time it shows no indication of life.  You’ll think that the program has hung.  It hasn’t.
  11. Select the first partition (/dev/sda1 and choose “resize”.  It may indicate that it is already the same size as the drive.  Don’t believe it.  If the partition is still small, change it to the maximum available.  If the partition indicates that it is already large, change it by a tiny amount, say, 1Mb, just enough to enable the “resize” button.  Hit “apply”.  This process takes about 5 minutes.
  12. Shutdown.  Remove the distro CD and boot from the new drive.  Check the new size with the “df -h” command.  It should show the full drive size.  Congrats.  You’ve just installed your new drive.

See, that wasn’t difficult – much easier than futzing around in windows.

Posted by neonjohn on February 17th, 2009 under Computing |
  3 Comments »

:::del.icio.us :Digg it :StumbleUpon :Yahoo MyWeb ::
Google Reader or Homepage Add to My Yahoo! Subscribe with Bloglines Add to Technorati Favorites!



Meta: