Saturday, October 18, 2014

Notes on OSX Virtalization, booting raw disks

I picked up a Mac to use for work, primarily, a couple weeks ago.  It's a late 2013 Macbook Pro Retina 13" w/ 8GB ram and 256GB SSD.  I wanted to be able to dual boot with Linux for home and dev type stuff, but being short on time needing to encrypt my primary hdd, I didn't get to figure out boot camp before finalizing my setup for work, so I'm not about to mess with my primary disk's partition/efi config and risk downtime at work.

Sooo, I'm trying to get set up to boot to boot Linux on an SD card.  This has been a bumpy road.  I purchased a Nifty SD card adapter and a Sandisk 80MBps 64G card to install to, and I'm doing the install via a VM while I work on other stuff.  This presents first problem - support for booting SD card by Parallels, Fusion, or VirtualBox.

I'm defaulting to Parallels since everyone else on mac uses it at work, but Fusion looks pretty comparable.  I've used VirtualBox a lot in the past, but I got tired of bugs and crashing at some point and switched to VMWare workstation at home.

That said, VirtualBox seems to be the most flexible and capable when it comes to booting SD cards on Mac.  It is the only virtualization solution of the three that doesn't have  "dumbed down" interface for people who expect things to just work without understanding them on mac.  It comes with command line tools for creating virtual disk files, etc.  This is what you need if you want to make a "bootcamp" disk image to use with Fusion.

Parallels and Fusion only support booting raw disks via "bootcamp".  You can't just point them to a block device and have them use it without tricking them.

  • To trick Fusion into booting to the SSD, set up a linux VM with a virtual disk that you'll replace.  Then use virtualbox command line tools to provision a raw disk vmdk and copy this over in place of the virtual disk that Fusion created.   That's it. It should boot fine. 
  • To trick Parallels, and note that I haven't found a way to do a raw disk vmdk like with Fusion, create a VM without a disk and enable efi.  You may want to remove Network from the boot order.  In the VM settings menu, attach the SD card to the VM.  Power up your VM and wait for it to drop you to the efi shell.  Then run:
    • fs0:
    • cd EFI\ubuntu
    • grubx64.efi
Getting the macbook to boot the SSD is another matter, and it's where I'll look at the actual Ubuntu build.  

Macbook Pro Dualboot Ubuntu SD Card & Virtual in OSX

This is just a quick post outlining how I finally was able to get ubuntu installed on an SD card that would boot from the option menu when I power on the Macbook and would also boot in either Fusion or Parallels in OSX.

For reference I'm on a Q4 2013 Macbook Pro 13" Retina with a SanDisk Extreme Pro 64GB SD card.  Not the fastest, but nearly the fastest.  I finally had success using VMWare Fusion Pro 7.0.  This may work with Parallels, which seems to have better EFI support, but has not been very stable for me.  This solution would not boot in Fusion 6.x.

  • Install the SD card in the laptop.
  • Use VirtualBox to create a .vmdk disk file that points to the SD card (/dev/disk2, in my case).  This effectively creates a "bootcamp" disk image to use with Parallels or Fusion. 
  • Download the rEFInd (the more recent version of rEFIt) usb stick disk image and dump it to the SD card:
    • Disk Manager to see what the device name is for the SD card.  It was /dev/disk2 for me. 
    • Use mount to see which partitions of the SD card are mounted:
      • mount | grep /dev/disk2
      • blah blah /dev/disk2s1...
    • Use diskutil to unmount:
      • diskutil unmount /dev/disk2s1
      • ...
    • Write the rEFInd image to the card:
      • dd if=rEFInd_blah.img of=/dev/disk2
    • Use diskutil info to take note of the name of the disk2s1 partition name. Something like ElTorito. 
  • Download your choice of linux distro.
  • Create a VM in VMWare Fusion and:
    • Attach a 20GB IDE hdd to it.
    • Attach the Linux install CD and set it as default boot. 
    • In the VM config directory, remove the 20G disk image vmdk files and rename the .vmdk that was created in VirtualBox to match. 
    • Add firmware="efi" to the vmx config file for the VM. 
  • Boot the new VM and when it loads into the Linux Live CD:
    • Dump /dev/disk2s1 (this is the 4.5MB partition that was created when we dumped the rEFInd image file to the card) to a tmp file:
      • dd if=/dev/sda1 of=./sda1.dd
    • Use gdisk (the GPT equivelant of fdisk) to:
      • Take note of the starting and ending sectors  and the partition type of the refind partition (sda1)
      • Wipe the GPT table (under expert optoins somewhere) and then re-open gdisk.
      • Create a new partition of for rEFInd using the same starting and ending sectors we used earlier and the same partitoin type. 
      • Set the name of this partition to the same as it was before (ElTorito?)
      • Create a linux partition using the remaining space.  You may want to leave some buffer space after the first partition in case you ever need to update refind and don't want to have to resize or rebuild the linux partition. 
      • save and exit gdisk
    • Restore our image of sda1 to the new partition we created along with the new GPT partition table:
      • dd if=./sda1.dd of=/dev/sda1
  • Reboot the VM with SD card as default and verify that it comes up in the rEFInd boot menu. 
  • Reboot the VM and boot to the CDROM again and start the install. 
    • I'd recommend using ext2 filesystem, since it doesn't use journaling and is fast, and no swap, to conserve writes to the SD card.  
    • Tell Ubuntu (or your distro) to install the bootloader to sda2!!! this is critical, so it doesn't bork your rEFInd bootloader.  
  • When the Linux install completes, reboot the VM to SD card and verify that rEFInd loads and that, from there, you can select the linux image and boot to it.  
  • if that works, then reboot the laptop and hold "option" to get the boot menu.  You should see the SD card and be able to boot Linux via rEFInd!
So Simple!    You don't want to know how many days of tinkering and repeated linux installs it took to find a setup that would work like this.  Fack!

Trying to boot to CD and install ubuntu directly to the SD card never worked for me, as the MAC bootloader wouldn't see the installation and present it in the boot menu when the laptop powers up, despite Ubuntu installing EFI boot junk on the SD card.  It wasn't too difficult to get the SD card to boot in EFI or legacy mode inside of a VM once figuring out how to do the bootcamp vmdk, which raises another important thing to note:

Fusion and Parallels are playthings only.  They really have no advanced tools included via command line or the standard UI.  There's no way to "bootcamp" to an SD card with them without hand-editing their config files and using 3rd party tools to create the disk image files.  VirtualBox is indispensable when manipulating VM images.  It comes with a bunch of great command line utilities for creating and managing disk images, and who knows what else.  I don't use it normally, though, because I used to run into issues with stability when I used it in Linux.  Meh... It worked great before Oracle bought it.  :P

Update:  Just saw the following, about creating raw disk iamges with vmware tools... might be helpful:
http://techrem.blogspot.com/2012/12/add-physical-disk-to-vmware-fusion.html

And this, which talks about using "tmpfs" to redirect IO-intensive filesystem activity to ramdisks and cut down on SD card activity:  http://www.zdnet.com/raspberry-pi-extending-the-life-of-the-sd-card-7000025556/
He suggests:

tmpfs    /tmp    tmpfs    defaults,noatime,nosuid,size=100m    0 0
tmpfs    /var/tmp    tmpfs    defaults,noatime,nosuid,size=30m    0 0
tmpfs    /var/log    tmpfs    defaults,noatime,nosuid,mode=0755,size=100m    0 0
tmpfs    /var/run    tmpfs    defaults,noatime,nosuid,mode=0755,size=2m    0 0
tmpfs    /var/spool/mqueue    tmpfs    defaults,noatime,nosuid,mode=0700,gid=12,size=30m    0 0

Might be useful... but my system wouldn't boot with all of these enabled... will have to actually think about this and do some testing later.

Tuesday, February 04, 2014

Why I love bash - simple remote connections

Bash is my go-to language for busting out sysadmin stuff.  I've been ramping up with python for a while now, and for more complicated tasks, it's fantastic;  it should probably even be fantastic for simple tasks.  But, there is one thing that I do all of the time in Bash that I have yet to find a way to replicate without unnecessary complexity in python:

#!/bin/bash
agregateResults (); { while read line; do echo "Program Output: $line"; done; }
clientCode (); { echo "I am client $(hostname)."; }
if [[ "${1}" == "client" ]]; then
    clienCode
else
    for server in  ${*}; do
        cat ${0} | ssh ${server} -s client
    done | agregateResults
fi

This is so freaking useful.  I have servers across multiple domains and I need to run script to aggregate data from all of them, but I can't always count on network file systems being available, etc.  This lets me have a script in once place and execute the same code everywhere without fumbling with copying it ahead of time, etc.   It's so clean and easy.

When I try to find out how to do this with Python, I read all kinds of stuff about parallels, Paramiko, blah blah blah... it's all so complicated.  There has to be a five-liner example out there somewhere of how to do the same kind of thing.  

Started a code-golf at stackexchange for this one: http://codegolf.stackexchange.com/questions/20008/self-execute-over-ssh

Monday, November 25, 2013

Commentary on the "Enders Game" movie

The producer tried to cram way too much stuff into 100 minutes of film.  There was no character development, huge elements of the story were omitted or glossed over, and who the heck knows what was going on the entire time...  That was a train-wreck.

I suspect that the film was going to be way over budget if they tried to create the battle-room scenes with squads of kids flying all over.  I don't believe that it's possible to create realistic zero-g interactions between mobs of people using wires...  The whole scenes would need to be animated, at huge cost.

I hope that, someday, this movie is re-done as a 3 part anime series, allowing more like 6 hrs to tell the whole story.  An animated format would let the characters be presented at proper ages, allow for way better zero-g fighting and battle room scenes, and so much of the story could be left true to the book.

Wednesday, October 23, 2013

Google Drive Cleint for Linux

It's been like two years now, and there is still no Google Drive client for Linux.  What gives, Google?

A lot of us Linux users use your services exhaustively, but we have to fall back to 3rd party tools to sync files into Google Drive.

Despite all of the privacy controversy, I would LOVE to have a central place where I can sync back (and edit) some of my scripts, application config files, etc.  I use gdrive for everything frikin' else.  It feels even less security having to grant some 3rd party middle-man access to my account so I can sync from my computers.... Argh...

Thursday, January 17, 2013

vCenter 5.1 is a pain!

I'm wishing I could just use 5.0, but I need some of the 5.1 functionality for nested virtual servers...

After running the Simple Setup, I get an error trying to log into the web client:

Failed to connect to VMware Lookup Service
https://:744/lookupservice/sdk - SSL certificate verification failed. 

Rebooting and waiting ten minutes for the service to start up fixes this.

After successfully logging into the web client with admin@System-Domain account, it doesn't list any available vCenter servers.  It appears that the vCenter service never registered with the Lookup Service, so this has to be done manually, per the doc at http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2033620.  See the "Registering vCenter Server with Inventory Service" section.

cd C:\Program Files\VMware\Infrastructure\VirtualCenter Server\isregtool

register-is.bat https://machinename.corp.com:443/sdk https://machinename.corp.com:10443 https://machinename.corp.com:7444/lookupservice/sdk


This seemed to work well.  Loads of text scrolling by during the register.  After connecing back up to teh Web Client, There was a message showing the following (not exactly, I didn't capture this verbatem):

Unable to connect to one or more vCenter Servers:
https://:xxxx

So I'm pretty back to where I was before trying to register the vCenter server with Inventory Manger.

Saturday, November 10, 2012

Linux Swap Create Init Script

I use an SSD as my primary drive.  I used to have a swap partition on it (I know, bad idea on SSD) and it had probs w/ data corruption and the swap randomly being disabled.  Linux supports trim for SSDs, but trim won't do anything for swap that's always on a specific set of blocks in the SSD...

If we need swap space, but don't want it to always hit the same part of the SSD, we should be able to use a swap file in /tmp, allocating 2GB at boot. At least in Ubuntu, /tmp is cleaned up each boot, so when we create our swap file, trim has the opportunity to put the file on some fresh SSD sectors.

Here's the init script for this:


#!/bin/sh
#
# setup swap file init script
#

### BEGIN INIT INFO
# Provides:          atop
# Required-Start:    $syslog
# Required-Stop:     $syslog
# Should-Start:      $local_fs
# Should-Stop:       $local_fs
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# Short-Description: Monitor for system resources and process activity
# Description:       Atop is an ASCII full-screen performance monitor,
#                    similar to the top command, but atop only shows
#                    the active system-resources and processes, and
#                    only shows the deviations since the previous
#                    interval.
### END INIT INFO

PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
SWAPFILE=/tmp/swapfile
NAME=init_swap.sh
DESC="tmp swapfile creator"

case "$1" in
        start)
                echo -n "Starting $DESC: "
                if $0 status >/dev/null
                then
                        echo "    Already Running."
                        exit 0
                fi

                test -f ${SWAPFILE} && rm ${SWAPFILE}
                dd if=/dev/zero of=${SWAPFILE} bs=1M count=2048
                mkswap ${SWAPFILE}
                swapon ${SWAPFILE}
                echo "        Done."
                ;;
        stop)
                echo -n "Stopping $DESC: "
                $0 status >/dev/null && swapoff ${SWAPFILE}
                test -f ${SWAPFILE} && rm ${SWAPFILE}
                echo "        Done."
                exit 0
                ;;
        status)
                if swapon -s | grep -q ^/tmp/swapfile
                then
                        echo "Swap is enabled"
                        exit 0
                else
                        echo "Swap is not enabled"
                        exit 1
                fi
                ;;
        *)
                N=/etc/init.d/$NAME
                echo "Usage: $N {start|status}" >&2
                exit 1
                ;;
esac

exit 0