Category Archives: Coding

Coding, Science

Installing Ubuntu 12.04 LTS on a Dell Windows 8 machine with UEFI

Leave a comment

Quick post this – originally on evolve.sbcs but behind a wall there so reposted here – both in case anyone’s interested and in case something breaks and we have to remember what we did, exactly (the far more likely scenario)

Ubuntu. Everyone loves it – probably the world’s easiest dual-boot install going and you get a lot of features out of the box as well. So, for the Windows user who needs Linux features / reliability / configurations but wants to keep a copy of Windows on their system for Office or just build testing, Ubuntu is a great solution.

Well, at least, it used to be. Then Microsoft went and released Windows 8. Quite apart from being an even worse resources hog than W7, it also introduced the horror that is UEFI and a whole load of hell was unleashed. That’s because UEFI – a seemingly harmless set of ‘improvements’ to bootloader management (the bit of firmware on your motherboard which loccates the OS, introduces it to RAM and disk controllers, and generally makes everything you see happen) – is actually a right pain in the arse. Typically for Microsoft, UEFI isn’t just overloaded with features which dilute it’s core purpose: it actually introduces unpredictable behaviour which makes life for anyone nosing beneath the surface of the operating system (Windows) the machine comes with (not just on those trying to dual boot – although I suspect they were the main targets, as millions of people trying Ubuntu for free and realising what an unstable rip-off Windows 8 is wouldn’t play well with shareholders…)

Which is all a ranty way to say that the lovable Ubuntu install process I’ve used for pretty much a decade on Windows 7, XP, Vista and even 98 machines – all of which happily budge up a bit to make room for a dual-boot Ubuntu installation – has been well and truly borked by Windows 8 and their crappy UEFI lock-in. In particular, Hernani, one of our new students, has been issued a shiny new Dell (good for him) with Windows 8 loaded via UEFI (not so good). The install process was markedly more complicated than other Windows / BIOS versions, and while this is a common problem, I couldn’t find a single tutorial online able to help our specific problem – so here you go (PS: I / the lab / QMUL disclaim all liability, express or implied, should you attempt the steps below and suffer…):

  • Create an Ubuntu Live / bootable install USB
  • Shrink your Windows partition
  • Back up your files and make a Windows Restore CD / USB
  • Access the UEFI bootloader
  • Install Ubuntu
  • Re-activate UEFI
  • Check GRUB
  • Install boot-repair
  • Check GRUB
  • Keep backing up your files!

Create an Ubuntu Live / bootable install USB (or CD)

First we need to create a CD or a USB which we’ll use to install Ubuntu (and/or run a ‘live’ session, where you run the OS directly off the disk without installing to the hard drive. This lets you test Ubuntu and verify it works on your machine). This is a very well covered topic, not least from Ubuntu.com themselves, but I’ll just note that we installed Ubuntu 12.04 LTS (long-term stable), 64-bit release onto a USB – the machine in question having no optical drive. We used Unetbootin for this.

Shrink your Windows partition to create space on the hard drive for Ubuntu

To install Ubuntu we’ll need at least 100Gb free disk space. How much of the available space you want to give to Ubuntu is up to you, and probably depends on whether you plan to use your machine mainly in Windows 8 or Ubuntu; Ubuntu can see the Windows partitions, but not the other way round. To do this we need to shrink the existing Windows partition and repartition the existing space. This subject is well covered in this guide (which we started with) but the subsequent steps gave us more trouble (hence this post)…

Back up your files and make a Windows Restore CD / USB

There is a fairly high chance that this will result in a fried filesystem, and you may not even be able to get Windows to boot. You might also lose data if you screw up the disk partitions. So back up all your files. This is very important – not doing so practically guarantees the jinns of Murphy will screw up your hard drive for you. I don’t care how you do this, just do it.

Equally, since we might bork Windows well-and-truly (possibly a good thing but it would be nice to do so on purpose rather than accidentally) it makes sense to burn a Windows Recovery CD (or USB) at the very least. Again, you can do this easily and there’s lots of tutorials out there, so I won’t deal with it here, except to say you’ll need this to be on a separate USB/CD to the Ubuntu one we just made.

Access the UEFI layer and switch to legacy BIOS

The operating system itself (Windows, OSX, Linus etc) doesn’t actually run straight away when you start the computer. This is, in fact, why you can install / upgrade / change / dual-boot various different OSes. Instead, a small piece of software hardcoded onto your motherboard which runs when you hit the power button to start the machine. This is the black screen you glimpse as the machine reboots. Most commonly this software (‘firmware’ in fact, since it is rarely changed or updated) is called BIOS, but a new standard called EFI has become more common in recent years (Windows’ own flavour is the hellish UEFI). It’s main job is to find all the various device drivers (monitor, keyboard, hard disks, USB ports), introduce them to the CPU and RAM, and then hand the whole lot over to an operating system on a disk somewhere to make the good stuff happen. Note that I said ‘an‘ operating system, located somewhere on ‘a’ disc – there’s no reason why Windows should be picked, and why the disc should be the hard drive, and not… say… an Ubuntu install USB! 🙂

So, to access this magical BIOS-y, UEFI-y world where we can tinker about with the default OS we need to shut the computer down, then restart it, and in most personal computers until now hitting a setup key on restart (usually something like F2 or F12) repeatedly would stop the BIOS/UEFI from loading and access a special setup screen. Unfortunately, Windows 8 has a ‘feature’ (read: ‘restriction’) built in called Secure Boot which normally prevents you from doing this insanely simple procedure. Instead there’s a vastly more complicated process which is luckily well explained in this SO thread. Scroll down to ‘Windows 8 + Ubuntu‘ to do this.

Once you have rebooted into BIOS/UEFI, the version that shipped with this Dell (an Opteron 9020 ultra-compact) has a screen with an ‘advanced boot/UEFI options’ submenu and a ‘secure boot’ submenu. Enter the secure boot menu, disable secure boot and click ‘apply’ to save changes then enter the UEFI screen. The most important option on this screen selects between bootloader modes: ‘UEFI’ and ‘legacy’ (which means BIOS). We’ll need the legacy / BIOS mode to install Ubuntu from USB, so select this. The previous set of UEFI options will be replaced by a set of devices. This is the main difference from our point of view – in BIOS mode the computer will try each device for a bootloader file, in the order in which they appear, and the first one found will be run. EFI, on the other hand, is able to look non-sequentially on a device (e.g. a disc) for a specific bootloader. However: we just want to get on with our Ubuntu installation, so make sure the device with your Ubuntu installer (USB or CD) is listed first, click ‘apply’ then ‘exit’ (making sure said media is actually inserted!)

Install Ubuntu

The process to install Ubuntu itself is actually simple enough, and covered in truly exhaustive (some might say ‘tedious’) detail elsewhere – but there’s a couple of installation points we need to note at this stage for our specific application. Again, if you’re doing a complete wipe of Windows you can just to a complete erase and you’ll be fine:

  • Select ‘something else’  in the main install options – do not erase Windows if you want to dual-boot.
  • There’s probably not much point in installing the updates and third-party stuff at this stage – I had to reinstall Ubuntu a few times by trial and error, and the extra packages take a while to download, unpack, and install. Anyway, you can get them later on when the installation’s debugged.
  • The order and location of your partitions matters, a lot. This process is covered in more detail in this SO thread, but to summarise (you should read the whole SO post though):
    • You’ll need 50Gb (50000Mb) for the Ubuntu OS itself, formatted as ext4 and mounted as root ‘/’. On our Dell this is /dev/sda7
    • You’ll need 16Gb (15999Mb) for swap space (Windows users will know this as ‘virtual RAM’. This is technically optional but running without it is very likely to crash frequently from RAM shortages.
    • Don’t touch the existing (Windows) partitions
    • You’ll want to partition the remaining disk space as ext4 (probably) and mount it at ‘/home’
  • Make a note of the password you select for Ubuntu! Otherwise you’ll have to reinstall…
  • Finally – and this is important – although other guides say the bootloader should be written to: 
    /dev/sda

    we found this didn’t work with our Dell UEFI. Instead we had to install the bootloader to:

    /dev/sda1

    which worked fine.

Re-activate UEFI

The install process complete, we now need to switch the UEFI back on; this will continue to be the main way you call the bootloader from now on, but hopefully we’ll be using GRUB (a linux bootloader) instead of the Windows bootloader. GRUB allows you to pick which OS to boot every time you restart the computer (it defaults to Ubuntu, but you can choose Windows 8 if you want, it won’t care!) and should be configured automatically. However, we’ll need to turn UEFI back on first: shut the computer down. Wait 5-10 seconds. Now restart it, hitting F2 (or other BIOS / UEFI setup menu hotkey) as soon as you hit the power button to access the UEFI / BIOS screen again.

From the BIOS / UEFI, find the advanced boot options, deselect Legacy mode (BIOS) and reselect UEFI. This will put us in UEFI mode again, so we have to configure the UEFI options to choose the GRUB bootloader. You should see that an ‘ubuntu’ option has appeared in the UEFI list as well as the Windows boot manager. Select this, then click ‘view’ to see which bootloader file it points to.

In our installation, ‘ubuntu’ pointed to (filesystem FS0): /EFI/boot/shimx64.efi. This will simply load ubuntu directly. If you only plan to use Ubuntu, you can stop here, as this option will find Ubuntu every time you reboot. However, we wanted to use GRUB to pick Ubuntu or Windows, so there’s an additional step here: Click ‘add boot option’ (or ‘path’, or similar) to create a new UEFI boot option. Give it a name (we went for ‘GRUB’, logically enough). Then we need to pick the bootloader file itself – this is the file the UEFI will try and find, which will then handle the OS selection and initialisation. In our case, the file we were looking for was (again on filesystem ‘FS0’): /EFI/boot/grubx64.efi. This is the file to load GRUB, but while you’re poking about in here you may also see a directory for Windows and the bootloaders (there’s a few) associated with it. Select this file, save the UEFI option and make sure this is the first boot option in the list (use arrows to move it up) and/or uncheck (don’t delete…) the other options (‘Windows boot manager’ and ‘ubuntu’ probably).

Click ‘apply’ then ‘exit’ and the machine will reboot…

 

Check GRUB

If we were installing next to Windows 7, Vista, XP, NT, 2000, 98 – or almost any other OS – we’d be able to have a cup of tea and a celebratory hobnob at this point, as after rebooting the machine the friendly GRUB boot selection screen would pop up at this point and let us select Ubuntu, Ubuntu recovery mode, memtest or indeed Windows with a simple dialog. We found that GRUB loaded OK, but although all the linux-y options came up, it couldn’t show the Windows boot option. Damn you, Windows 8 – in W7 this is no problem at all. We also wouldn’t have had to prat about with UEFI settings, either but hey, Windows 8 is shiny and looks a bit like iOS, so it must be better, right…

A fair few people have actually managed to get GRUB to function correctly at this stage on W8, but then they’re not reading this blog, are they? 😉

Install boot-repair

If the GRUB bootloader works (that is, you see the GRUB selection screen) but the Windows installation you irrationally cherish so much isn’t shown, you’ll need to edit the GRUB config files to include the location of the Windows bootloader files so that GRUB can display them as an option for you. You can sleuth about the hard drive to find both the GRUB config and Windows .efi files but luckily there’s a handy script from YannUbuntu called boot repair which does this for you. I followed instructions from the SO thread we used before (here, section ‘REPAIRING THE BOOT’). In their words, “In Ubuntu after it boots, open a Terminal and type the following:



sudo add-apt-repository ppa:yannubuntu/boot-repair  

sudo apt-get update

sudo apt-get install boot-repair

“Now run

boot-repair"

That will bring up the boot-repair script (even has a handy GUI). This is pretty easy to use; the ‘recommended repair’ will probably fix things for you assuming you’ve set everything else up OK.

Check GRUB again

Reboot. Pray. Sorry, but ours was working after this step, so you’re on your own if you haven’t had much luck by now. The first thing I would do if this failed would be to check the BIOS was set to UEFI, and that the GRUB option you edited with boot-repair was active (and first) in the list.

Keep backing up your files!

The forums seem to suggest there’s a chance your system will randomly fall over in the next few weeks. So while you should back up your files in general anyway – especially if you’re working with RCUK data – it pays to be extra-vigilant for the next month.

 

That’s all.. good luck!

 

Post to Twitter Tweet this Post to Digg Digg Post to LinkedIn Post to LinkedIn Post to Slashdot Slashdot Post to StumbleUpon Stumble This

Coding, Science

Using x-windows to render GUIs for server-client phylogenomics

Leave a comment

The power of modern computers allied to high-throughput next-generation sequencing will usher in a new era of quantitative biology that will deliver the moon on a stick to every researcher on Earth…

Well, quite.

But something we’ve run up against recently in the lab is actually performing useful analyses on our phylogenomic datasets after the main evolutionary analyses. That is, once you’ve assembled, aligned, built phylogenies and measured interesting things like divergence rates and dN/dS – how do you display and explore the collected set of loci?

I’ll give you an example – suppose we have a question like “what is the correlation between tree length and mean dS rate in our 1,000 loci?” It’s fairly simple to spit the relevant statistics out into text files and then cat them together for analysis in R (or Matlab, but please, for the love of God, not Excel…). And so far, this approach has got a lot of work done for us.


But suppose we wanted to change the question slightly – maybe we notice some pattern when eyeballing a few of the most interesting alignments – and instead ask something like “what is the correlation between tree length and mean dS rate in our 1,000 loci, where polar residues are involved with a sitewise log-likelihood of ≤ 2 units?” Or <3? or <10? Most of this information is already available at runtime, parsed or calculated. We could tinker with our collation script again, re-run it, and do another round of R analyses; but this is wasteful on a variety of levels. It would be much more useful to interact with the data on the server while it’s in memory. We’ve been starting to look at ways of doing this, and that’s for another post. But for now there’s another question to solve before that becomes useful, namely – how do you execute a GUI on a server but run it on a client?

There are a variety of bioinformatics applications (Geneious; Galaxy; CLCBio) with client-server functionality built-in, but in each case we’d have to wrap our analysis in a plugin layer or module in order to use it. Since we’ve already got the analysis in this case, that seems unnecessary, so instead we’re trialling X-windows. This is nothing to do with Microsoft Windows. It’s just a display server which piggybacks an SSH connection to display UNIX Quartz (display) device remotely. It sounds simple, and it is!

To set up an x-windowing session, we just have to first set up the connection to the bioinformatics server:

ssh -Y user@host.ac.uk

Now we have a secure x-window connection, our server-based GUI will display on the client if we simply call it (MDK.jar):

User:~ /usr/bin/java -jar ~/Documents/dev-builds/mdk/MDK.jar

Rather pleasingly simple, yes? Now there’s a few issues with this, not least a noticeable cursor lag. But – remembering that the aim here is simply to interact with analysis data held in memory at runtime, rather than any do complicated editing – we can probably live with that, for now at least. On to the next task!

(also published at evolve.sbcs)

Post to Twitter Tweet this Post to Digg Digg Post to LinkedIn Post to LinkedIn Post to Slashdot Slashdot Post to StumbleUpon Stumble This

Activism, Coding

George Monbiot: Robot Wars II

2 Comments

George Monbiot recently raised the issue of automated online astroturf campaigns, in this article for the Guardian, repeated on Monbiot.com. For those that don’t know, this involves a special interest group recruiting a team to create and maintain multiple digital personas (Facebook accounts, etc). This army of virtual ghost warriors can then be used to create the impression of mass support (a.k.a. ‘grassroots’ support – geddit?) for a given cause or campaign.

Now, while this might be used for what you or I might term ‘good’ causes, the truth is that as always, those with more money and other resources (usually, the baddies) are more likely to exploit it. If ‘one person: one vote’ is the apex of democracy, then the possibility that faceless corporations or governments (or even NGOs) can use the methods of geniune campaigners to further their aims is distinctly chilling. Especially so when you consider the spread of slacktivism – the tendency for individuals’ political/social engagement to stretch no further than the online petition or the Facebook ‘like’ button..

But… a thought occurred, and I wrote to St. George. Here’s what I said:

Dear George,

read your article on astroturf campaigns with interest – had noticed it going on subliminally, but your article suggests it’s more widespread than I could possibly have imagined. A big problem, especially in the age of slacktivism…

How to deal with it? I had a quick idea, possibly seeded by Blade Runner (which I saw last night):

An online standard, or score, authenticating online personas as real people. Volunteers could specifically query individuals in what would amount to a Turing test; and/or some interaction algorithm could assess a persona’s authenticity. Of course in the latter case interested malign parties could easily (especially if the algorithm was open-source). Why would people participate? Because a

I wonder a) whether it’s worth doing, and b) if it would catch on.

I personally think that anonymity online is a bit of a curse – fine to protect political dissidents from reprisals, but an open door for abusive cowards. More and more of my generation (b:1981) place more faith in identifiable online personas, linked to either personal websites or managed through Disqus, etc.

And of course philosophically it’s an incredible moment if we’ve arrived at a point in history where a majority of human communication is online, but we can no longer even tell human from machine, let alone friend from foe!

Anyway from a quick straw poll of programming friends it seems an interesting idea to explore, so I’d love to know what you think.

Cheers,
Joe

Ideas, anyone?

Post to Twitter Tweet this Post to Digg Digg Post to LinkedIn Post to LinkedIn Post to Slashdot Slashdot Post to StumbleUpon Stumble This

Blog, Coding, Music, Science

Barcelona, LP preparations and some (lap)topless coding…

1 Comment

LP prepppppp

The nearer we get to recording the LP (2 weeks to go, and counting..) at Furnace, the more I seem to listen to PJ Harvey, Pulp, Pavement and (cat) Power. Obsession with all things ‘P’? Will we have a plethora of parping polyphones all over the place? I THINKK SO>

Barcelona

Well. A very kind person took me on a weekend trip to Barcelona to see the sights. The sights in question being the excellent capital of Catalunya herself, but mainly mates Nathan, Bobby (The Beaux Hardts) and Paola. Not been there since 2000, and since I was drunk then and drunk now I can’t really claim any deep Theroux style observations on the ephemeral nature of change in the world’s great cities, or the permanence of poverty, or anything like that, beyond the usual ones everyone makes about Barca.

These are:

  • The girls in Barcelona are all REALLY HOT
  • Why is Park Güell so far from the centre of town?
  • Supporting Barca (a winning team) is (whisper it) lots more fun than supporting one that is slightly less than triumphant like Saints [but see 1].
  • Yes, the girls in Barcelona really are all REALLY REALLY HOT [2].

Oh, and their Boris bike system (or ‘hire bike’, as they quaintly put it) works with a lot less fuss than ours. Anyway, here’s some quick crappy sketches:

Sun. Drenched. Rooftop. Morning. Coffee. MMMMMMM
... because every dickhead has to have a go at drawing it...
MEGA quick sketch from the top of Park Güell. FYizzle...

Laptopless coding

Years and years of touring experience, science fieldwork and sofa sofing mean I am a seasoned and unflappable traveller. ‘Seasoned’ like a piece of rotten wood is…

So having nearly missed my flight out, I also managed to leave my laptop charger in London. Instead of doing something sensible like getting pissed in the sun I decided to wireframe all the code for the next HADPACK iteration on pen-and-paper. WHOOOOAH!

Yes, it is oldschool, but it really is true what the textbooks say: working on paper first really helps you to think clearly about your code, especially if you’re working with complex class inheritance structures (oh, yes…)

Coding HADPACK (HIV Antigen Determining PACKage) from the ground up
More HADPACK work - thinking about how the UI interrelates to the experimental design process

OK, so there’s a bit of work left to do still. Back to the laptop…

Footnotes

1 – Although Bob informed me that Bilbao (one of the earliest Spanish teams ever founded) first played in red-and-white striped shirts that they got from which South Coast port’s city team? That would be  Southampton St. Mary’s, known nowadays as Saints. And Athletico Madrid followed suit. So Saints can at least claim the credit for kitting out the first ever Spainish football teams.

2 – No, I’m not being silly. Bob & Paola’s is just a tennis ball’s boff from the Ramblas, where Orwell (nearly) had a shootout with the Facists in la Guerra Civil. My guess is it wasn’t a lack of munitions that stopped them firing (as claimed) but the beauties of Barcelona. Make love, not war…

Post to Twitter Tweet this Post to Digg Digg Post to LinkedIn Post to LinkedIn Post to Slashdot Slashdot Post to StumbleUpon Stumble This

Coding, Science

Befi-BaTS v0.1.1 alpha release

1 Comment

Long-overdue update for beta version of Befi-BaTS.

Software: Befi-BaTS

Author: Joe Parker

Version: 0.1.1 beta (download here)

Release notes: Befi-BaTS v0.1 beta drops support for hard polytomies (tree nodes with > 2 daughters), now throwing a HardPolytomyException to the error stack when these are parsed. This is because of potential bugs when dealing with topology + distance measures (NTI/NRI) of polytomies. These bugs will be fixed in a future release. The current version 0.1.1 improves #NEXUS input file parsing.

Befi-BaTS: Befi-BaTS uses two established statistics (the Association Index, AI (Wang et al., 2001), and Fitch parsimony score, PS) as well as a third statistic (maximum exclusive single-state clade size, MC) introduced by us in the BaTS citation, where the merits of each of these are discussed. Befi-BaTS 0.1.1 includes additional statistics that include branch length as well as tree topology. What sets Befi-BaTS aside from previous methods, however, is that we incorporate uncertainty arising from phylogenetic error into the analysis through a Bayesian framework. While other many other methods obtain a null distribution for significance testing through tip character randomization, they rely on a single tree upon which phylogeny-trait association is measured for any observed or expected set of tip characters.

Post to Twitter Tweet this Post to Digg Digg Post to LinkedIn Post to LinkedIn Post to Slashdot Slashdot Post to StumbleUpon Stumble This

Blog, Coding

About this new site

Leave a comment

As you can see, I’m running a new site now. Some content will be loaded pretty soon (old blogs, lyrics, music, software etc) but the design I want to do is pretty complicated, especially as the client-side stuff, so I want to wireframe it first somewhere else offline. Might get it done in a couple of months, so look out for a bit redesign after Christmas!

This is partly an ego trip, but mainly because I’m pissed off with Facebook / Myspace and all their popularity contests and formulaic box-filling shit, and want to get back to the good old Netscape days of the 1990s, when a personal website really was just that, and you could put whatever you liked up, however you wanted to.

I’ve installed plugins for Twitter and Soundcloud, and will get flickr in there too soon. Also, for commenting, I’ve activated the really-pretty-cool DisQus engine, so that those of you on social networking sites that want to comment, can.

In the meantime I’m going to slowly let the Facebook and Myspace accounts die. Just as a little piece of me dies every time I login to those damn things.

Post to Twitter Tweet this Post to Digg Digg Post to LinkedIn Post to LinkedIn Post to Slashdot Slashdot Post to StumbleUpon Stumble This