[Segway-users] Segway on Amazon AWS

[Jay Hesselberth]

FYI I forgot to mention: it's silly to launch large cluster for segmentation
and spend your time waiting to transfer genomedata archives (which can be
BIG).  I usually fire up small instances, mount the EBS volume to those and
do the transfer.  Then, when transfer is done, mount that EBS volume on a
larger cluster for analysis.

--
Jay Hesselberth, PhD
Biochemistry & Molecular Genetics
University of Colorado School of Medicine
off: (303) 724-5384
lab: (303) 724-5486
jay.hesselberth at gmail.com


On Wed, Apr 13, 2011 at 6:03 PM, Michael Hoffman <mmh1 at uw.edu> wrote:


> Thanks very much, Jay! This is really cool stuff.

>

> Please note that the Segway 1.0.3 snapshot Jay made (snap-dbe720b4) is

> only available in the US East Region, which is probably what you want

> to use for compute anyway, as it is cheaper.

>

>

> Michael

>

> On Wed, 13 Apr 2011, Jay Hesselberth wrote:

>

>  Dear Segway users,

>>

>> Michael asked me to put together some instructions for getting Segway

>> (and genomedata & segtools) running on Amazon Web Services.  This

>> turns out to be straightforward to do, and I've outlined the process

>> here (

>> http://sc3id.ucdenver.edu/jhessel/segway-cloud/segway-amazon-cloud.html).

>>

>> Essentially it amounts to:

>>

>> 1.  Signing up for an AWS account (you'll need a CC number)

>> 2.  Installing starcluster on your local client

>> (http://web.mit.edu/stardev/cluster/)

>> 3.  Copying the public segway snapshot over to your own EBS volume

>> 4.  Launching a AWS cluster instance of arbitrary size with

>> starcluster, with specific instructions via starcluster to mount the

>> segway volume (instructions above)

>> 5.  Transfer some data in (e.g. a genomedata archive) onto your

>> cluster (e.g. via scp).  You'll want to create a separate EBS volume

>> for this so that the data is persistent (I typically make 50 Gb

>> volumes as I need them).

>> 6.  Launch a segmentation.

>>

>> We have had good luck running multi-day segmentations for reasonably

>> cheap.  By the way, I was mainly motivated to do this because the

>> cluster that I have routine access to is a Mac OS X cluster, and it

>> turns out to be non-trivial to install segway in that environment

>> (mainly SGE/drmaa limitations).  I've also figured out a workaround

>> for that, if anyone is interested.

>>

>> Best,

>> --

>> Jay Hesselberth, PhD

>> Biochemistry & Molecular Genetics

>> University of Colorado School of Medicine

>> off: (303) 724-5384

>> lab: (303) 724-5486

>> jay.hesselberth at gmail.com

>> _______________________________________________

>> Segway-users mailing list

>> Segway-users at u.washington.edu

>> http://mailman2.u.washington.edu/mailman/listinfo/segway-users

>>

>>

> --

> Michael Hoffman, PhD

> Senior Fellow

> Department of Genome Sciences

> University of Washington

> PO Box 355065

> Seattle, WA 98195-5065

>

> mmh1 at uw.edu

> _______________________________________________

> Segway-users mailing list

> Segway-users at u.washington.edu

> http://mailman2.u.washington.edu/mailman/listinfo/segway-users

>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman2.u.washington.edu/pipermail/segway-users/attachments/20110421/1f0a780b/attachment.html