How to use Xgrid?

Question

I have read a little about Xgrid on Mac OS X, and I read that I can use it to get more power for some applications.

What type of software can use Xgrid? And how can I use it?

Solution 1

One of the best options I've found for seeing if Xgrid is even what you are looking for is to read the Xgrid Tutorials from MacResearch.

Suffice to say Xgrid is predominantly aimed at the scientific community where they write their programs to take advantage of it, so wanting to use it to take advantage of programs such as iMovie, Photoshop, Final Cut, etc. is a non starter.

See the FAQ from the Xgrid mailing list:

If you want to use something similar to Xgrid with the Final Cut suite or Maya check out QMaster which is part of the Final Cut suite.

Solution 2

Chealion answered the how part of your question quite well.

To address which types of applications can use Xgrid:

Xgrid is well suited to any task that can be distributed to run in parallel on multiple computers, whether workstations or servers. Sometimes these problem sets are referred to as "embarrassingly parallel". Computations which take a long time to complete, but require little data to cross the network, are excellent candidates (like Seti@Home).

Xgrid consists of 3 main parts (controller, agent, client). Tasks are issued from the client (command-line or API) to the controller and run on any available agent, similar to earlier distributed scheduling systems like SGE (bsub/qsub) or MPI. In addition to custom-code utilizing xgrid APIs, any application that can be invoked from the command line can utilize xgrid. Running a task is as easy as the following (where uname could be any command-line application, like matlab, R, blender, maya, etc.):

$ xgrid -h controller.mygrid.com -job run /usr/bin/uname -a  
Darwin node006.mygrid.com 10.2.0 Darwin Kernel Version 10.2.0 ...

Common uses of Xgrid:

• scientific computing
• distributed 3D rendering
• distributed audio / video processing

Xgrid is not well suited to tasks which are inter-dependent (have to be executed in serial) or those which require moving large amounts of data (where copying data takes longer than just running locally).

Xgrid itself simply executes a given task on a remote system, it cannot break down an operation performed by an application into smaller parts (like photoshop applying an effect to a single image). Tasks which can't be broken down into smaller pieces benefit more from co-processors or GPU processing (via CUDA / OpenCL / GPGPU).

In the case of Photoshop, xgrid would be useful if you had to apply transforms / effects against thousands of images, as each computer in the grid could work on a separate image concurrently.

Additional resources:

• Apple FAQ
  http://lists.apple.com/faq/pub/xgrid_users/index.php?sid=16080

• Tutorials by Charles Parnot at Macresearch
  http://www.macresearch.org/the_xgrid_tutorials

• a blog by Tom Elliott
  http://telliott99.blogspot.com/search/label/Xgrid

• Xgrid and distributed rendering with Maya
  http://himiko.dnsalias.net/twiki/bin/view/Main/AppleXgrid

• Apple docs (guide and update)
  http://developer.apple.com/mac/library/documentation/MacOSXServer/Conceptual/Xgrid_Programming_Guide/Introduction/Introduction.html
  http://developer.apple.com/mac/library/documentation/Performance/Reference/XgridFoundationRefUpdate/Introduction/Introduction.html

• "Xgrid, a 'just do it' grid solution for non-ITs"
  http://images.apple.com/science/pdf/embnet_news11_3abstract.pdf

• Interoperability with other software (MATLAB, BLAST, Maya, etc.)
  http://lists.apple.com/faq/pub/xgrid_users/index.php?sid=16080&aktion=anzeigen&rubrik=001&seite=2

Related Applications...

• xgridstatus
  http://cmgm.stanford.edu/~cparnot/xgrid-stanford/html/goodies/xgridstatus-info.html

• GridStuffer
  http://cmgm.stanford.edu/~cparnot/xgrid-stanford/html/goodies/GridStuffer-info.html

• XgridLite
  http://edbaskerville.com/software/xgridlite/

Cross Platform Xgrid Components...

• Java Agent by Curtis Campbell
  http://sourceforge.net/projects/xgridagent-java/

• Linux Agent by Daniel Cote
  http://unu.novajo.ca/simple/archives/000026.html

• Java Xgrid Client
  http://www.ing.unitn.it/~luttero/javaonMac/index.html#XGridClient

Comments

[ParisNakitaKejser]

Cool, can i use xgrid for Adobe Photoshop CS4 e.g.? :)

[Chealion]

FWIW, Xgrid and Compressor don't actually work together out of the box - instead Apple has a product included with Compressor called QMaster that does that job.

[Chopper3]

@NeoNman nopers, sorry, that would be like some kind of Holy Grail, with extra cheese and a free priority pass for space mountain, sorry.