PoolParty is undergoing a major core rewrite. Will be updated shortly

Description

PoolParty ( http://poolpartyrb.com ), Ari Lerner ( http://blog.xnot.org , http://blog.citrusbyte.com ) - PoolParty is a framework for maintaining and running auto-scalable applications on Amazon's EC2 cloud. Run entire applications using the EC2 cluster and the unlimited S3 disk.

Basics

PoolParty is written with the intention of being as application-agnostic as possible. It installs only the basic required software to glue the cloud together on the instances as listed below.

PoolParty is easily configurable. In fact, it makes little assumptions about your development environment and allows several options on how to begin configuring the cloud.

For an excellent getting started Amazon EC2 tutorial, check out Robert Sosinski's at http://www.robertsosinski.com/2008/01/26/starting-amazon-ec2-with-mac-os-x/

Install

gem sources -a http://gems.github.com
sudo gem install auser-poolparty

Development setup

IN THE ENVIRONMENT

There are 5 values that PoolParty reads from the environment, you can set these basic environment variables and leave the rest to the PoolParty defaults. Those values are:

ENV["ACCESS_KEY"] => AWS access key
ENV["SECRET_ACCESS_KEY"] => AWS secret access key
ENV["CONFIG_FILE"] => Location of your config yaml file (optional)
ENV["EC2_HOME"] => EC2 home directory (defaults to ~/.ec2)
ENV["KEYPAIR_NAME"] => The keypair used to launch instances

PoolParty assumes that you are storing your keypairs in your EC2_HOME directory with the style of: id_rsa-KEYPAIR_NAME.

IN A CONFIG FILE

PoolParty assumes your config directory is set in config/config.yml. However, you can set this in your environment variables and it will read the config file from the environment variable

WITH A RAKE TASK

PoolParty comes with a rake task that can setup your environment for you. Set the environment variables above and run rake dev:setup

This will write a .KEYPAIR_NAME_pool_keys into your home directory. Then you can just run

source ~/.KEYPAIR_NAME_pool_keys

and your environment will be all setup for you everytime you want to work on the cloud

Basics

PoolParty can work in two ways to load balance it's traffic. It can either do server-side or client-side load-balancing. Since every instance load balances itself, you can either set the client to grab an instance and send it to that using client-side load balancing (with a js library). Alternatively, you can set the master in dns and reference it when referring to the application.

Since PoolParty makes no assumptions as to what you will be hosting on the application, the world is your oyster when running a cloud. You can set each instance to register with a dynDNS service so that your application has multiple points of entry and can run load-balanced on the fly.

Every instance will auto-mount the s3 bucket set in the config file (if it is set up) into the /data folder of the instance. This gives each instance access to the same data regardless of the instance. It uses s3fuse and caching through s3fuse in the /tmp directory to work as fast as possible on the local instances.

The instances all are loaded with the following software:

When an instance is started or brought down, the master is responsible for reloading every instance with the new data on each instance. If the master goes down, the next in succession will take over as the master (using heartbeat) and it will reconfigure the cloud, setting itself as the master and take over the maintenance of the cloud.

Your cloud is never guaranteed to be maintained, but with more than 1 instance unless you have more than 1 instance up

Each instance has a /etc/hosts file that has each node listed as the node name listed in the cloud:list (rake task).

CloudSpeak - Communicating to your cloud(s)

Included in PoolParty are two binaries to communicate back with your clouds. Those are:

The cloud can be maintained entirely through rake tasks, although there are a few front-ends being developed (one in cocoa).

It is simple to include these tasks in your Rakefile. Just add the following lines: require "poolparty"
PoolParty.include_cloud_tasks # or PoolParty.tasks or PoolParty.include_tasks

All the cloud rake tasks are in the cloud namespace and can be viewed with:

rake -T cloud The instance rake tasks are in the instance namespace and can be listed with:

rake -T instance

Note

PoolParty is not married to EC2 - less than 20 lines of code are EC2-specific. Adapting PoolParty to maintain other cloud clusters would be an endeavor measured in hours, not days.

Resources

http://poolpartyrb.com
PoolParty