The compute service is implemented fully for EC2 - so you can follow the guides to using the ComputeService API.

For credentials you will want you access key id and secret access key (don't use your user id or anything else). For practical EC2 usage - you will probably want to check out AWS Getting Started Guide

Compatibility

Private clouds often expose EC2-compatible interfaces. Very commonly, users install clones on non-https, or self-signed servers. If you do so, make sure you set the following properties:

jclouds.trust-all-certs=true
jclouds.relax-hostname=true

Here are a few configuration examples of common EC2 clones:

Eucalyptus

The following properties should help use the ec2 provider on a eucalyptus install:

# in version 3, lowercase 'e' version 2, uppercase 'E'
jclouds.regions=eucalyptus
# note the trailing slash is important
ec2.endpoint=http://host:8773/services/Eucalyptus/

Images

Default image

The default image for version 1.0.0 is Amazon Linux and 64 bit. This by default chooses t1.micro size. The m1.small instance size does not support 64 bit images, if you need this, you'll have to revise the template:

// use the m1 small with amazon linux
Template template = compute.templateBuilder().hardwareId(InstanceType.M1_SMALL).osFamily(OsFamily.AMZN_LINUX).build();

In order to match an Ubuntu, or CentOs image, you'll need to see the Image Parsing section.

Image Parsing

We had feedback from one of our users that our default image list for EC2 didn't contain CentOs. As you may know, parsing all images would take minutes due to the delay in calling ec2 across all 4 regions. We now by default parse images from Amazon, Canonical, Alestic, and RightScale. With the addition of RightScale, we now support CentOs.

// pick the highest version of the RightScale CentOs template
Template template = compute.templateBuilder().osFamily(CENTOS).build();

// pick version 10.04 of ubuntu from canonical
Template template = compute.templateBuilder().hardwareId(InstanceType.M1_SMALL)
                  .osVersionMatches("10.04").imageDescriptionMatches("ubuntu-images").osFamily(OsFamily.UBUNTU).build();

Image Filters [v1.1.0+]

Even refining lists to a set of user ids, using EC2 can be bogged down, parsing the thousands of public images published by the Amazon EC2 community. Several users have expressed the desire to refine further, based on qualifications they choose. You can set a property to restrict the list to a subset matching any recognized [http://docs.amazonwebservices.com/AWSEC2/latest/APIReference/ApiReference-query-DescribeImages.html image query].

overrides = new Properties();
// choose only amazon images that are ebs-backed
overrides.setProperty(AWSEC2Constants.PROPERTY_EC2_AMI_QUERY,
                     "owner-id=137112412989;state=available;image-type=machine;root-device-type=ebs");
context = ContextBuilder.newBuilder("aws-ec2")
                  .credentials(access, secret)
                  .overrides(overrides)
                  .modules(ImmutableSet.<Module> of(new Log4JLoggingModule(),
                                                    new SshjSshClientModule()))
                  .buildView(ComputeServiceContext.class);

Properties to set cluster compute images

Cluster compute images are assigned the following default query, corresponding to the property key constant AWSEC2Constants.PROPERTY_EC2_CC_AMI_QUERY

    virtualization-type=hvm;architecture=x86_64;owner-id=137112412989,099720109477;hypervisor=xen;state=available;image-type=machine;root-device-type=ebs

In order to save time, we only look for cluster compute images in regions that support them. In order to change the default for this, affect the property PROPERTY_EC2_CC_REGIONS, currently set to "us-east-1"

Lazy Image Fetching

By default, the ComputeService will prefetch images from Alestic, Canonical, and RightScale. This allows you to query the images, which is great, if you don't know what you are looking for. However, if you already know the imageId you want, this is wasteful. We now provide the option to lazy-fetch images, which speed-ups execution.

Release 1.0.0 and below

Properties overrides = new Properties();
// set owners to nothing
overrides.setProperty(EC2Constants.PROPERTY_EC2_AMI_OWNERS, "");

context = new ComputeServiceContextFactory().createContext("aws-ec2",
   accessid, secretkey, ImmutableSet.of(new Log4JLoggingModule()), overrides);

Template template = context.getComputeService().templateBuilder().imageId(
         "ami-ccb35ea5").build();

Release 1.1.0 to 1.5.x

Properties overrides = new Properties();

// set AMI queries to nothing
overrides.setProperty(AWSEC2Constants.PROPERTY_EC2_AMI_QUERY, "");
overrides.setProperty(AWSEC2Constants.PROPERTY_EC2_CC_AMI_QUERY, "");

context = new ComputeServiceContextFactory().createContext("aws-ec2",
  accessid, secretkey, ImmutableSet.of(new Log4JLoggingModule()), overrides);

Template template = context.getComputeService().templateBuilder().imageId(
        "ami-ccb35ea5").build();

Release 1.6.0 and above

Properties overrides = new Properties();

// set AMI queries to nothing
overrides.setProperty(AWSEC2Constants.PROPERTY_EC2_AMI_QUERY, "");
overrides.setProperty(AWSEC2Constants.PROPERTY_EC2_CC_AMI_QUERY, "");

context = ContextBuilder.newBuilder("aws-ec2")
                  .credentials(accessid, secretkey)
                  .overrides(overrides)
                  .modules(ImmutableSet.<Module> of(new Log4JLoggingModule(),
                                                    new SshjSshClientModule()))
                  .buildView(ComputeServiceContext.class);

Template template = context.getComputeService().templateBuilder().imageId(
        "ami-ccb35ea5").build();

Private Images

Amazon EC2 has the concept of private-but-shared amis - images that were bundled on another account, and not made public, but shared with a specified account. This is useful in the consolidated billing scenario - where you have one user managing image curation, and others using them (but all billed together).

This private image may not show up in listImages, and jclouds will struggle with this.

Here's the solution:

Release 1.0.0 and below

Properties props = new Properties();

//have a myFavouriteOwner - the user ID of the owner of the image.
props.setProperty(EC2Constants.PROPERTY_EC2_AMI_OWNERS, myFavoriteOwner + ",063491364108,099720109477");

// or.. you can set to '*' but this will take forever on amazon's ec2 service
props.setProperty(EC2Constants.PROPERTY_EC2_AMI_OWNERS, "*");


ComputeServiceContext context = new ComputeServiceContextFactory().createContext("aws-ec2", "accesss", "secret", ImmutableSet.<Module> of(new JschSshClientModule()), props);

Releases 1.1.0 through 1.5.x

Properties props = new Properties();

//have a myFavoriteOwner - the user ID of the owner of the image.
props.setProperty(EC2Constants.PROPERTY_EC2_AMI_QUERY, "owner-id=137112412989,063491364108,099720109477,411009282317,"+myFavoriteOwner+";state=available;image-type=machine");

// or.. you can remove the owner part of the query, but this will take forever on amazon's ec2 service
props.setProperty(EC2Constants.PROPERTY_EC2_AMI_QUERY, "state=available;image-type=machine");


ComputeServiceContext context = new ComputeServiceContextFactory().createContext("aws-ec2", "accesss", "secret", ImmutableSet.<Module> of(new JschSshClientModule()), props);

Release 1.6.0 and abovex

Properties props = new Properties();

//have a myFavoriteOwner - the user ID of the owner of the image.
props.setProperty(EC2Constants.PROPERTY_EC2_AMI_QUERY, "owner-id=137112412989,063491364108,099720109477,411009282317,"+myFavoriteOwner+";state=available;image-type=machine");

// or.. you can remove the owner part of the query, but this will take forever on amazon's ec2 service
props.setProperty(EC2Constants.PROPERTY_EC2_AMI_QUERY, "state=available;image-type=machine");


ComputeServiceContext context = ContextBuilder.newBuilder("aws-ec2")
                      .credentials(access, secret)
                      .overrides(props)
                      .modules(ImmutableSet.<Module> of(new Log4JLoggingModule(),
                                                        new SshjSshClientModule()))
                      .buildView(ComputeServiceContext.class);

You can then create nodes using the templateBuilder.imageId() method.

Parsing private images

If you publish your own images, you probably would like to be able to choose the latest version using templateBuilder. For example, you might want to say templateBuilder.imageNameMatches("my-image").imageVersionMatches("1.1.0-.*").build() which would pick the latest version of your image within the 1.1.0 range.

Amazon EC2 does not have typed fields for things like operating system family, image version, etc. In jclouds, we attempt to parse these fields from well known naming conventions. However, you might not be using naming conventions we know of, so often images show up as OsFamily.UNRECOGNIZED, with no image version. Here's how to instruct jclouds to parse your images.

Create a custom parser

You'll need to override the default image parser with one that knows all the intimate secrets of your image.

package com.foo;

import java.util.Map;

import javax.inject.Inject;

import org.jclouds.aws.ec2.compute.strategy.AWSEC2ReviseParsedImage;
import org.jclouds.compute.domain.ImageBuilder;
import org.jclouds.compute.domain.OperatingSystemBuilder;
import org.jclouds.compute.domain.OsFamily;
import org.jclouds.domain.Credentials;

import com.google.common.collect.ImmutableMap;
import com.google.inject.Singleton;

@Singleton
public class FooAWSEC2ReviseParsedImage extends AWSEC2ReviseParsedImage {

   @Inject
   public FooAWSEC2ReviseParsedImage() {
      super(ImmutableMap.<OsFamily, Map<String, String>> of());
   }

   /**
    * revise image
    */
   @Override
   public void reviseParsedImage(org.jclouds.ec2.domain.Image from, ImageBuilder builder, OsFamily family,
            OperatingSystemBuilder osBuilder) {
      // I always build Ubuntu 10.10 images
      osBuilder.family(OsFamily.UBUNTU);
      osBuilder.version("10.10");

      // our image version naming convention is /us-west-1/foo/1.1.0.20110224-0001
      builder.version(from.getImageLocation().substring(from.getImageLocation().lastIndexOf('/') + 1));
      builder.name("foo");

      // our image has a default user that's sudo-enable named foo-user
      builder.defaultCredentials(new Credentials("foo-user", "password"));
   }
}

Create a test for your parser

package com.foo;

import static org.easymock.EasyMock.expect;
import static org.easymock.classextension.EasyMock.createMock;
import static org.easymock.classextension.EasyMock.replay;
import static org.easymock.classextension.EasyMock.verify;

import org.jclouds.compute.domain.ImageBuilder;
import org.jclouds.compute.domain.OperatingSystemBuilder;
import org.jclouds.compute.domain.OsFamily;
import org.jclouds.domain.Credentials;
import org.testng.annotations.Test;

@Test
public class FooAWSEC2ReviseParsedImageTest {
   public void testFooImage() {

      ImageBuilder builder = createMock(ImageBuilder.class);
      OperatingSystemBuilder osBuilder = createMock(OperatingSystemBuilder.class);
      org.jclouds.ec2.domain.Image from = createMock(org.jclouds.ec2.domain.Image.class);

      expect(from.getImageLocation()).andReturn("/us-west-1/foo/1.1.0.20110224-0001").times(2);

      expect(osBuilder.family(OsFamily.UBUNTU)).andReturn(osBuilder);
      expect(osBuilder.version("10.10")).andReturn(osBuilder);
      // ensure our version parses properly
      expect(builder.version("1.1.0.20110224-0001")).andReturn(builder);
      expect(builder.name("foo")).andReturn(builder);
      expect(builder.defaultCredentials(new Credentials("foo-user", "password"))).andReturn(builder);

      // replay mocks
      replay(builder);
      replay(osBuilder);
      replay(from);

      new FooAWSEC2ReviseParsedImage().reviseParsedImage(from, builder, OsFamily.UNRECOGNIZED, osBuilder);

      // verify mocks
      verify(builder);
      verify(osBuilder);
      verify(from);
   }

}

Instruct jclouds to use your parser

Properties overrides = setupProperties();
// your owner id
overrides.setProperty(EC2Constants.PROPERTY_EC2_AMI_OWNERS, "123123123213123");

context = ContextBuilder.newBuilder("aws-ec2")
                      .credentials(access, secret)
                      .overrides(overrides)
                      .modules(ImmutableSet.<Module> of(new Log4JLoggingModule(),
                                                        new SshjSshClientModule(),
                                                        new AbstractModule(){

            @Override
            protected void configure() {
               bind(AWSEC2ReviseParsedImage.class).to(FooAWSEC2ReviseParsedImage.class);
            })))
                      .buildView(ComputeServiceContext.class);

Use your image version

jclouds will now pick the lexicographic highest version, as it now can parse your images.

Template template = context.getComputeService().templateBuilder().imageVersionMatches("1.1.0.*").build();

ComputeService API extensions

Power users have requested more control over the choices jclouds ComputeService makes when provisioning nodes. By default, we automatically create a security group and keypair for your nodes before launching them. We now allow you to control this a bit through extended template options.

Spot Instances

If you are using the aws-ec2 provider, you can use spot instances via the spotPrice parameter on template options:

options.as(AWSEC2TemplateOptions.class).spotPrice(0.3f);

AWSEC2ComputeServiceLiveTest.testExtendedOptionsAndLogin() uses the spot price option on the portable interface.

Details

In the code, AWSEC2CreateNodesInGroupThenAddToSet actually manages creating the spot instance request. The way jclouds manages this is that it looks for both regular reservations and a\ so spot requests when listing nodes. The convergence of these 2 is what you'll see in a listNodes command (AWSEC2ListNodesStrategy ex. does these multiple listings in parallel)

If there's an error on the parameters requesting nodes, you'll receive an HttpResponseException in the 400 range from ec2 itself. The other condition you should be aware of is when the\ spot request goes through, but perhaps takes longer to provision the nodes than the jclouds default timeout (jclouds.compute.timeout.node-running), something I've not seen.

Security Groups

// specify your own groups which already have the correct rules applied
template.getOptions().as(EC2TemplateOptions.class).securityGroups(group1, group2);

Key Pairs

// specify your own keypair for use in creating nodes
template.getOptions().as(EC2TemplateOptions.class).keyPair(group);

// if your image doesn't use keypairs (ex enstratus), skip creating one
template.getOptions().as(EC2TemplateOptions.class).noKeyPair();

Amazon doesn't store the private key data, so if you supply an existing public key for jclouds to use, you'll also need to supply the private key correlating to it.

There are two ways you can tell jclouds to use your keypair for an EC2 AMI.

The preferable way is to just pass it as Template Option. If you are using an image where jclouds doesn't know the login user, you'll need to specify the option.overrideCredentialsWith and pass along the user that's baked in.

overrideLoginCredentialWith(your_id_rsa_string)
overrideCredentialsWith(new Credentials("root", your_id_rsa_string))

If you want to authorize your keypair, you can use the authorizePublicKey(yourKey) method

client.createNodesInGroup(group, 1, authorizePublicKey(myKey));

Here's how you can run a script using the overrideLoginCredentialsWith

templateOptions.runScript(_script_). authorizePublicKey().overrideLoginCredentialWith(private)

Keep in mind that authorizePublicKey() is redundant, if it is the same as what corresponds to the keyPair() option.

With respect to the security group, jclouds creates a security group for you, with rules corresponding to the inboundPorts() option (defaults to open port 22), unless you use the option EC2TemplateOptions.securityGroups().

The other way is the push your credentials into the credentials store so that jClouds uses it.

When a keypair is automatically created, jclouds puts the keypair into the Credentials Map. You can use the same option to put your credentials into the Credentials Map using the credentialStore

ComputeServiceContext.credentialStore

Note, the more you use features like security groups and keypairs, the less portable your code will be across clouds.

VPC

To create nodes in a subnet under Amazon VPC add the following option to your template options. Note that VPCs and Security Groups are mutually exclusive.

TemplateOptions options = compute.templateOptions();
options.as(AWSEC2TemplateOptions.class).subnetId(subnetId);

Set<? extends NodeMetadata> nodes = client.runNodesInGroup(group, 1,
               options);

You can also checkout EC2ComputeServiceListTest.testExtendedOptionsWithSubnetId()

Monitoring (Cloud Watch)

Note this is currently only in 1.0-SNAPSHOT

To create nodes that are automatically monitored in CloudWatch, add the following to your template options:

TemplateOptions options =  compute.templateOptions();
options.as(EC2TemplateOptions.class).enableMonitoring();

Set<? extends NodeMetadata> monitoredNodes = compute.runNodesInGroup(group, 1, options);

You can then use the !CloudWatchClient to get statistics on your nodes.

RestContext<CloudWatchClient, CloudWatchAsyncClient> cloudWatchContext =
              ContextBuilder.newBuilder("cloudwatch")
                      .credentials(access, secret)
                      .build();

String region = node.getLocation().getParent().getId();

Set<Datapoint> datapoints = monitoringContext.getApi().getMetricStatisticsInRegion(region,
                  "CPUUtilization", before, new Date(), 60, "Average");

GovCloud

To operate within a GovCloud instance, you will need to override the aws ec2 endpoint. For EC2:

public static final String GOV_CLOUD_ENDPOINT = "https://ec2.us-gov-west-1.amazonaws.com";
overrides.setProperty("aws-ec2.endpoint", GOV_CLOUD_ENDPOINT);

You can find GovCloud endpoints for other services here: http://docs.amazonwebservices.com/general/latest/gr/rande.html#govcloud_region