X

Java S3 Example

In this tutorial I will explain how to use Amazon’s S3 storage with the Java API provided by Amazon. The example shows you how to create a bucket, list it’s content, create a folder into a bucket, upload a file, give the file a public access and finally how to delete all this items.

Setting Up Your Project

  1. You will need the AWS SDK for Java for this example to work. If you didn’t downloaded the SDK by now, please download it here. You will also need to integrate the .JAR files from the archive into your project. Alternately you can use Maven with following dependency
    <dependency>
        <groupId>com.amazonaws</groupId>
        <artifactId>aws-java-sdk</artifactId>
        <version>1.9.2</version>
    </dependency>
  2. Create a user in Amazon IAM (https://console.aws.amazon.com/iam) if you don’t have one. Here you will get a “Access key” and “Secret Access Key”. You will need this credentials to connect to S3.
  3. Add the “AWSConnector” and “AmazonS3FullAccess” permissions in IAM to your new user. Without this you want be able to authenticate with the server.

Authenticate with Amazon S3

There are 4 different methods to authenticate your requests against Amazon S3

1. Use the default credential profiles file – this is the recommend way by Amazon. Create a file with following structure and fill in the access keys:

# Move this credentials file to (~/.aws/credentials)
# after you fill in your access and secret keys in the default profile

# WARNING: To avoid accidental leakage of your credentials,
#          DO NOT keep this file in your source directory.
[default]
aws_access_key_id=
aws_secret_access_key=

save this file under the file-name “credentials” in your .aws folder by default C:\Users\user\.aws\credentials for Windows users or your home directory in Linux

If you use this method you can create a Credentials object in your code like this:

AWSCredentials credentials = new ProfileCredentialsProvider().getCredentials();

2. Use Environment Variables – set the values of following environment variables in your system AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY

3. Java System Propertiesaws.accessKeyId and aws.secretKey. Use SystemPropertiesCredentialsProvider to load the variables in your program

4. Programmatically set credentials – in this example I will use this method, because it is easier to follow

Use following in your code:

AWSCredentials credentials = new BasicAWSCredentials("YourAccessKeyID", "YourSecretAccessKey");

Create S3 Client

To be able to communicate with S3 you have to use an implementation of AmazonS3. You will use the instance to address requests to the server

AmazonS3 s3client = new AmazonS3Client(credentials);

Create Bucket

Buckets must have unique names within the whole S3 realm

String bucketName = "javatutorial-net-example-bucket";
s3client.createBucket(bucketName);

List Buckets

You can get the list off all buckets like this

for (Bucket bucket : s3client.listBuckets()) {
	System.out.println(" - " + bucket.getName());
}

Create Folder in S3 Bucket

Use this code to create an empty folder in your bucket

public static void createFolder(String bucketName, String folderName, AmazonS3 client) {
	// create meta-data for your folder and set content-length to 0
	ObjectMetadata metadata = new ObjectMetadata();
	metadata.setContentLength(0);

	// create empty content
	InputStream emptyContent = new ByteArrayInputStream(new byte[0]);

	// create a PutObjectRequest passing the folder name suffixed by /
	PutObjectRequest putObjectRequest = new PutObjectRequest(bucketName,
				folderName + SUFFIX, emptyContent, metadata);

	// send request to S3 to create folder
	client.putObject(putObjectRequest);
}

Upload file in S3

If you want to upload a file into a folder use this

String fileName = folderName + SUFFIX + "testvideo.mp4";
s3client.putObject(new PutObjectRequest(bucketName, fileName, 
		new File("C:\\Users\\user\\Desktop\\testvideo.mp4")));

Just remove the folder and the suffix in the file-name to upload directly to the bucket. If you want to make your file public (files are private by default in Amazon S3) set this to your PutObjectRequest (see the complete example below for more details)

.withCannedAcl(CannedAccessControlList.PublicRead)

Deleting Files, Folders and Buckets

To delete a bucket use this. Buckets must be empty or you can’t delete them

s3client.deleteBucket(bucketName);

To delete files use:

s3client.deleteObject(bucketName, fileName);

To delete a folder you have to delete all files in it first. Please look at the complete example below for more info.



Complete Example

Here you will find all the snippets above in one working program. Exception handling is eexcluded for readability, please don’t forget to add it in your code

import java.io.ByteArrayInputStream;
import java.io.File;
import java.io.InputStream;
import java.util.List;

import com.amazonaws.auth.AWSCredentials;
import com.amazonaws.auth.BasicAWSCredentials;
import com.amazonaws.services.s3.AmazonS3;
import com.amazonaws.services.s3.AmazonS3Client;
import com.amazonaws.services.s3.model.Bucket;
import com.amazonaws.services.s3.model.CannedAccessControlList;
import com.amazonaws.services.s3.model.ObjectMetadata;
import com.amazonaws.services.s3.model.PutObjectRequest;
import com.amazonaws.services.s3.model.S3ObjectSummary;

public class AmazonS3Example {
	
	private static final String SUFFIX = "/";
	
	public static void main(String[] args) {
		// credentials object identifying user for authentication
		// user must have AWSConnector and AmazonS3FullAccess for 
		// this example to work
		AWSCredentials credentials = new BasicAWSCredentials(
				"YourAccessKeyID", 
				"YourSecretAccessKey");
		
		// create a client connection based on credentials
		AmazonS3 s3client = new AmazonS3Client(credentials);
		
		// create bucket - name must be unique for all S3 users
		String bucketName = "javatutorial-net-example-bucket";
		s3client.createBucket(bucketName);
		
		// list buckets
		for (Bucket bucket : s3client.listBuckets()) {
			System.out.println(" - " + bucket.getName());
		}
		
		// create folder into bucket
		String folderName = "testfolder";
		createFolder(bucketName, folderName, s3client);
		
		// upload file to folder and set it to public
		String fileName = folderName + SUFFIX + "testvideo.mp4";
		s3client.putObject(new PutObjectRequest(bucketName, fileName, 
				new File("C:\\Users\\user\\Desktop\\testvideo.mp4"))
				.withCannedAcl(CannedAccessControlList.PublicRead));
		
		deleteFolder(bucketName, folderName, s3client);
		
		// deletes bucket
		s3client.deleteBucket(bucketName);
	}
	
	public static void createFolder(String bucketName, String folderName, AmazonS3 client) {
		// create meta-data for your folder and set content-length to 0
		ObjectMetadata metadata = new ObjectMetadata();
		metadata.setContentLength(0);

		// create empty content
		InputStream emptyContent = new ByteArrayInputStream(new byte[0]);

		// create a PutObjectRequest passing the folder name suffixed by /
		PutObjectRequest putObjectRequest = new PutObjectRequest(bucketName,
				folderName + SUFFIX, emptyContent, metadata);

		// send request to S3 to create folder
		client.putObject(putObjectRequest);
	}

	/**
	 * This method first deletes all the files in given folder and than the
	 * folder itself
	 */
	public static void deleteFolder(String bucketName, String folderName, AmazonS3 client) {
		List fileList = 
				client.listObjects(bucketName, folderName).getObjectSummaries();
		for (S3ObjectSummary file : fileList) {
			client.deleteObject(bucketName, file.getKey());
		}
		client.deleteObject(bucketName, folderName);
	}
}

If you find this tutorial helpful, please check out our other tutorials – we are sure you will find other interesting stuff on this page. Feel free to ask questions in “comments” section or just show us some love by sharing this tutorial 🙂

4.7 9 votes
Article Rating
filip:

View Comments (21)

  • I have added aws-java-sdk-1.9.7.jar jar in my project but still its not working. Can you please list out the jar file require to setting up the project.

    • Hi,

      if you use Maven you can add following dependency in your .pom file

      <dependency>
      <groupId>com.amazonaws</groupId>
      <artifactId>aws-java-sdk</artifactId>
      <version>1.9.2</version>
      </dependency>

      If you don't use Maven you need to add all of this (you can copy the JAR-s from AWS SDK)
      * aspectjrt.jar
      * aspectjweaver.jar
      * aws-java-sdk-1.9.0.jar
      * commons-codec-1.3.jar
      * commons-logging-1.1.1.jar
      * freemarker-2.3.18.jar
      * httpclient-4.2.3.jar
      * httpcore-4.2.jar
      * jackson-annotations-2.1.1.jar
      * jackson-core-2.1.1.jar
      * jackson-databind-2.1.1.jar
      * joda-time-2.2.jar
      * mail-1.4.3.jar
      * spring-beans-3.0.7.jar
      * spring-context-3.0.7.jar
      * spring-core-3.0.7.jar

  • I get java.lang.ClassNotFoundException: org.apache.commons.logging.LogFactory exception on line AmazonS3 s3client = new AmazonS3Client(credentials);

    Any ideas please?

    I am struggling to export a file to local system in my java web application. Please help!

    • Hi,
      are you using maven for your build or you have the jars? If you use the jars, make sure you have commons-logging in your path

  • I have implemented above code and also have added all mentioned jars in my classpath.
    I am getting following error: Please help

    Exception in thread "main" java.lang.IllegalArgumentException
    at com.amazonaws.auth.SignerFactory.createSigner(SignerFactory.java:118)
    at com.amazonaws.auth.SignerFactory.lookupAndCreateSigner(SignerFactory.java:107)
    at com.amazonaws.auth.SignerFactory.getSigner(SignerFactory.java:80)
    at com.amazonaws.AmazonWebServiceClient.computeSignerByServiceRegion(AmazonWebServiceClient.java:311)
    at com.amazonaws.AmazonWebServiceClient.computeSignerByURI(AmazonWebServiceClient.java:284)
    at com.amazonaws.AmazonWebServiceClient.setEndpoint(AmazonWebServiceClient.java:160)
    at com.amazonaws.services.s3.AmazonS3Client.(AmazonS3Client.java:200)
    at com.amazonaws.services.s3.AmazonS3Client.(AmazonS3Client.java:182)
    at com.aspireiten.util.UploadObjectSingleOperation.main(UploadObjectSingleOperation.java:26)

    • Hi,

      if you take a look at createSigner method in here https://github.com/aws/aws-sdk-java/blob/master/aws-java-sdk-core/src/main/java/com/amazonaws/auth/SignerFactory.java

      ... the only chance to throw IllegalArgumentException is when the signerClass is NULL. The signerClass can be NULL only if String signerType is incorrect and can not be found among those 4:

      SIGNERS.put(QUERY_STRING_SIGNER, QueryStringSigner.class);
      SIGNERS.put(VERSION_THREE_SIGNER, AWS3Signer.class);
      SIGNERS.put(VERSION_FOUR_SIGNER, AWS4Signer.class);
      SIGNERS.put(NO_OP_SIGNER, NoOpSigner.class);

      Now going back the trace down and looking at AmazonWebServiceClient class I see that AWS backand and signer type are calculated based on your credentials.

      I will suggest you check again your credentials and your account with AWS. Please share your results here. Hope this helps

  • I am getting below error and i am using aws-java-sdk-1.11.299,
    error on AmazonS3 s3client = new AmazonS3Client(credentials);
    java.lang.NoClassDefFoundError: Could not initialize class com.amazonaws.http.conn.ssl.SdkTLSSocketFactory

    java.lang.NoClassDefFoundError: Could not initialize class com.amazonaws.http.conn.ssl.SdkTLSSocketFactory
    at com.amazonaws.http.apache.client.impl.ApacheConnectionManagerFactory.getPreferredSocketFactory(ApacheConnectionManagerFactory.java:88)
    at com.amazonaws.http.apache.client.impl.ApacheConnectionManagerFactory.create(ApacheConnectionManagerFactory.java:65)
    at com.amazonaws.http.apache.client.impl.ApacheConnectionManagerFactory.create(ApacheConnectionManagerFactory.java:58)
    at com.amazonaws.http.apache.client.impl.ApacheHttpClientFactory.create(ApacheHttpClientFactory.java:50)
    at com.amazonaws.http.apache.client.impl.ApacheHttpClientFactory.create(ApacheHttpClientFactory.java:38)
    at com.amazonaws.http.AmazonHttpClient.(AmazonHttpClient.java:314)
    at com.amazonaws.http.AmazonHttpClient.(AmazonHttpClient.java:298)
    at com.amazonaws.AmazonWebServiceClient.(AmazonWebServiceClient.java:165)
    at com.amazonaws.services.s3.AmazonS3Client.(AmazonS3Client.java:562)
    at com.amazonaws.services.s3.AmazonS3Client.(AmazonS3Client.java:542)
    at com.amazonaws.services.s3.AmazonS3Client.(AmazonS3Client.java:520)
    at com.amazonaws.services.s3.AmazonS3Client.(AmazonS3Client.java:487)
    at com.amazonaws.services.s3.AmazonS3Client.(AmazonS3Client.java:466)
    at _jsp._clzone._ImageTextDetect__jsp._jspService(clzone/ImageTextDetect.jsp:10)
    at _jsp._clzone._ImageTextDetect__jsp._jspService(_ImageTextDetect__jsp.java:53)
    at com.caucho.jsp.JavaPage.service(JavaPage.java:64)
    at com.caucho.jsp.Page.pageservice(Page.java:548)
    at com.caucho.server.dispatch.PageFilterChain.doFilter(PageFilterChain.java:194)
    at com.caucho.server.webapp.WebAppFilterChain.doFilter(WebAppFilterChain.java:156)
    at com.caucho.server.webapp.AccessLogFilterChain.doFilter(AccessLogFilterChain.java:95)
    at com.caucho.server.dispatch.ServletInvocation.service(ServletInvocation.java:290)
    at com.caucho.server.hmux.HmuxRequest.handleInvocation(HmuxRequest.java:476)
    at com.caucho.server.hmux.HmuxRequest.handleRequestImpl(HmuxRequest.java:374)
    at com.caucho.server.hmux.HmuxRequest.handleRequest(HmuxRequest.java:341)
    at com.caucho.network.listen.TcpSocketLink.dispatchRequest(TcpSocketLink.java:1353)
    at com.caucho.network.listen.TcpSocketLink.handleRequest(TcpSocketLink.java:1309)
    at com.caucho.network.listen.TcpSocketLink.handleRequestsImpl(TcpSocketLink.java:1293)
    at com.caucho.network.listen.TcpSocketLink.handleRequests(TcpSocketLink.java:1201)
    at com.caucho.network.listen.TcpSocketLink.handleAcceptTaskImpl(TcpSocketLink.java:997)
    at com.caucho.network.listen.ConnectionTask.runThread(ConnectionTask.java:117)
    at com.caucho.network.listen.ConnectionTask.run(ConnectionTask.java:93)
    at com.caucho.network.listen.SocketLinkThreadLauncher.handleTasks(SocketLinkThreadLauncher.java:169)
    at com.caucho.network.listen.TcpSocketAcceptThread.run(TcpSocketAcceptThread.java:61)
    at com.caucho.env.thread2.ResinThread2.runTasks(ResinThread2.java:173)
    at com.caucho.env.thread2.ResinThread2.run(ResinThread2.java:118)

  • I am getting this exception :- DEBUG com.amazonaws.internal.config.InternalConfig - Configuration override awssdk_config_override.json not found.
    When running this code.

  • the listing of the buckets doesn't show the bucket that has the zip files , need to download the zip objects and then modify it and then upload to different bucket

    • After putting above code to create a folder and upload a file, I am able to run the program successfully but when I logged into aws console I am able to see only the folder structure and not the file for ex. I see only A/B/C and there is no file in it.

  • Couldn't stop myself from commenting. Thanks a ton :). It worked great. Much needed support for starting.

  • hi, i am looking java code to worked with amazon s3 and amazon cloudfront CDN to deliver data to viewers.
    can anyone give me idea how can i code over them.

Related Post