In this tutorial I will explain how to use Amazon’s S3 storage with the Java API provided by Amazon. The example shows you how to create a bucket, list it’s content, create a folder into a bucket, upload a file, give the file a public access and finally how to delete all this items.
Setting Up Your Project
- You will need the AWS SDK for Java for this example to work. If you didn’t downloaded the SDK by now, please download it here. You will also need to integrate the .JAR files from the archive into your project. Alternately you can use Maven with following dependency
<dependency> <groupId>com.amazonaws</groupId> <artifactId>aws-java-sdk</artifactId> <version>1.9.2</version> </dependency> - Create a user in Amazon IAM (https://console.aws.amazon.com/iam) if you don’t have one. Here you will get a “Access key” and “Secret Access Key”. You will need this credentials to connect to S3.
- Add the “AWSConnector” and “AmazonS3FullAccess” permissions in IAM to your new user. Without this you want be able to authenticate with the server.
Authenticate with Amazon S3
There are 4 different methods to authenticate your requests against Amazon S3
1. Use the default credential profiles file – this is the recommend way by Amazon. Create a file with following structure and fill in the access keys:
# Move this credentials file to (~/.aws/credentials) # after you fill in your access and secret keys in the default profile # WARNING: To avoid accidental leakage of your credentials, # DO NOT keep this file in your source directory. [default] aws_access_key_id= aws_secret_access_key=
save this file under the file-name “credentials” in your .aws folder by default C:\Users\user\.aws\credentials for Windows users or your home directory in Linux
If you use this method you can create a Credentials object in your code like this:
AWSCredentials credentials = new ProfileCredentialsProvider().getCredentials();
2. Use Environment Variables – set the values of following environment variables in your system AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY
3. Java System Properties – aws.accessKeyId and aws.secretKey. Use SystemPropertiesCredentialsProvider to load the variables in your program
4. Programmatically set credentials – in this example I will use this method, because it is easier to follow
Use following in your code:
AWSCredentials credentials = new BasicAWSCredentials("YourAccessKeyID", "YourSecretAccessKey");
Create S3 Client
To be able to communicate with S3 you have to use an implementation of AmazonS3. You will use the instance to address requests to the server
AmazonS3 s3client = new AmazonS3Client(credentials);
Create Bucket
Buckets must have unique names within the whole S3 realm
String bucketName = "javatutorial-net-example-bucket"; s3client.createBucket(bucketName);
List Buckets
You can get the list off all buckets like this
for (Bucket bucket : s3client.listBuckets()) {
System.out.println(" - " + bucket.getName());
}
Create Folder in S3 Bucket
Use this code to create an empty folder in your bucket
public static void createFolder(String bucketName, String folderName, AmazonS3 client) {
// create meta-data for your folder and set content-length to 0
ObjectMetadata metadata = new ObjectMetadata();
metadata.setContentLength(0);
// create empty content
InputStream emptyContent = new ByteArrayInputStream(new byte[0]);
// create a PutObjectRequest passing the folder name suffixed by /
PutObjectRequest putObjectRequest = new PutObjectRequest(bucketName,
folderName + SUFFIX, emptyContent, metadata);
// send request to S3 to create folder
client.putObject(putObjectRequest);
}
Upload file in S3
If you want to upload a file into a folder use this
String fileName = folderName + SUFFIX + "testvideo.mp4";
s3client.putObject(new PutObjectRequest(bucketName, fileName,
new File("C:\\Users\\user\\Desktop\\testvideo.mp4")));
Just remove the folder and the suffix in the file-name to upload directly to the bucket. If you want to make your file public (files are private by default in Amazon S3) set this to your PutObjectRequest (see the complete example below for more details)
.withCannedAcl(CannedAccessControlList.PublicRead)
Deleting Files, Folders and Buckets
To delete a bucket use this. Buckets must be empty or you can’t delete them
s3client.deleteBucket(bucketName);
To delete files use:
s3client.deleteObject(bucketName, fileName);
To delete a folder you have to delete all files in it first. Please look at the complete example below for more info.
Complete Example
Here you will find all the snippets above in one working program. Exception handling is eexcluded for readability, please don’t forget to add it in your code
import java.io.ByteArrayInputStream;
import java.io.File;
import java.io.InputStream;
import java.util.List;
import com.amazonaws.auth.AWSCredentials;
import com.amazonaws.auth.BasicAWSCredentials;
import com.amazonaws.services.s3.AmazonS3;
import com.amazonaws.services.s3.AmazonS3Client;
import com.amazonaws.services.s3.model.Bucket;
import com.amazonaws.services.s3.model.CannedAccessControlList;
import com.amazonaws.services.s3.model.ObjectMetadata;
import com.amazonaws.services.s3.model.PutObjectRequest;
import com.amazonaws.services.s3.model.S3ObjectSummary;
public class AmazonS3Example {
private static final String SUFFIX = "/";
public static void main(String[] args) {
// credentials object identifying user for authentication
// user must have AWSConnector and AmazonS3FullAccess for
// this example to work
AWSCredentials credentials = new BasicAWSCredentials(
"YourAccessKeyID",
"YourSecretAccessKey");
// create a client connection based on credentials
AmazonS3 s3client = new AmazonS3Client(credentials);
// create bucket - name must be unique for all S3 users
String bucketName = "javatutorial-net-example-bucket";
s3client.createBucket(bucketName);
// list buckets
for (Bucket bucket : s3client.listBuckets()) {
System.out.println(" - " + bucket.getName());
}
// create folder into bucket
String folderName = "testfolder";
createFolder(bucketName, folderName, s3client);
// upload file to folder and set it to public
String fileName = folderName + SUFFIX + "testvideo.mp4";
s3client.putObject(new PutObjectRequest(bucketName, fileName,
new File("C:\\Users\\user\\Desktop\\testvideo.mp4"))
.withCannedAcl(CannedAccessControlList.PublicRead));
deleteFolder(bucketName, folderName, s3client);
// deletes bucket
s3client.deleteBucket(bucketName);
}
public static void createFolder(String bucketName, String folderName, AmazonS3 client) {
// create meta-data for your folder and set content-length to 0
ObjectMetadata metadata = new ObjectMetadata();
metadata.setContentLength(0);
// create empty content
InputStream emptyContent = new ByteArrayInputStream(new byte[0]);
// create a PutObjectRequest passing the folder name suffixed by /
PutObjectRequest putObjectRequest = new PutObjectRequest(bucketName,
folderName + SUFFIX, emptyContent, metadata);
// send request to S3 to create folder
client.putObject(putObjectRequest);
}
/**
* This method first deletes all the files in given folder and than the
* folder itself
*/
public static void deleteFolder(String bucketName, String folderName, AmazonS3 client) {
List fileList =
client.listObjects(bucketName, folderName).getObjectSummaries();
for (S3ObjectSummary file : fileList) {
client.deleteObject(bucketName, file.getKey());
}
client.deleteObject(bucketName, folderName);
}
}
If you find this tutorial helpful, please check out our other tutorials – we are sure you will find other interesting stuff on this page. Feel free to ask questions in “comments” section or just show us some love by sharing this tutorial 🙂
View Comments (21)
I have added aws-java-sdk-1.9.7.jar jar in my project but still its not working. Can you please list out the jar file require to setting up the project.
Hi,
if you use Maven you can add following dependency in your .pom file
<dependency>
<groupId>com.amazonaws</groupId>
<artifactId>aws-java-sdk</artifactId>
<version>1.9.2</version>
</dependency>
If you don't use Maven you need to add all of this (you can copy the JAR-s from AWS SDK)
* aspectjrt.jar
* aspectjweaver.jar
* aws-java-sdk-1.9.0.jar
* commons-codec-1.3.jar
* commons-logging-1.1.1.jar
* freemarker-2.3.18.jar
* httpclient-4.2.3.jar
* httpcore-4.2.jar
* jackson-annotations-2.1.1.jar
* jackson-core-2.1.1.jar
* jackson-databind-2.1.1.jar
* joda-time-2.2.jar
* mail-1.4.3.jar
* spring-beans-3.0.7.jar
* spring-context-3.0.7.jar
* spring-core-3.0.7.jar
I get java.lang.ClassNotFoundException: org.apache.commons.logging.LogFactory exception on line AmazonS3 s3client = new AmazonS3Client(credentials);
Any ideas please?
I am struggling to export a file to local system in my java web application. Please help!
Hi,
are you using maven for your build or you have the jars? If you use the jars, make sure you have commons-logging in your path
Hi,
This is really helpful. was stuck with creating credentials for s3client.
Thanks
I have implemented above code and also have added all mentioned jars in my classpath.
I am getting following error: Please help
Exception in thread "main" java.lang.IllegalArgumentException
at com.amazonaws.auth.SignerFactory.createSigner(SignerFactory.java:118)
at com.amazonaws.auth.SignerFactory.lookupAndCreateSigner(SignerFactory.java:107)
at com.amazonaws.auth.SignerFactory.getSigner(SignerFactory.java:80)
at com.amazonaws.AmazonWebServiceClient.computeSignerByServiceRegion(AmazonWebServiceClient.java:311)
at com.amazonaws.AmazonWebServiceClient.computeSignerByURI(AmazonWebServiceClient.java:284)
at com.amazonaws.AmazonWebServiceClient.setEndpoint(AmazonWebServiceClient.java:160)
at com.amazonaws.services.s3.AmazonS3Client.(AmazonS3Client.java:200)
at com.amazonaws.services.s3.AmazonS3Client.(AmazonS3Client.java:182)
at com.aspireiten.util.UploadObjectSingleOperation.main(UploadObjectSingleOperation.java:26)
Exception is occurring at line
AmazonS3 s3client = new AmazonS3Client(credentials);
Hi,
if you take a look at createSigner method in here https://github.com/aws/aws-sdk-java/blob/master/aws-java-sdk-core/src/main/java/com/amazonaws/auth/SignerFactory.java
... the only chance to throw IllegalArgumentException is when the signerClass is NULL. The signerClass can be NULL only if String signerType is incorrect and can not be found among those 4:
SIGNERS.put(QUERY_STRING_SIGNER, QueryStringSigner.class);
SIGNERS.put(VERSION_THREE_SIGNER, AWS3Signer.class);
SIGNERS.put(VERSION_FOUR_SIGNER, AWS4Signer.class);
SIGNERS.put(NO_OP_SIGNER, NoOpSigner.class);
Now going back the trace down and looking at AmazonWebServiceClient class I see that AWS backand and signer type are calculated based on your credentials.
I will suggest you check again your credentials and your account with AWS. Please share your results here. Hope this helps
Hi,
very useful and well defined steps.
Thanks
I am getting below error and i am using aws-java-sdk-1.11.299,
error on AmazonS3 s3client = new AmazonS3Client(credentials);
java.lang.NoClassDefFoundError: Could not initialize class com.amazonaws.http.conn.ssl.SdkTLSSocketFactory
java.lang.NoClassDefFoundError: Could not initialize class com.amazonaws.http.conn.ssl.SdkTLSSocketFactory
at com.amazonaws.http.apache.client.impl.ApacheConnectionManagerFactory.getPreferredSocketFactory(ApacheConnectionManagerFactory.java:88)
at com.amazonaws.http.apache.client.impl.ApacheConnectionManagerFactory.create(ApacheConnectionManagerFactory.java:65)
at com.amazonaws.http.apache.client.impl.ApacheConnectionManagerFactory.create(ApacheConnectionManagerFactory.java:58)
at com.amazonaws.http.apache.client.impl.ApacheHttpClientFactory.create(ApacheHttpClientFactory.java:50)
at com.amazonaws.http.apache.client.impl.ApacheHttpClientFactory.create(ApacheHttpClientFactory.java:38)
at com.amazonaws.http.AmazonHttpClient.(AmazonHttpClient.java:314)
at com.amazonaws.http.AmazonHttpClient.(AmazonHttpClient.java:298)
at com.amazonaws.AmazonWebServiceClient.(AmazonWebServiceClient.java:165)
at com.amazonaws.services.s3.AmazonS3Client.(AmazonS3Client.java:562)
at com.amazonaws.services.s3.AmazonS3Client.(AmazonS3Client.java:542)
at com.amazonaws.services.s3.AmazonS3Client.(AmazonS3Client.java:520)
at com.amazonaws.services.s3.AmazonS3Client.(AmazonS3Client.java:487)
at com.amazonaws.services.s3.AmazonS3Client.(AmazonS3Client.java:466)
at _jsp._clzone._ImageTextDetect__jsp._jspService(clzone/ImageTextDetect.jsp:10)
at _jsp._clzone._ImageTextDetect__jsp._jspService(_ImageTextDetect__jsp.java:53)
at com.caucho.jsp.JavaPage.service(JavaPage.java:64)
at com.caucho.jsp.Page.pageservice(Page.java:548)
at com.caucho.server.dispatch.PageFilterChain.doFilter(PageFilterChain.java:194)
at com.caucho.server.webapp.WebAppFilterChain.doFilter(WebAppFilterChain.java:156)
at com.caucho.server.webapp.AccessLogFilterChain.doFilter(AccessLogFilterChain.java:95)
at com.caucho.server.dispatch.ServletInvocation.service(ServletInvocation.java:290)
at com.caucho.server.hmux.HmuxRequest.handleInvocation(HmuxRequest.java:476)
at com.caucho.server.hmux.HmuxRequest.handleRequestImpl(HmuxRequest.java:374)
at com.caucho.server.hmux.HmuxRequest.handleRequest(HmuxRequest.java:341)
at com.caucho.network.listen.TcpSocketLink.dispatchRequest(TcpSocketLink.java:1353)
at com.caucho.network.listen.TcpSocketLink.handleRequest(TcpSocketLink.java:1309)
at com.caucho.network.listen.TcpSocketLink.handleRequestsImpl(TcpSocketLink.java:1293)
at com.caucho.network.listen.TcpSocketLink.handleRequests(TcpSocketLink.java:1201)
at com.caucho.network.listen.TcpSocketLink.handleAcceptTaskImpl(TcpSocketLink.java:997)
at com.caucho.network.listen.ConnectionTask.runThread(ConnectionTask.java:117)
at com.caucho.network.listen.ConnectionTask.run(ConnectionTask.java:93)
at com.caucho.network.listen.SocketLinkThreadLauncher.handleTasks(SocketLinkThreadLauncher.java:169)
at com.caucho.network.listen.TcpSocketAcceptThread.run(TcpSocketAcceptThread.java:61)
at com.caucho.env.thread2.ResinThread2.runTasks(ResinThread2.java:173)
at com.caucho.env.thread2.ResinThread2.run(ResinThread2.java:118)
I am getting this exception :- DEBUG com.amazonaws.internal.config.InternalConfig - Configuration override awssdk_config_override.json not found.
When running this code.
the listing of the buckets doesn't show the bucket that has the zip files , need to download the zip objects and then modify it and then upload to different bucket
After putting above code to create a folder and upload a file, I am able to run the program successfully but when I logged into aws console I am able to see only the folder structure and not the file for ex. I see only A/B/C and there is no file in it.
Couldn't stop myself from commenting. Thanks a ton :). It worked great. Much needed support for starting.
hi, i am looking java code to worked with amazon s3 and amazon cloudfront CDN to deliver data to viewers.
can anyone give me idea how can i code over them.