I want to recursively get all the files in an S3 bucket folder. I want to achieve the same behaviour as the ls -R mand in Linux.

Here's an illustration

my_folder
  |_abc
  | |_abc_file.txt
  | |_xyz
  |   |_xyz_file.txt
  |
  |_file.txt

Considering the above directory structure. if I do

 const data = await s3.listObjectsV2({
    Prefix: 'my_folder/',
    Bucket: bucket,
    Delimiter: `/`,
}).promise();

Currently, I get the following data:

-CommonPrefixes: ["abc"]
-Contents:["file.txt"]

The expected behaviour is:

- Contents: ["abc/abc_file.txt", "abc/xyz/xyz_file.txt", "file.txt"]

I tried using ES6 generator functions and recursive functions to solve this problem and end up with a lot of messed up code.

I am using S3 node SDK

I want to recursively get all the files in an S3 bucket folder. I want to achieve the same behaviour as the ls -R mand in Linux.

Here's an illustration

my_folder
  |_abc
  | |_abc_file.txt
  | |_xyz
  |   |_xyz_file.txt
  |
  |_file.txt

Considering the above directory structure. if I do

 const data = await s3.listObjectsV2({
    Prefix: 'my_folder/',
    Bucket: bucket,
    Delimiter: `/`,
}).promise();

Currently, I get the following data:

-CommonPrefixes: ["abc"]
-Contents:["file.txt"]

The expected behaviour is:

- Contents: ["abc/abc_file.txt", "abc/xyz/xyz_file.txt", "file.txt"]

I tried using ES6 generator functions and recursive functions to solve this problem and end up with a lot of messed up code.

I am using S3 node SDK

• javascript
• node.js
• amazon-web-services
• amazon-s3
• ecmascript-6

• 1 S3 is a flat object store, it doesn't known any "folders" (and you cannot really create "empty folders") and there is no recursion. Your code should work. How are you actually storing this structure? What is the object stored at my_folder_abc? – Bergi Commented Jul 4, 2021 at 10:55
• ok ignore the empty folder. but still getting the files recursively is a challenge – Zainul Abideen Commented Jul 4, 2021 at 10:56
• 1 when I do listObjects() in "my_folder" then I only get "abc" not the file inside "abc" – Zainul Abideen Commented Jul 4, 2021 at 10:58
• 1 That's interesting. Normally I only see CommonPrefixes e back when a Delimiter is provided. If you do not specify a Prefix, does everything e back as expected? How about adding a slash to the end of the Prefix? – John Rotenstein Commented Jul 4, 2021 at 12:17
• 2 @ZainulAbideen Since you want all the files, not just the top level, you should just not use the delimiter parameter at all! See also realguess/2014/05/24/amazon-s3-delimiter-and-prefix – Bergi Commented Jul 4, 2021 at 14:50

1 Answer 1

Amazon S3 is a flat object storage system that does not use directories. Instead, the Key (filename) of an object includes the full path of the object. Directories magically 'appear' based on the paths of existing objects, and can later disappear when there are no objects in that path.

A CommonPrefix is the Amazon S3-equivalent of showing a sub-directory.

When calling ListObjects() with a Delimiter parameter (eg Delimiter='/'), a list of subdirectories is returned in the CommonPrefixes field. This allows recursion through directories much like traditional storage systems. If you remove this parameter, all objects with the given Prefix will be returned, rather than just the 'current directory'.