Skip to content

AWS S3

URI Format

s3://bucket-name/path/to/object

Credentials

CloudFS uses boto3's credential chain.

Option 1 — Environment variables:

export AWS_ACCESS_KEY_ID=...
export AWS_SECRET_ACCESS_KEY=...
export AWS_DEFAULT_REGION=ap-northeast-1

Option 2 — AWS CLI:

aws configure

Required IAM policy: AmazonS3FullAccess (or a scoped policy with s3:GetObject, s3:PutObject, s3:DeleteObject, s3:ListBucket).

Usage

from cloudfs import Path

p = Path("s3://my-bucket/data/report.csv")
p.write_text("hello")
print(p.read_text())

Caveats

Directories are virtual. S3 has no real directories — only object keys containing slashes.

  • mkdir() creates a zero-byte placeholder object (prefix/). If you delete all files under a directory but the placeholder remains, is_dir() still returns True.
  • Writing directly to a sub-path never requires a prior mkdir().
  • A path can simultaneously satisfy is_file() and is_dir() if an object named foo and another named foo/bar both exist.

rmdir() only removes placeholder objects. A "virtual" directory that was never explicitly created via mkdir() will raise FileNotFoundError from rmdir() even if is_dir() returns True.

open() buffers in memory. S3 has no native streaming file object. Read modes download the full object into memory; write modes buffer in memory and upload on close().

Performance. Each exists(), is_file(), and is_dir() call makes at least one API request. Prefer iterdir() or walk() for bulk operations.