AWS S3 SDK Credential Auto-Detection Issue with Public Buckets (ClickHouse)

This was found in realation with ClickHouse example on clickhouse-modeling-rill-example/sources/noaa-weather.yaml.

ClickHouse fails to read from public S3 buckets with error:

1

InvalidClientTokenId: The security token included in the request is invalid

ClickHouse automatically tries to use AWS credentials (from environment variables, IAM roles, etc.) even for public buckets. When these credentials are invalid/expired, AWS SDK calls sts:GetCallerIdentity and fails.

Force anonymous access by adding NOSIGN parameter:

1
2
3
4
5

FROM s3(
    's3://bucket/path/file.csv.gz',
    NOSIGN,  -- Forces anonymous access
    'CSV'
)

In your Rill model YAML, update the s3() function call:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

# Before (fails with credential errors)
FROM s3(
    's3://noaa-ghcn-pds/csv.gz/by_year/2025.csv.gz',
    'CSV'
)

# After (works with public buckets)
FROM s3(
    's3://noaa-ghcn-pds/csv.gz/by_year/2025.csv.gz',
    NOSIGN,  -- Add this parameter
    'CSV'
)

Complete working example: GitHub - ClickHouse Rill Example

• You have AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY set in environment
  • having these set locally means it also works without NOSIGN
• Credentials are expired/invalid
• Accessing public S3 buckets that don’t require authentication
• Working across different machines/environments with different AWS setups

1
2

# Verify bucket is public
aws s3 ls s3://bucket-name/ --no-sign-request

Remember: Always use NOSIGN when accessing public S3 buckets in ClickHouse to avoid credential validation issues.

Origin: AWS S3, @ssp.sh on Bluesky