When working with AWS S3 or S3-compatible services like MinIO, you may need to use custom endpoints instead of the default AWS endpoints. This is particularly common when you're working with a self-hosted S3 service or when you're accessing S3 services in a private network.
Here are short notes on using custom endpoints with the AWS CLI for S3 operations, specifically for MinIO:
Setting Custom Endpoints: To communicate with an S3-compatible service like MinIO, use the
--endpoint-urloption in the AWS CLI to specify the service's custom endpoint URL.
aws --endpoint-url http://minio-service.minio.svc:9000 s3 ls s3://data-science-pipelines
This command lists the contents of the
data-science-pipelines bucket on a MinIO server accessible at
- Handling Credentials: If you encounter an "Unable to locate credentials" error, configure the AWS CLI with the necessary credentials using the aws configure command. You can set the custom endpoint again and provide the access key and secret key when prompted:
Enter the MinIO access key and secret key when prompted.
No Region or Output Format: Since MinIO does not use AWS regions, you can leave the default region name and default output format blank during configuration.
Successful Configuration and Usage: After configuring the credentials, you can use the AWS CLI to access the MinIO service with the custom endpoint, as shown in the previous example.
Persistence: These settings are saved in the AWS CLI configuration files, typically located in
~/.aws/config. This means that after the initial setup, you do not need to re-enter the credentials.
Session-Based Usage: If you prefer not to save the credentials in the configuration files, you could use environment variables or pass credentials directly in the CLI commands for single sessions.
Set the Custom Endpoint in the Profile: After configuring the profile with your credentials, you can add the endpoint URL to the profile's configuration in the
~/.aws/configfile to avoid passing the endpoint URL each time in your cli.
Open the file in a text editor and add the following lines under your profile:
endpoint_url = http://minio-service.minio.svc:9000
By following these steps, you can easily interact with MinIO or any other S3-compatible storage service using the AWS CLI with custom endpoints.