Ceph and Litestream development snippets
I’ve been working on a project that involves Ceph and Litestream. These are some of the snippets I’ve found useful.
Bucket UI
I have deployed the Ceph cluster using rook and using ObjectBucketClaims which creates S3 buckets with their own user. While I can use s5cmd to explore the bucket, sometimes I just want to look at the bucket in a web UI. I found minio and Sree(https://github.com/cannium/Sree) as the options however, both have issues. Sree seems unmaintained and Minio has removed the UI functionality in 2022.
However, this is for local debugging, so I am OK running a very old version of Minio.
export bucket=litestream-backups
export AWS_HOST=https://ceph-rgw-objectstore.<tailnet.name> # replace with your RGW endpoint.
export AWS_ACCESS_KEY_ID=$(kubectl -n rook-ceph get secret $bucket -o jsonpath='{.data.AWS_ACCESS_KEY_ID}' | base64 --decode)
export AWS_SECRET_ACCESS_KEY=$(kubectl -n rook-ceph get secret $bucket -o jsonpath='{.data.AWS_SECRET_ACCESS_KEY}' | base64 --decode)
docker run -it --rm -p 9000:9000 -p 35501:35501 -e "MINIO_ROOT_USER=$AWS_ACCESS_KEY_ID" -e "MINIO_ROOT_PASSWORD=$AWS_SECRET_ACCESS_KEY" minio/minio:RELEASE.2022-10-24T18-35-07Z minio gateway s3 --console-address :35501 $AWS_HOST
It will print the following:
WARNING: s3 gateway is running in-memory IAM store, for persistence please configure etcd
Copyright: 2015-2022 MinIO, Inc.
License: GNU AGPLv3 <https://www.gnu.org/licenses/agpl-3.0.html>
Version: RELEASE.2022-10-24T18-35-07Z (go1.19.2 linux/arm64)
API: http://172.17.0.2:9000 http://127.0.0.1:9000
RootUser: XXXXXXX_KEY_ID_XXXXXXX
RootPass: XXXXXXX_SECRET_XXXXXXX
Console: http://172.17.0.2:35501 http://127.0.0.1:35501
RootUser: XXXXXXX_KEY_ID_XXXXXXX
RootPass: XXXXXXX_SECRET_XXXXXXX
Command-line: https://min.io/docs/minio/linux/reference/minio-mc.html#quickstart
$ mc alias set mys3 http://172.17.0.2:9000 XXXXXXX_KEY_ID_XXXXXXX XXXXXXX_SECRET_XXXXXXX
Documentation: https://min.io/docs/minio/linux/index.html
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ You are running an older version of MinIO released 2 years ago ┃
┃ Update: Run `mc admin update` ┃
┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛
You can now access the Minio UI at http://localhost:35501 and use the RootUser and RootPass to login. This will show you the bucket and the files in it.
Litestream restore
I also use a snippet that restores the backup to my local system so I can develop against the data.
export AWS_HOST=https://ceph-rgw-objectstore.<tailnet.name> # replace with your RGW endpoint.
export bucket=litestream-backups
export AWS_ACCESS_KEY_ID=$(kubectl -n rook-ceph get secret $bucket -o jsonpath='{.data.AWS_ACCESS_KEY_ID}' | base64 --decode)
export AWS_SECRET_ACCESS_KEY=$(kubectl -n rook-ceph get secret $bucket -o jsonpath='{.data.AWS_SECRET_ACCESS_KEY}' | base64 --decode)
cat << EOF > litestream.yml
dbs:
- path: /Users/goutham/go/src/github.com/gouthamve/librascan/.db/librascan.db
replicas:
- access-key-id: $AWS_ACCESS_KEY_ID
bucket: $bucket
endpoint: $AWS_HOST
path: librascan
region: us-east-1
secret-access-key: $AWS_SECRET_ACCESS_KEY
type: s3
EOF
litestream restore -config litestream.yml ./.db/librascan.db