Backup (preview)

Copies, drills and S3 Preview

One backup is not enough. copy keeps an independent replica up to date, drill rehearses a restore without writing files, and repositories can live in an S3-compatible bucket.

Replicas with copy#

sqz backup copy <repo> <replica> [--mirror] [--dry-run]

sqz backup copy D:\backups\laptop E:\offsite\laptop
sqz backup copy D:\backups\laptop s3://my-bucket/laptop

copy makes a replica, or brings an existing one up to date. It needs no password: it moves encrypted objects as they are, so it can run on a machine that is never trusted with the key.

It moves only what the replica is missing, in the same safe order a backup uses (data, then index, then snapshot records, then the config last), so the replica always holds either its previous state or a newer complete one.
An interrupted copy resumes where it stopped; a second run with nothing new moves nothing.
Every object is read and checked against its name before it is written, so damage in the source is reported and never copied.
Without --mirror, the replica keeps snapshots the source has since forgotten, which gives you history. With --mirror it also removes what the source no longer has, after forget and prune.
Afterwards it opens the replica and checks that it lists its snapshots. It does not re-read objects already there: run check --read-data on the replica for that.

A replica is a complete repository: you can list, restore and check it with the same password.

Restore drills#

sqz backup drill <repo> [<snapshot>] [--sample N] [--seed N] [--full]

sqz backup drill D:\backups\laptop
sqz backup drill D:\backups\laptop latest --sample 100
sqz backup drill D:\backups\laptop --full

A drill reads files back exactly as a restore would (find, decrypt, check each piece and the whole file against what was recorded) but writes nothing. By default it checks 20 random files from the latest snapshot and prints the seed, so --seed repeats the same choice. --full reads the whole snapshot and, like check --read-data, records it as the latest verified recovery point. list shows the last drill. Any failure is named per file and gives a non-zero exit code.

S3-compatible storage#

Not yet tried against a real bucket

S3 support is tested against a simulated server that checks every request signature (and against AWS's published signing examples), cuts uploads in half and injects errors. It has not yet been run against a real AWS, MinIO or other service. Start with a small repository and run check --read-data after the first backup.

Anywhere a repository folder goes, you can use s3://bucket/prefix, including both sides of copy:

sqz backup init s3://my-bucket/laptop
sqz backup run s3://my-bucket/laptop C:\Users\me\Documents

Credentials and the endpoint come from the environment:

Variable	Meaning
`AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`	Access key
`AWS_SESSION_TOKEN`	Optional, for temporary credentials
`AWS_REGION`	Region, default `us-east-1`
`SQZ_S3_ENDPOINT`	For anything that is not AWS: MinIO, Ceph, Backblaze B2, Cloudflare R2

How it behaves:

Uploads are one PUT per object, which S3 makes atomic, so a dropped connection never leaves a partial object. Finished packs are uploaded four at a time, and copy moves four objects at a time. Every pack is complete before any index or snapshot refers to it, so an interrupted run leaves the repository as consistent as before.
Retries: up to 5, with doubling backoff from 0.5 s, on connection errors, 408, 429 and 5xx. 403 and other client errors are reported at once.
Reads of one piece are ranged GETs, so restoring a file fetches its pieces, not whole packs.
Costs: after every command SQZ prints the number of GET, PUT, DELETE and LIST requests and the bytes moved, so you can estimate the bill. A small backup took 4 PUT and 3 LIST; a restore 8 GET.
Requests are path-style and SigV4-signed; TLS uses the system's trust store; HTTPS_PROXY is honoured.

Locking, versioning and Object Lock#

The writer lock uses a conditional PUT (If-None-Match: *), which AWS S3 and MinIO honour. On a service that ignores it, two backups at once still leave a consistent repository, but two prunes at once could delete what the other needs: do not prune such a bucket while anything else writes to it.

A repository never overwrites its data, index or snapshot objects; only the config, the lock and protection markers change. With bucket versioning, deletes by forget and prune become delete markers and space returns when your lifecycle rules expire old versions. With Object Lock, keep the lock and config out of the default retention, or the lock can become impossible to release; a better pattern is a locked replica bucket filled by copy without --mirror.

Not done yet#

Multipart upload and virtual-hosted-style addressing are not used. Listings are not cached on purpose: a run lists only a few times, and a stale listing could hide another machine's writes.