Updates and backups

This page covers keeping ClientOS healthy over time: updating, backing up and restoring, rotating secrets, your license, health checks, the background jobs, and fixing the most common problems.

The clientos command#

The installer adds a clientos command to the server. Run it with sudo.

Command	What it does
sudo clientos status	The version, what's running and whether it's healthy
sudo clientos version	The version running, and the newest your license includes
sudo clientos update	Backs up, downloads the newest release and restarts
sudo clientos rollback	Goes back to the release before the last update
sudo clientos backup	Database, files and settings into backups/
sudo clientos restore FILE [FILES]	Puts a backup back
sudo clientos logs [service]	Follows the logs, of everything or of one service (api, portal, admin, caddy, …)
sudo clientos restart [service]	Restarts everything, or one service
sudo clientos stop / start	Stops or starts everything. start also applies a change to deploy/.env.
sudo clientos setup-key	The setup page's key
sudo clientos env	Where your configuration file is

Updating#

sudo clientos update

That's all. It:

1. asks Vibemonk for the newest release your license's updates include, and stops if you already run it;
2. shows the release notes;
3. downloads the new release and checks every file, while the current one keeps serving;
4. takes a backup;
5. restarts on the new release. Database migrations run by themselves before the new API starts.

The release before stays on the server, so sudo clientos rollback can go back to it. Run sudo clientos status to see when the API is healthy again.

When your license's updates have ended, ClientOS keeps working on the version you have, and clientos update tells you a newer one exists.

Releases that change existing data#

While the new release starts, the old API is still serving. That's fine for most releases. A few change existing data in a way the old version can't handle, and their release notes say so. For those, run:

sudo clientos update --stop-first

The apps stop before the backup and come back on the new release: about a minute of downtime instead of the old version writing old-style data into the new database. When in doubt, use it.

Rolling back#

sudo clientos rollback

This goes back to the release from before the last update, once. A database migration the newer release ran stays applied. If the older version can't work with it, restore the backup clientos update took just before updating (see Restoring).

Never run two versions at once#

Never run a second copy of ClientOS against the same database, for example a test install pointed at your production data. Two versions over one database can corrupt it.

Backups#

sudo clientos backup

Each backup writes three files into the backups folder of your install:

File	What it holds
db-DATE.dump	Your whole database
files-DATE.tar.gz	Every file stored on the server
env-DATE	A copy of deploy/.env, including CREDENTIAL_SECRET

It keeps the newest seven of each and deletes older ones. Redis isn't backed up because it holds nothing you'd miss.

Schedule it#

Nothing runs backups for you until you schedule them. A daily backup at 03:00:

echo '0 3 * * * root /usr/local/bin/clientos backup >/var/log/clientos-backup.log 2>&1' | sudo tee /etc/cron.d/clientos-backup

Copy them off the server#

A backup on the same disk is lost with the disk. Copy the backups folder somewhere else regularly: another server, your computer, or a bucket with a tool such as rclone. The env files hold your secrets, so keep the copies private.

If you moved files to Cloudflare R2, clientos backup no longer covers them. Back the bucket up separately, for example by syncing it to a second bucket on a schedule.

Test them#

A backup you've never restored is a hope, not a backup. Restore one on a scratch server before your first real client goes in, and then every few months. Also check now and then that recent files exist in backups: a backup job that silently stopped weeks ago is the most common failure.

Restoring#

sudo clientos restore backups/db-DATE.dump backups/files-DATE.tar.gz

It asks you to type restore to confirm, because it replaces the whole database, and every stored file if you pass a files backup. The apps stop while it works and come back afterwards. Leave out the second file to restore only the database.

Onto a new server#

1. Install ClientOS on the new server with the installer, using the same domain. When it warns that DNS points at the old server, continue anyway.
2. Before the setup page, stop it: sudo clientos stop.
3. Copy your backup files to the new server. In the new deploy/.env, replace the CREDENTIAL_SECRET line with the one from the backup's env-DATE file, so the keys saved in Settings can be read. Leave every other line as the new installer wrote it.
4. Run sudo clientos start, then sudo clientos restore with the database and files backups.
5. Point DNS at the new server.

The restored database already has an owner, so the setup page doesn't open, and everyone signs in again. Your license follows the domain; if Settings → License shows a problem, activate the key there again.

Rotating secrets#

All of these live in deploy/.env. After a change, run sudo clientos start.

Secret	Effect of changing it
SESSION_SECRET	Everyone is signed out. Stored credentials are unaffected.
FORWARDED_IP_SECRET	None.
APP_DB_PASSWORD	None. The migration step updates the database login before the API starts.
CREDENTIAL_SECRET	Follow the steps below, or every key saved in Settings becomes unreadable.

Rotating the credential secret#

CREDENTIAL_SECRET encrypts every key you've saved in Settings. To change it without losing them:

1. In deploy/.env, set CREDENTIAL_SECRET_PREVIOUS to the current value.

2. Set CREDENTIAL_SECRET to a new value (openssl rand -base64 48).

3. Run sudo clientos start. The API can now read credentials sealed with either key.

4. Re-encrypt every stored credential with the new key, from the deploy folder:

   sudo docker compose exec api node dist/common/crypto/reseal-credentials.cli.js
   

   It's safe to run more than once. It exits with an error if it finds a credential that neither key can open.

5. Empty CREDENTIAL_SECRET_PREVIOUS and run sudo clientos start again.

Your next backup includes the new deploy/.env. Older backups still hold the old secret, which matches the data in those backups.

Your license#

Settings → License shows your key, the domain it's bound to and its last check-in with Vibemonk. ClientOS checks in twice a day.

• If it can't reach Vibemonk, it keeps working, and your team sees a warning saying when that grace period ends: seven days after the last successful check-in.
• After that, or at once if the key is revoked or released, the admin becomes read-only. The client portal keeps working, checkout and webhooks keep working, and nothing is deleted.

To fix it, check that the server can reach the internet, or activate a key again at Settings → License.

Moving to another domain#

Your license is bound to one domain. To move:

1. At Settings → License, choose Release this domain. The admin is read-only from this moment.
2. Point the two new addresses' DNS at the server, as in Installation.
3. In deploy/.env, change PORTAL_DOMAIN and ADMIN_DOMAIN to the new addresses, and DOMAIN to the domain both are under. Then run sudo clientos start. Caddy gets certificates for the new addresses.
4. Sign in at your new admin address, go to Settings → License and activate your key.

Everyone signs in again on the new addresses. Update any webhook or Google redirect URI you added by hand, and the lead-capture snippet on your website.

Health checks#

Endpoint	Healthy	Unhealthy
https://YOUR_PORTAL/v1/health	200, with PostgreSQL and Redis reported	503 when PostgreSQL can't be reached. Redis is reported but never makes it fail.
https://YOUR_PORTAL/healthz	200	The portal isn't answering
https://YOUR_ADMIN/healthz	200	The admin isn't answering

sudo clientos status shows the same health for each container. Point an external uptime monitor at https://YOUR_PORTAL/v1/health so you hear about problems before your clients do. Docker restarts a container whose process crashes, but not one that's running and unhealthy.

Background jobs#

ClientOS runs its scheduled work inside the API. Jobs are stored and claimed in PostgreSQL, so there's no separate worker and no cron to set up. If a job fails, it records the error and tries again later.

Job	What it does
system.heartbeat	Runs every five minutes and does nothing else, which proves the scheduler itself is running.
system.license.check	Checks your license in with Vibemonk, every 12 hours and at startup.
system.invoices.overdue	Sends reminders for invoices past their due date.
system.checkout.abandoned	Sends a nudge for checkouts that were started but never paid.
system.checkout.reconcile	Asks payment providers about recent checkouts, so a paid checkout becomes an order even if the buyer closed the tab and the webhook never arrived.
system.payments.unmatched	Alerts your team when a payment arrived without an order.
system.orders.brief_missing	Reminds clients to fill in their requirements for a new order.
system.orders.auto_approve	Accepts deliveries a client hasn't responded to after the wait you set. Only runs if you turn auto-approve on in Settings → Orders.
system.subscriptions.renew	Raises each subscription cycle's invoice and order on its renewal date.
system.subscriptions.charge	Charges saved cards for renewal invoices, and retries failed charges on your dunning schedule.
system.subscriptions.upcoming	Sends renewal reminders ahead of the renewal date.
system.cards.expiring	Warns clients about 30 days before a saved card expires.
system.chat.reminders	Sends the messaging reminders you set in Settings → Messaging, when a client or your team is waiting for a reply.
system.tickets.auto_close	Closes tickets left pending on the client for the number of days you set in Settings → Tickets.
system.tickets.unsnooze	Reopens snoozed tickets when their snooze time is up.
system.tasks.due	Notifies people about tasks coming due on the Todo board.
system.tasks.order_cards	Keeps each order's card on the Todo board in step with the order.
system.leads.follow_up	Sends the opt-in follow-up emails to new leads while they're still in the New stage.
system.referrals.accrue	Records referral commissions from payments, and adjusts them for refunds and lost chargebacks.
system.expenses.recurring	Creates recurring expenses when they come due.
system.reports.monthly	Writes last month's client reports once a new month starts.
system.notifications.stuck	Warns when notification emails are stuck and not being delivered.

Alongside the scheduled jobs, two loops run continuously: one turns events into notifications, and one delivers notification email.

Scaling#

ClientOS runs as one API on one server, and that's the supported setup. A few things live in the API's memory, such as rate limits and the live notification streams, so a second API would split them. One API on the recommended server size is plenty for a typical agency. If you outgrow it, move to a bigger server.

Troubleshooting#

The setup page doesn't open#

Or it shows a certificate warning. DNS isn't pointing at the server yet, or hasn't spread. Check from the server:

getent hosts YOUR_ADMIN

It should print the server's IP address. Caddy keeps retrying the certificate by itself; sudo clientos logs caddy shows how it's going. On Cloudflare, make sure the records are DNS only (grey cloud) for the first certificates.

The API refuses to start#

Run sudo clientos logs api. If it starts with Invalid environment:, each line names a setting in deploy/.env and what's wrong with it. The usual cause is a line someone added by hand:

Message mentions	Fix
NODE_ENV: must be production	Remove NODE_ENV from deploy/.env.
RESEND_API_KEY, R2_* or S3_ENDPOINT	Remove it. Connect Resend and R2 in the admin instead.
LICENSE_SERVER_URL or LICENSE_PUBLIC_KEY	Remove it.
DATABASE_ROLE_CHECK	Remove it.

Then run sudo clientos start.

The migration fails#

sudo clientos logs migrate shows why. If an update caused it, sudo clientos rollback brings back the previous release while you contact support.

Emails aren't sent#

ClientOS sends email only through the Resend account you connect. Until you do, sign-in codes and notification emails fail.

1. Check Settings → Integrations → Resend shows connected, not needs attention.
2. Check Settings → Emails has a sender address on a domain you verified in Resend.
3. Click Send test.
4. Check Resend's dashboard for bounces or a suspended domain.

The admin is read-only#

The license isn't active. Settings → License says why: it couldn't reach Vibemonk for seven days, or the key was revoked, released or replaced. See Your license.

Uploads fail#

With the server's own storage, check that https://YOUR_PORTAL has a certificate (sudo clientos logs caddy). With R2, the bucket is usually missing its CORS policy: copy it from Settings → Storage into Cloudflare under R2 → your bucket → Settings → CORS policy.

Everyone gets rate limited together#

If visitors get "too many attempts" errors they didn't cause, or the signed-in devices screen shows the same address for everyone, you've probably turned on Cloudflare's proxy without telling ClientOS. See Cloudflare's proxy.

The containers' network clashes with yours#

If the server loses a VPN or private network after installing, change SUBNET in deploy/.env to a free range, then run sudo clientos stop and sudo clientos start.

The setup key is rejected#

Run sudo clientos setup-key and copy it exactly. After too many wrong attempts, the form locks for a few minutes.