# HalaVoice / AgentLabs Application

## Database Configuration
- **Database**: PostgreSQL `agentlabs` on port 5432
- **Connection String**: stored in `.env` as `DATABASE_URL` — never hardcode credentials in docs/scripts; always read from `.env`
- **Important**: Server routing uses external IP (203.161.35.97) as source for local connections. Update `/var/lib/pgsql/data/pg_hba.conf` to include all server IPs if connection fails with "no pg_hba.conf entry"
- **Test connection**: `psql "$(grep '^DATABASE_URL=' .env | cut -d= -f2-)" -c "SELECT 1;"`

## Application Architecture
- **Type**: Single Page Application (SPA) with client-side routing
- **Main Process**: PM2 `agentlabs` on port 5003
- **WhatsApp Process**: PM2 `halavoice-whatsapp` on port 6100 (separate service)
- **Frontend**: React + Vite in `/client` directory
- **Backend**: Express.js in `/server` directory, built to `/dist/index.js`
- **Installer**: `/install` route redirects to `/login` when app is already installed (client-side redirect)

## Deployment & Services
- **PM2**: `pm2 start ecosystem.config.cjs` - manages main app and WhatsApp service
- **⚠️ AFTER any plugin change you MUST run `node scripts/build-plugin-backend.js`** (compiles `plugins/*/*.ts` → `.js` in-place and rewrites relative imports with `.js`) — otherwise `agentlabs` crashes with `ERR_MODULE_NOT_FOUND .../services/<name>.service` (plugins ship as .ts only).
- **Port 5003 (2026-08-11)**: Traccar (veedon.ai bus tracking, container `veedon-traccar`, host networking) binds protocol default ports 5001–5270+; its **Gl100 default = 5003** collided with AgentLabs. Fixed by `<entry key='gl100.port'>50090</entry>` in `/home/ashraffarid2010/traccar/traccar.xml` + `docker restart veedon-traccar`. If 5003 ever goes "not listening" again, check `ss -ltnp | grep :5003` — Traccar restart re-reads traccar.xml.
- **Nginx**: `/etc/nginx/conf.d/users/ashraffarid2010/halavoice.store.happy-times.me/agentlabs-api.conf`
  - Only proxies `/api/` and `/ws` to Node.js backend
  - Static files served via Apache cPanel backend
- **Restart app**: `pm2 restart agentlabs`
- **View logs**: `pm2 logs agentlabs --lines 50`
- **Always running**: `pm2 startup systemd -u ashraffarid2010 --hp /home/ashraffarid2010` created `pm2-ashraffarid2010.service` (enabled) which runs `pm2 resurrect` at boot from `/home/ashraffarid2010/.pm2/dump.pm2` (holds both apps). Keep the dump fresh with `pm2 save` after any app change.

## Backup, Sync & Disaster Recovery (`ops/`)
- **GitHub (SOURCE only, private)**: `github.com/NewBabyFace/halavoice-store`, branch `main`. Code only — DB/secrets/KYC/models never pushed (enforced by `.gitignore` + secret guard in `ops/git-sync.sh`).
- **Daily automatic backup**: `/etc/cron.d/halavoice-backup` runs `ops/backup.sh` at 04:00 UTC **as user ashraffarid2010** (the account owner — its gh token lives in `/home/ashraffarid2010/.config/gh`, NOT in the suspect pakchoi user) → full backup in `backups/full-backup-<TS>/` (DB dump + code tar + .env + configs, dir chmod 700, keeps newest 7) then pushes source to GitHub.
- **Scripts**: `ops/backup.sh` (full backup + git push), `ops/git-sync.sh` (push code, blocks secret leaks), `ops/setup-modules.sh` (npm install + build plugins + build on a fresh server), `ops/restore.sh` (`--from-backup` | `--from-github` | `--db-only`). Full runbook: `ops/README-DISASTER-RECOVERY.md`.
- **Restore code from GitHub, DB from a downloaded backup dump** — the two are intentionally separate (DB holds PII/KYC).
- Git ops run as **ashraffarid2010** (gh account `NewBabyFace`, credential helper via `gh auth setup-git`). `.env`, `.git`, `ops`, `backups`, `logs` are owned by ashraffarid2010; the app still runs as pakchoi=root and reads them fine. Each user that runs git here needs `git config --global --add safe.directory /home/ashraffarid2010/halavoice.store`.

## Common Issues
- **Database connection errors**: Check pg_hba.conf includes all server IPs (127.0.0.1, 127.0.0.200, 203.161.35.97, 172.17.0.1)
- **PM2 process keeps restarting**: Check logs with `pm2 logs agentlabs --err`
- **WhatsApp service crashes**: Separate service - check `pm2 logs halavoice-whatsapp`
- **⚠️ SECURITY — confirmed compromise indicators (escalate to host + rebuild)**: `ops/security-scan.sh` detects them. Found: malicious cron in `/var/spool/cron/root` + `/var/spool/cron/pakchoi` (`curl -s cavsystem.com/... | sh` daily; a `*/30` job recreating user `pakchoi` with password `Kermit123@` + writing `/etc/sudoers.d/99-pakchoi` NOPASSWD); `/root/.ssh/authorized_keys` holds ~50 keys (verify each). Run `ops/security-scan.sh` (read-only) or `--clean` (quarantines only the attacker cron lines + `99-pakchoi`, backs them up; leaves legit cPanel/AntMedia sudoers alone). A daily read-only scan runs via `/etc/cron.d/halavoice-security-scan` → `logs/security-scan.log`. Cleanup is containment, NOT remediation — the host needs credential rotation + a clean rebuild; prefer GitHub source (not host backups) for clean recovery.