Docs / Running the server

Server tools

An Onyx Voice server looks after its own operating system: its network address and firewall, its disks, Ubuntu's services, time, remote access, logs and backups. You do this from the Server pages of the console, or from the text menu on the server's own screen, without needing to know Linux. This chapter covers Server › Network (briefly), Storage, System & updates, Logs & diagnostics, and the console menu. Backup and Cluster have their own chapters: Backup and restore and High availability.

The Server pages are for system administrators. They work on a server installed from the Onyx Voice installer ISO, the VM images or the package. On anything else they show The server tools work on an Onyx Voice server installed from the ISO, the VM image or the package and nothing more.

Long tasks (updates, growing a disk, backups, restores) run as jobs on the server. The page shows their output as they run. A job keeps running if you close the page or if Onyx Voice itself restarts during it, and the last jobs are listed under Logs & diagnostics › Recent jobs.

Network

Server › Network shows the server's network interfaces with their addresses and traffic, the gateway and DNS servers, a connectivity test and the firewall. From there you can:

• Change address: switch between DHCP and a static address, IPv6, DNS servers, search domains and MTU. After Apply, open the console at the new address within 5 minutes and select Keep. If you do not, the previous configuration comes back by itself, so a wrong address cannot lock you out.
• Run the test: checks the gateway, the internet, DNS, the update servers, the server's public address (and whether it is behind NAT), and whether the server's own name resolves.
• Manage the Firewall: turn it on or off, open and close ports, or Reset to phone system ports.

A phone system needs a fixed address, because phones are provisioned to it. Everything about addresses, NAT, port forwards and the firewall is in Network, NAT and firewall.

Storage

Server › Storage shows how full the disks are, what uses the space, and how to add more. When a disk is getting full, the page and the console menu warn: The disk is getting full. When it is full, voicemail and call recordings cannot be saved.

Reading the page

• File systems: every mounted file system, how much is used and free, and a state: OK, Getting full (less than 15% free) or Almost full (less than 5% free, or less than 2 GB free on a volume of 20 GB or more).
• What uses the space: call recordings, voicemail, prompts and hold music, the database, the engine's logs, the system logs and the package cache, each with its size and folder. Voicemail and recordings live under /var/spool/onyx-voice, prompts and hold music under /var/lib/onyx-voice.

Call recordings usually grow fastest. Retention periods per tenant keep them in check: see Call recordings.

Growing the system volume

When you enlarge the server's virtual disk in Hyper-V or VMware, the extra space is not used until the volume grows into it:

1. Enlarge the virtual disk in Hyper-V or VMware. The server can keep running.
2. Open Server › Storage. The Grow the system volume card now says how much unused disk space can be added.
3. Select Grow now.

The partition, and the LVM volume when the system is on LVM (as on the Onyx Voice images), grow into the free space. The phone system keeps running; calls are not affected. If the card says There is no unused space next to it, enlarge the virtual disk first, or add a disk.

Adding a disk

You can also attach a second virtual disk to the VM. It appears on the page without a restart. It must be empty: no partitions and no file system, and at least 1 GB.

1. Attach a new, empty virtual disk to the VM.
2. Open Server › Storage. The Add a disk card lists it under Disk.
3. Under Use it, choose:
   • Add to the system volume (no downtime): the disk becomes part of the system volume, which grows by its size. Offered when the system volume is on LVM.
   • Move voicemail and call recordings onto it (calls stop while it copies): the disk gets its own file system and voicemail and recordings move onto it. Offered while voicemail and recordings do not have a disk of their own yet.
4. In Confirm, type the disk's name exactly as listed, for example /dev/sdb.
5. Select Erase and add.

Everything on the chosen disk is erased.

With the second choice, Onyx Voice and the phone engine stop while voicemail and recordings are copied: calls in progress drop and phones cannot call until the copy is done. The copy is checked file by file before anything changes; if it fails, nothing is changed and the phone system starts again. Do it outside business hours. A separate disk for recordings is a good idea when you record many calls, so a full recording disk can never stop the database or the system.

Cleaning up

The Clean up card lists what can be removed safely, with sizes:

• Downloaded update packages: installer files kept after updates.
• Old system log entries: system logs beyond the newest 200 MB.
• Rotated log files: older log files already rotated away, the engine's included.
• Old support bundles: support bundles older than a week.
• Copy kept by a restore or disk move and Database kept by a restore: the data and database as they were before a restore or a disk move (see Backup and restore). These are not ticked at first. Remove them once you are sure everything works.

Tick what you want gone and select Remove selected. The page says about how much was freed.

System & updates

Server › System & updates (the page is titled System) is about the operating system: the version, updates, time, services, remote access and restarts.

At the top are the Onyx Voice version and the engine's version, the engine's state (calls in progress and phones registered, or Stopped), how long the server has been running, the load and the memory in use. A notice appears when installed updates need a restart to finish, and when a restart or shutdown is scheduled.

Updates

The Updates card checks for and installs updates of Ubuntu and Onyx Voice, and switches the Onyx Voice update source on or off. See Updates.

Host name

The This server card shows the host name, the operating system and the time. To change the host name:

1. Type the new, fully qualified name in Host name, for example pbx.example.com.
2. Select Change host name.

This is the machine's own name. It does not change Server name on Server settings, the name phones, softphones and Let's Encrypt use, which the first-boot setup set to the same name: change both (see Changing the name). Onyx Voice restarts with the new name; the engine keeps running, so calls in progress continue. Afterwards, update the DNS records, the certificate (see Names and certificates) and any phone provisioning addresses that use the old name.

Time

Correct time matters for call records, business hours, two-factor codes and certificates. In the Time card:

• Time zone: the server's time zone. Each tenant also has its own time zone for business hours and voicemail times (see Tenants (companies)).
• Time servers: optional time servers, separated by commas, for example your domain controllers. Empty means Ubuntu's default time servers.
• Keep the clock synchronized: leave it ticked.

Select Save. The This server card shows whether the clock is synchronized or NOT synchronized. If it stays unsynchronized, the server cannot reach its time servers (UDP port 123 outbound).

Services

The Services card lists the parts of the server and whether they run: Onyx Voice (control plane, web console), the phone engine, database and configuration preparation, the database, the OS tools for the web console, the scheduled backup, SSH, the firewall, the network, the DNS resolver and time synchronization. Starts with the server says whether each one starts at boot.

Some services can be restarted with Restart:

A service shown in red is meant to run but does not. Look at its log under Logs & diagnostics before restarting it.

Remote console (SSH)

The Remote console (SSH) card turns SSH (port 22) on or off. With it off, administrators sign in on the VM's own screen only. Turning it off keeps sessions that are already open; new ones are refused. Leave it off when nobody needs it.

Restarting or shutting down

In the Restart or shut down card, choose when (in 1 minute, in 5 minutes, in 15 minutes or in 1 hour) and select Restart or Shut down, then confirm. Phones cannot call until the server is back, and calls in progress drop. A scheduled restart or shutdown shows a notice at the top of the page with Cancel it. After a shutdown, start the VM again from Hyper-V or VMware.

Logs & diagnostics

Server › Logs & diagnostics (the page is titled Diagnostics) is where you find out what went wrong.

Reading the logs

1. In the Logs card, choose the Log:
   • Control plane (Onyx Voice service): the console, portal, phone provisioning, configuration changes, intrusion prevention, certificates.
   • Engine (calls, phones, trunks): calls, registrations, trunks, refused and unrouted calls.
   • Security (failed sign-ins, blocked addresses): failed SIP and web sign-ins and the blocks they caused.
   • Database, OS tools and jobs (updates, backups, disk changes), Network, SSH sign-ins, Firewall blocks (packets the firewall dropped), Kernel, and Everything (warnings and errors).
2. Choose From the last: 1 hour, 24 hours or 7 days.
3. Choose Show: Everything, Warnings and errors or Errors only.
4. Optionally type a word in Search to see only lines containing it, for example a phone's SIP username, a phone number or an address.
5. Select Show.

The page shows the newest 500 matching lines, oldest first, with warnings marked WARN and errors ERR.

On the server itself, the engine writes its logs to /var/log/onyx-voice (engine.log, security.log and the call-centre queue_log, rotated weekly with 8 weeks kept). The control plane logs to the system journal: journalctl -u onyx-voice.

Connectivity test

The Connectivity test card runs the same test as Server › Network: gateway, internet, DNS, updates, the public address and the server's own name.

Support bundle

A support bundle is one file with what someone helping you needs: versions, disks and volumes, network addresses, routes, DNS, the firewall and the blocked addresses, listening ports, the services, the last two days of the control plane, engine and database logs and of all warnings, and the engine's current channels, endpoints, registrations and transports. Passwords and secrets are removed. The engine's generated configuration files are left out on purpose, because they hold its internal secrets.

1. In the Support bundle card, select Make a bundle.
2. Select the file name in the list to download it.

Bundles are kept on the server in /var/lib/onyx-voice/support until you select Delete; bundles older than a week show up under Storage › Clean up.

Recent jobs

Recent jobs lists the last updates, backups, restores and disk changes with their state (done, running or failed) and their last message. The full output of a job is in the OS tools and jobs log.

The console menu on the server's screen

The server has a text menu for the same tasks. It works when the web console does not, for example when the network address is wrong or Onyx Voice is not running.

To open it:

• On the server's own screen (the VM console in Hyper-V or VMware): sign in as the administrator created during setup (for example onyxadmin). The menu opens by itself. The sign-in screen also shows the web console's address.
• Over SSH: sign in and run sudo onyx-console.
• After leaving the menu you are at a shell; sudo onyx-console opens it again.

Above the menu the server shows its name and address, the Onyx Voice version, the web console's address, the engine's state, and anything that needs attention (a full disk, a service that is not running, a restart needed for updates).

The menu uses the same operations as the web console, so a change made in one shows in the other. Jobs started from the menu show their output on the screen; pressing Ctrl+C stops watching, not the job.

The onyx-os tool

Both the console menu and the Server pages use a root tool, onyx-os, which you can also run from a shell for scripts. It takes an operation name and its arguments as JSON, and prints JSON. For example:

sudo onyx-os overview
sudo onyx-os backup.run
sudo onyx-os jobs

sudo onyx-os help lists every operation. For the phone system itself (tenants, extensions, trunks and so on) use the onyx command instead: see Command line.