Docs / Running the server

Troubleshooting

This chapter goes through the problems that come up most when setting up and running a phone system: phones that do not register, calls without audio, calls from the carrier that do not arrive, outside calls that are refused, desk phones that do not set themselves up, and certificate trouble. It ends with where to look: the logs, the support bundle, onyx engine status and the health check.

First checks

Before looking at one phone or one call, check that the phone system itself runs:

• Live calls: if the page shows The call engine is not connected., the engine is down or restarting. Look at Server › System & updates › Services: Phone engine (calls, phones, trunks) should be active.
• Dashboard: the Registration log shows phones and trunks signing on and off, newest first.
• Server › Storage: a full disk stops voicemail and recordings, and can stop the database.

Phones do not register

A phone that does not register shows Offline on its extension page (Extensions & numbers, the Phones card) and is missing from Live calls › Phones online. Work through these in order.

Wrong username or password

Every device has its own generated SIP username, which is not the extension number, and a 24-character password. On the extension's page, select Show settings on the device to see the username, password, server and port to enter in the phone. A desk phone added under Desk phones gets them by itself (see the provisioning section below).

If the phone has the wrong ones, it appears under Security › Recent failed sign-ins with its address, and What shows the failure and the username it tried. Correct the phone. After New password, a phone is signed out until it has the new password.

The address is blocked

A phone that keeps failing gets its address blocked by intrusion prevention: by default after 10 failures within 10 minutes, for 60 minutes. Every other phone behind the same public address is blocked with it, so a whole remote office can go offline because of one phone.

1. Open Security and look at Blocked now.
2. Fix the phone that failed first (see above), or it gets blocked again.
3. Select Unblock on the address.
4. Add the office's public address to Never block under When to block, and select Save.

See Security.

The firewall or the router blocks SIP

The phone must reach the server on the SIP port: 5060 over UDP or TCP, or 5061 over TCP for encrypted calls. On Server › Network, the Firewall card warns when the phone system's ports are closed, for example SIP port 5060 is closed for UDP and TCP. Reset to phone system ports opens them all again.

Phones outside the office reach the server through your router: SIP and the audio ports must be forwarded to the server, and Public address must be set. See Network, NAT and firewall.

Encrypted calls set on one side only

A device set to Encrypt calls should register over TLS on port 5061 and send encrypted audio. If the phone is still set for plain SIP, it may register, but its calls fail, because the server accepts only encrypted audio from it. A phone set for TLS cannot register while port 5061 is closed on the firewall. Set both sides the same. Many desk phones also refuse a self-signed certificate for TLS: see the certificate section below.

One-way audio or no audio

Calls connect, but one side or both hear nothing. Audio (RTP) travels separately from the call set-up, on UDP ports 10000 to 20000 by default, and it is what NAT and firewalls break.

Signs to look for: calls that hang up by themselves after about a minute without audio. The engine ends a phone's call when no audio has arrived from it for 60 seconds (120 seconds on a trunk), so a call with no audio in one direction ends after a minute.

Check, in this order:

1. The audio ports are open. On Server › Network, the Firewall card warns The RTP media ports 10000:20000/udp are not all open: calls connect without audio or with one-way audio when they are not. If the server is behind a router, forward the same UDP range from the router to the server for phones and carriers outside.
2. The public address is set, if the server is behind NAT. Run the test under Server › Network › Connectivity test. If the Public address line says the server is behind NAT, open Server settings and set Public address (engine.external_address) to the public address shown, for example 203.0.113.10. Without it, the server tells outside phones and carriers to send audio to its private address, which they cannot reach.
3. Local networks list every network reached without NAT. Local networks (engine.local_nets, in Server settings) lists the networks the server reaches directly. By default these are the three private address ranges. For an address on a local network the server uses its own address; for everything else it uses the public address. If a remote office reaches the server over a VPN with public-looking addresses, or a network is missing from the list, its phones get the wrong address and have one-way audio. Add the network, for example 10.0.0.0/24, 172.20.0.0/16.
4. SIP ALG is off on the routers. Many routers and firewalls have a "SIP ALG" or "SIP helper" that rewrites SIP messages, and it often breaks audio and registrations with a phone system that handles NAT itself. Turn it off on the router at the server's site and at remote sites.
5. The audio port range matches the forwarding. If you change Audio ports from and Audio ports to in Server settings, change the firewall rules and the router's forwarding to match.

Changes to these settings reach the engine within a second; network changes may restart it once no calls are up. Details: Network, NAT and firewall.

Calls from the carrier do not arrive

People call your number and hear an error, or nothing rings.

Is the trunk up?

On Trunks (carriers), the Status column shows Up (with the round-trip time), Not answering, Checking, Not checked (checks are off for this trunk) or Switched off. From a shell, sudo onyx trunk status shows the engine's registrations to carriers and whether each carrier answers.

• Registration trunks: the trunk must register with the carrier for calls to reach you. A wrong username, auth username or password shows in the trunk's registration state and in the engine log. After the carrier refuses a registration, the engine waits 10 minutes before it tries again.
• IP authentication trunks: the carrier sends calls from its own servers, and Onyx Voice only accepts them from the address in Carrier server and the addresses in Other carrier addresses. If the carrier sends from other addresses, calls are turned away as from an unknown sender, and that address can even be blocked by intrusion prevention. Add every address range the carrier publishes to Other carrier addresses, and to Never block on the Security page.

See Trunks (carriers).

Does the number match a phone number?

An incoming call is matched against Phone numbers (DIDs) by its number. Onyx Voice stores and matches numbers as digits only; in North America a 10-digit number gets its leading 1, so (612) 555-0100, +16125550100 and 16125550100 are the same number.

A call for a number that no phone number entry matches is hung up, and the engine log shows a line like Onyx: unrouted DID 16125550100 on ... from .... Search for unrouted DID in the Engine log under Server › Logs & diagnostics. Then:

• Compare the number in the log with the numbers on Phone numbers (DIDs). A carrier that sends only the last digits, or something else entirely, needs the phone number entered the way it arrives.
• Check the trunk's Phone number is in setting. Automatic reads the number from the To header on registration trunks and from the Request URI on IP-authenticated trunks. Some carriers put it in the other place: choose Request URI or To header to match.
• A phone number tied to one trunk only matches calls on that trunk.

Is the trunk full?

If the trunk has Maximum simultaneous calls set and that many calls are up, further incoming calls are refused and the engine log says the trunk is at its channel limit.

Outside calls are refused

Someone dials an outside number and hears an announcement or a fast busy tone.

• "The number is not in service", at once: no outbound route of the tenant matches what was dialled. Check the dial patterns on Outbound routes: with a prefix such as 9, the person must dial it. See Phone numbers and routes. The same announcement plays when the tenant's outside calls are stopped by a call limit (next point).
• Call limits: international calls, premium numbers and too many calls at once are refused by the tenant's limits, and a crossed hourly or daily limit stops all of its outside calls. New tenants cannot call abroad at all. Look at Refused calls on the Call limits page or run onyx toll refused; it shows the extension, the number and why. See Call limits.
• "All circuits are busy": every trunk of the route failed, was at its call limit or did not answer the checks, or the tenant reached Outside calls at once. Check the trunks' Status.
• The carrier refuses the call: the call reaches the carrier, which rejects it, often because of the number format or the caller ID. The engine log shows the carrier's answer. Check the trunk's Send numbers as, Send caller ID as, Caller ID header and Dial prefix against what the carrier wants.

Desk phones do not set themselves up

A desk phone fetches its settings from the server's provisioning address (shown on Desk phones). When it does not get them:

• The phone's address is not allowed. Phone settings files contain SIP passwords, so by default only addresses on the Local networks may fetch them. A phone elsewhere gets an error, and the Control plane log shows Provisioning request for acme from ... refused. For remote phones, set Phones may fetch settings from (provisioning.allowed_nets) in Server settings to their networks, or any.
• The phone is not added, or under another MAC. A phone that asks for its settings but is not assigned to an extension appears under Phones waiting to be added on Desk phones, with its MAC and address, and gets nothing. Select Assign. If you added it by hand, check the MAC on the label under the phone.
• The model is unknown. The phone's model decides which settings file it gets. If the model is not one Onyx Voice knows and the make cannot be told from the phone's own request either, the phone gets nothing. Set the model on the phone's entry.
• The phone has an old address. After New secret address, every phone must be pointed at the new address.
• HTTPS with a self-signed certificate. The HTTPS provisioning address works only for phones that trust the server's certificate. Use the HTTP address, or get a trusted certificate.
• The Status column helps: Never fetched settings means the phone never reached the server; Configured, offline means it fetched its settings but is not registered (see the registration section above); Last settings fetch shows when and from which address.

Plug-and-play (new phones on the local network finding the server by themselves) needs UDP port 5062 open; the Firewall card warns when it is closed. DHCP option 66, or typing the address into the phone, works without it. See Desk phones.

Certificate problems

The Certificate card on the Security page shows the certificate in use, its names and when it runs out.

• Browsers warn about the console, and phones refuse encrypted calls: the certificate is Self-signed (made by this server). Get a Let's Encrypt certificate (below) or install your own.
• Let's Encrypt is not possible yet: the card says why. Let's Encrypt needs Server name (engine.hostname, in Server settings) to be a fully qualified public name such as pbx.example.com (not an IP address, and not a private name ending in .local, .lan or .internal), a DNS record pointing that name at the server, and port 80 open to it from the internet.
• A request failed: the card shows the last attempt and the error. Usually the DNS record points elsewhere, or port 80 does not reach the server.
• Encrypted desk phones do not register after getting a trusted certificate: with a trusted certificate, encrypted phones connect to the server by its name. The phones must be able to resolve that name, to the server's local address when they are on the LAN.

Certificates from Let's Encrypt renew by themselves 30 days before they run out. See Names and certificates.

Where to look

Logs

Server › Logs & diagnostics reads every log of the server, with filters for time, level and a search word. The most useful:

On the server, the engine's log files are in /var/log/onyx-voice (engine.log, security.log); the control plane logs to the system journal (journalctl -u onyx-voice). See Server tools.

The support bundle

When you ask for help, send a support bundle: Server › Logs & diagnostics › Support bundle › Make a bundle, then download the file. It holds versions, disks, network settings, the firewall, the services, the last two days of logs, and the engine's current registrations and endpoints, with passwords removed.

onyx engine status

From a shell on the server:

sudo onyx engine status

It shows the engine's version, how long it has been running, every registered phone and trunk contact, and the number of active calls. If it answers cannot reach the engine, the engine is not running: check Services, or systemctl status onyx-voice-engine.

sudo onyx engine cli "pjsip show endpoints" runs any engine console command, for example to see one device's state.

The health check

https://pbx.example.com/healthz (or http:// with the server's address) answers without signing in:

{"status":"ok","engine":"connected"}

"engine":"disconnected" means the service runs but the engine does not. No answer at all means Onyx Voice itself is not running or not reachable. Monitoring tools can poll this address. Use it rather than sending SIP requests to the server many times a minute, which intrusion prevention may count as failures.