T.38 Fax SBC Configuration: A Setup Guide for ProSBC

The fax that “always works on the analog line” stops working the day someone replaces the PRI with a SIP trunk, and the ticket lands with the network engineer who hasn’t opened the SBC fax menu in a year. T.38 was supposed to fix this. In practice, T.38 reliability comes down to a handful of settings on the SBC, a clean tone path from the originating endpoint, and a NIC that doesn’t quietly drop packets with bad UDP checksums.

This guide walks through the exact ProSBC Fax Settings panel, the supporting NAT and codec controls, and the verification steps that turn a flaky T.38 deployment into a reliable one. For the conceptual background on what T.38 is, how UDPTL differs from RTP, and why fax breaks on VoIP networks in the first place, the Fax over IP (T.38) explainer is the companion piece to read first.

Pre-flight checklist before you change a single setting

Most “T.38 doesn’t work” tickets resolve to one of three problems that no SBC setting will fix: an endpoint that cannot do T.38, a tone that never makes it to the SBC, or a NIC quietly dropping packets. Confirm these before touching the Fax Settings panel.

Both endpoints can actually negotiate T.38 or G.711 passthrough. An ATA, a fax server, or a PRI gateway that only supports G.711 fax will never accept a T.38 re-INVITE, no matter how the SBC is configured. Confirm support on both legs in the device documentation before debugging the middle.

The fax path is identified end to end. Know which NAP the fax originates from, which NAP terminates it, the carrier in between, and whether re-INVITE messages traverse any firewall or NAT device that might rewrite SDP. SIP header manipulation further upstream can strip or rewrite attributes ProSBC needs to see intact.

NIC offloads are off if you have ever seen UDP checksum errors. Generic Receive Offload and UDP checksum offload on the SBC NIC are a known T.38 killer. Tickets land with the symptom “T.38 negotiates, then the page never completes.” The fix is at the OS layer (disable the relevant offloads with ethtool), not in the Fax Settings panel.

A test fax number that fails the same way every time. Intermittent failures produce intermittent traces. A reproducible test call from a known fax machine to a known destination, ideally with a single-page document, is what makes a tbsigtrace capture useful.

Where ProSBC’s fax settings actually live

Fax behavior on ProSBC is configured per NAP profile, not globally. The Web Portal path is Configuration By Web Portal Category → NAP Profiles → Fax Settings, and inside that panel five distinct mode pages cover the operational variants you might need.

• Configure Fax T38 for end-to-end T.38 relay between two endpoints that both negotiate it.
• Configure Fax Passthrough for G.711 fax-from-analog kept in audio from end to end.
• Configuring Fax Relay for the internal fax/modem relay mode used in gateway-style deployments.
• Configure Fax NSE for Cisco Named Signaling Events interop, where the far side expects NSE-based fax/modem signaling.
• Configure Fax VBD for voice-band-data passthrough used in some legacy environments.

Profiles are named (a common convention seen in real deployments is something like FAX_ISDN for the historical PRI-side profile), and the same profile can be attached to multiple NAPs. If you re-key a NAP from PRI to a SIP endpoint, the same fax profile follows it. This is the recommended pattern: configure once, attach as needed.

Choosing the right mode

The single biggest cause of fax failures is choosing the wrong mode for the endpoints involved. The decision is shaped by what each leg supports, not by what the SBC prefers.

Fax T38 is the right choice when both legs can negotiate T.38. The SBC detects the fax tone, sends a re-INVITE, and switches the media leg to UDPTL. This is the most resilient option over lossy networks because UDPTL redundancy reconstructs lost packets locally without retransmission.

Fax Passthrough applies when both legs prefer G.711 and the network between them is reliable. No re-INVITE is issued, no transcoding is required, and the fax modems negotiate over an unbroken G.711 audio stream. Passthrough is sensitive to packet loss and jitter; fax passthrough becomes increasingly unreliable as packet loss and jitter increase.

Fax NSE and Fax VBD exist for specific vendor-interop cases. Use Fax NSE when the far side is a Cisco gateway expecting NSE signaling. Use Fax VBD when the deployment specifies voice-band-data behavior on a particular trunk. If neither vendor nor specification calls for these modes, leave them aside.

Fax Relay is the internal fax/modem relay mode used in gateway-style topologies where ProSBC itself acts as the relay endpoint rather than negotiating with a downstream gateway. This is the less common scenario in modern SBC deployments, but it remains documented for legacy migrations.

Configuring Fax T38 field by field

The Configure Fax T38 page exposes a compact set of fields, and the same fields appear on customer support exchanges often enough that their function is well understood. The list below documents each field with the safe default and the reason to deviate from it.

Enable Fax/Modem Relay is the master toggle. It must be checked for any T.38 behavior to take effect. If this is unchecked, every other field in the panel is inert.

Detection Mode controls how aggressively ProSBC listens for fax tones in the audio stream. Standard is the documented default and the value seen in working deployments; reach for an alternative only if a vendor explicitly requires it.

Relay Mode selects between T38 (re-INVITE to UDPTL) and Passthrough (keep the leg in G.711 throughout). This is the single field that switches operational mode for the profile. Set it to T38 for the Fax T38 walkthrough; flip to Passthrough when you genuinely want the audio leg preserved end to end.

Modem vs Fax Distinction takes a millisecond threshold and controls how the detector distinguishes a fax tone from a modem tone. A value of 0 ms treats all detected tones as fax, which is the right starting point for fax-only deployments. Raise it only when you have modem traffic on the same trunk and need ProSBC to route the two differently.

Prevent direct invite in T.38 handles the case where the originating side opens the call with T.38 already in its initial SDP rather than starting in audio and re-INVITEing. Check this box when the far end can only handle the audio-first sequence and you have an upstream gateway that always emits direct-T.38 INVITEs. Leave it unchecked when both sides handle direct T.38 cleanly.

Detection Type covers detector tuning. “Silence suppression off” is the documented safe default; silence suppression interacts poorly with fax tones and is the source of more failure tickets than it solves.

Codec selects the audio-leg codec used before the re-INVITE. PCMU (G.711 µ-law) is the documented default and the codec that working deployments converge on. Set this to match the carrier-facing codec on your audio leg; do not list G.729 here, since fax cannot survive the lossy compression.

T.38 SDP attributes you will see on the wire

When the re-INVITE fires and the media leg switches to UDPTL, the SDP body advertises a specific set of T.38 attributes. Knowing the defaults makes tracing the call flow much faster, because every value below is something you can search for in a capture and confirm against expectation.

• m=image udptl t38 is the media line that signals the switch from audio RTP to UDPTL.
• a=T38FaxVersion:0 is the version offered by default. Most fax endpoints negotiate version 0 successfully; version overrides exist but are rarely required.
• a=T38MaxBitRate:14400 is the maximum modem bit rate offered, matching standard V.17 fax.
• a=T38FaxRateManagement:transferredTCF places the Training Check Frame management at the far end rather than locally.
• a=T38FaxMaxBuffer:200 and a=T38FaxMaxDatagram:200 describe the buffer and datagram sizes offered.
• a=T38FaxUdpEC:t38UDPRedundancy declares UDPTL redundancy as the error-correction method, the option that compensates for packet loss by carrying copies of previous IFP packets.

Capture a known-good fax call to your own deployment first to confirm exactly what ProSBC offers in your environment. Carrier-side normalization can rewrite any of these values before they leave the SBC, and the values you see on the carrier-facing leg may differ from the values you see on the endpoint-facing leg.

NAT, Force Passive Mode, and the trap most people hit

The NAP-level NAT configuration interacts with T.38 in a way that catches many deployments. The relevant setting is Remote Method for RTP on the NAP NAT configuration page, and one of its values, Force Passive Mode, changes when ProSBC opens the RTP port.

With Force Passive Mode set, ProSBC waits for the first RTP packet to arrive from the far end before opening its RTP port. The behavior is correct for NAT-traversal scenarios where the far side is behind a stateful firewall and ProSBC must learn the externally translated address from observed traffic. The downside is that if the far side never sends first, the port never opens, and the T.38 negotiation appears to succeed at the SIP layer but no UDPTL packets ever flow.

The diagnostic is straightforward: a tbsigtrace capture that shows the re-INVITE and 200 OK exchange completing cleanly, no UDPTL packets in either direction, and the call eventually timing out. The fix depends on the topology. Where ProSBC actually faces a NATed peer, Force Passive Mode is the correct choice and the originating side needs to be configured to send the first packet. Where ProSBC faces a non-NATed gateway, Force Passive Mode is the wrong choice and the Remote Method should be set to a value that lets ProSBC open the port immediately.

When G.711 passthrough is the right answer

Fax Passthrough is not a fallback for a failing T.38 deployment; it is a different choice with its own trade-offs. The conditions that favor passthrough are specific and worth checking before flipping the Relay Mode.

Choose passthrough when both endpoints prefer the G.711 fax-from-analog behavior, when the network between them shows packet loss reliably below 1%, when the carrier in the middle strips T.38 SDP or always offers G.711, or when one end is an analog ATA that does not implement T.38 at all. The configuration is in the same Fax Settings panel; the change is just Relay Mode set to Passthrough rather than T38, with the audio-leg codec held at PCMU and silence suppression off.

The trade-off is sensitivity to packet loss. T.38 with UDPTL redundancy reconstructs missing packets without retransmission. G.711 passthrough does not: lost samples become audio dropouts, and a fax modem trying to train through an audio dropout will simply fail. If your transport occasionally drops packets, T.38 will degrade gracefully where passthrough will not. The G.711 versus G.729 codec comparison covers the broader codec context if you are weighing other audio paths on the same trunk.

Verifying the fax call

The verification recipe TelcoBridges support requests when troubleshooting fax issues is the same procedure you should run yourself before opening a ticket. It produces the artifacts that prove what actually happened on the wire.

1. Raise trace levels on the relevant processesUse tbx_cli_tools_remote to set the trace level to 1 on gateway, toolpack_engine, and tbsyslog. Type T then 1 at the prompt for each. Trace level 1 is enough for fax negotiation without flooding the logs.
2. Start a tbrouter packet captureThe capture should cover both NAPs involved in the fax path. Confirm the capture is running before placing the call.
3. Place the failing fax callUse the reproducible test number identified during the pre-flight checklist. A single-page document is sufficient and produces a cleaner trace than a multi-page transmission.
4. Stop the capture and export the call traceStop tbrouter, export the call trace from the Web Portal, and generate a tbreport scoped to the date and time window of the failing call.
5. Inspect what the trace actually showsThe full pattern is: 200 OK on initial audio INVITE, fax tone detected, re-INVITE with T.38 SDP, 200 OK on the re-INVITE with matching T.38 attributes, the m-line switching to m=image udptl t38, CNG arriving before the first DIS frame, and IFP packets flowing in both directions with the UDPTL redundancy structure intact.

A trace that stops short of one of these milestones tells you exactly where to look next. No re-INVITE means the audio-leg detector never fired. A re-INVITE without UDPTL traffic means a NAT or RTP-port issue, with Force Passive Mode at the top of the suspect list. UDPTL flowing but pages failing usually points to UDP checksum errors on the NIC or CNG arriving after the DIS frame.

Common fault patterns and how to fix them

Most production fax failures match one of a small number of patterns. The list below is drawn from support resolutions on real ProSBC deployments and maps each pattern to the fix.

UDP checksum errors on the SBC NIC are the most common fax-killer. The symptom is T.38 negotiation completing cleanly followed by IFP packets that the kernel drops before the application sees them. The fix is at the OS layer: disable UDP checksum offload and Generic Receive Offload on the interface. The TelcoBridges SBC Troubleshooting documentation documents the steps; the same fix that resolves DTMF problems also resolves T.38 packet loss.

CNG arriving after DIS means the originating side sent the call setup faster than its tone, and ProSBC’s detector either missed the CNG entirely or detected it too late to trigger the re-INVITE. ProSBC will not fabricate a CNG; the fix is on the originating endpoint or the upstream gateway, often by tuning that device’s tone-generation timing or by switching its fax mode.

T.38 packets still appearing when you wanted passthrough indicates that the Relay Mode field is still set to T38. Switching the field from T38 to Passthrough suppresses the re-INVITE and keeps the leg in G.711.

A Diversion header inserted by a routing script breaks far-end acceptance. Several ProSBC deployments use Ruby scripts for STIR/SHAKEN dipping (the ClearIP script is one example). Older versions of the script inject a Diversion header that some fax-handling gateways reject. The fix is to update to the current script; TelcoBridges support has shipped corrected versions for the ClearIP path, and the same principle applies to any custom routing script that touches the request line on fax calls.

Stalled RTP after re-INVITE traces back to the NAP NAT configuration. Force Passive Mode requires the far end to send the first RTP packet; if it does not, no RTP flows in either direction. The fix is either to align Force Passive Mode with the actual NAT topology or to ensure the far end is configured to initiate RTP.

The SIP layer reports a MEGACO Error 442 (syntax error in command) in internal traces. This indicates a malformed SDP, often a stray space or character between attributes, on the path between the SBC’s media gateway controller and its media gateway. It is an internal symptom rather than a customer-facing fault, but if it appears in support traces alongside a failed fax, the SDP needs to be examined and the source of the malformation identified.

Vendor-specific notes

The ProSBC documentation library includes a dedicated configuration page for 3CX as a receiver of T.38 faxes (“Configuration for 3CX PBX Server with the ProSBC to receive T38 Faxes” under the 3CX provisioning section). The page is the right reference when 3CX is on the receiving side of a fax path; the configuration there interlocks with the Fax T38 settings on the ProSBC side.

For other PBX integrations (Asterisk, FreePBX, FreeSWITCH, FusionPBX, VitalPBX, Yeastar, Wildix, Brekeke, Sippy, Cisco UCM, Avaya IP Office, VoIP.ms), the per-vendor integration pages in the ProSBC documentation are the authoritative source. Fax behavior interacts with each PBX’s own fax handling, and the safest path is the documented integration pattern for that specific vendor rather than a generic “T.38 should just work” assumption.

Frequently Asked Questions

Conclusion

Reliable T.38 fax on an SBC reduces to four things done right. The right mode for what each endpoint actually supports, with Fax T38 as the default and Passthrough as the considered alternative. A clean tone path so CNG arrives at ProSBC before the DIS frame. A NAT configuration that does not strand the RTP port behind a Force Passive Mode mismatch. And a NIC with UDP checksum offload disabled, so the kernel does not silently drop the packets the application is trying to receive.

The ProSBC Fax Settings panel exposes the controls; tbsigtrace and tbreport provide the proof. Once those line up against a known-good trace, T.38 stops being a flaky feature and starts being part of the standard SIP-trunking baseline.