The adjustment of the GMSK modem output level is not very well documented. In my case, I carefully adjusted the Moencomm Star*Board output level to 200-300mV P-P as specified for the Friendcom FC301.
What I describe below may be very obvious to RF experts, but to mere mortals, like myself, it is a bit of a "Eureka" moment.
After the initial setup I got reasonable service out of my hotspot system with my IC-7100 and my ID-51A, however, once the ID5100 arrived I experienced the odd problem where the radio was unable to decode the DV. With a strong carrier the mode indicator on the radio was flicking between DV and FM and no voice was heard. The other effect was that data was a bit sluggish resulting in missing callsign / position info from the other station or that data only showed up some time into the QSO, even when the voice was decoding OK.
The IC-7100 and ID51A seemed mostly unaffected by this problem but the ID5100 refused to decode. Thanks to ZL1ICU, who pointed out that I should check the deviation, I started to investigate the modem output level and found that it was responsible for the somewhat pointy and steep curve seen on the spectrum analyser. I have always wondered about this difference, the output from my Icom radios had a nice and rounded "hill" shape but the output from my FC301 resembled a New Zealand mountain peak.
Adjustment of the modem output level resolved my problem. All radios are now decoding without fault and the data is displayed as soon as the carrier arrives.
The graph below shows the "before" curve in blue and the "after" curve in red - blue = problem, red = good.
The blue curve indicates a modem output level which is too low. Raising the level (R36 on Moencomm board) gives a more rounded curve and brings up the two humps either side of the main hill.
Note:
The above curve was recorded with the spectrum analyser probe located near the FC301 radio. The radio was not connected to the analyser - if you need to connect the RF output to the analyser ensure you are using an appropriately sized attenuator to avoid damaging your spectrum analyser.