SMS testing is a challenge regardless of location, device, or network- due to inaccurate reporting and regional regulations it’s almost impossible to consistently and accurately generate reports. DLR, or Delivery Reporting, is a feature of SMS message termination that reports back if the SMS message has been delivered. Typically, this message is sourced directly from the handset.

Telecoms providers can report whether an SMS has been delivered, but with no regulations in reporting, much of the data is inaccurate. To make matters more complex, many of the reports that are generated can only report if the transaction was a success or failure.

In this post, we demonstrate how we can leverage the HeadSpin platform for SMS testing in three high-volume countries: India, China, and Brazil, and discuss how each can be improved.

What is “active testing”?

Active testing means your service sends an SMS to a phone or device that you control and then checks to see if the SMS was received and what aspects of the message changed in the process. However, the scope of active testing is larger than running unit or end-to-end testing. Test SMS messages can be sent ad hoc when setting up a new service, or for detecting delivery issues on a particular carrier or within a specific country. The classic approach to “active testing” is to call your colleague in India and tell him you’re sending a message. Understandably, this approach doesn’t scale.

What does active testing look for?

Active testing is the best tool for determining if a message sent via a provider will arrive at the destination looking like it is expected to.Active testing can identifymany of the following issues:

Character encoding issues
Consolidator substitution
Concatenation behavior
Content modification
Sender ID modification
Blocking or content filtering
- URLs
- Telephone numbers
- Email address
Actual delivery (remember, DLRs can lie)
Invalid/valid phone number support

With HeadSpin, you can identify all kinds of telecom pitfalls from the user’s perspective including :

Is your Alpha-Sender ID preserved? Shortcode?
Are your US long codes working in Europe? EMEA? APAC?
Are any of your numbers re-written?
Does your app signup really work in the middle of Africa?

HeadSpin’s POP, or Point of Presence, phones allow any support or engineering team to have a physical presence across the globe.

Examples of failures identified with HeadSpin and “active testing” include:

Identifying delivery failures in China.
Customer’s improper regex phone number configuration in African countries..
Identifying carrier rewrites in numerous countries.

Active SMS Testing in India, China, and Brazil

Let’s see how a Leading CPaaS provider’s general ( CPaaS SMS ) and turnkey ( CPaaS Turnkey ) solutions work out of the box to deliver in three high-volume countries – India, China, and Brazil. India has a heavily regulated telecom system with many rules around when and what you can send. China, on the other hand, is constrained by the great firewall which could, at any time, suspend or delay messages. China is also quite draconian if you send messages with illicit content. Doing so may very well result in the suspension or loss of your SMS sending privileges. Brazil has historically had issues with carriers and Unicode making it another fun country to experiment with.

We’ll use the CPaaS Turnkey API’s default configuration to send OTPs. For India, this means it should deliver to recipients via a registered Alpha Sender ID. For Programmable SMS, we’ll simply register a US long code with CPaaS provider to send directly to the HeadSpin POP handsets.

Once we’ve found a phone to test with, the very first step is to simply text your own phone as a reality check. Honestly, it’s pretty amazing to initiate an SMS from the other side of the world and watch it show up to your own local phone within seconds. This first step just demonstrates that everything is working.

Discovery Data Points

The intent of this testing is to see the real-world SMS experience a user has on their phone in each location. We will record each data point as below:

Mobile Number: Phone numbers associated with HeadSpin device
Device: What kind of device?
Location: Where is the device located?
Carrier: Carrier the device is using
Country to US: Are we able to send an SMS from India to the device in the US
CPaaS SMS to Country: Are we able to send an SMS to the device from the US via Programmable?
CPaas Turnkey to Country: Are we able to send an SMS to the device?
Expected Body: Did the carriers process the body correctly?
Message Body : “Hello from San Francisco?!”
Long Message Body: “ The lazy brown fox jumped over the fence. The lazy brown fox jumped over the fence. ……”
Unicode Body: ★ ☆ ☇ ☈ ☉ ☊ ☋ ☌ ☍ ☏ ☐ ☒ ☓ ☚ ☛ ☜ ☞ ☟ ☡ ☤
DNA = Did Not Arrive

India Testing

A Note 5 Pro device on HeadSpin’s public cloud is depicted below. This is one of 3 devices we’re using to test telephony delivery in India. Each device uses a different carrier network and supports both automated and manual QA testing via any testing framework.

Data from Testing in India

India Conclusion

Admittedly, this is a pretty small sample size but it gives you a general idea of what kind of experience a user can have with different carriers in each location.

The most obvious issue here is the non-delivery of SMSs to India via the CPaaS SMS API. This is expected behavior as delivery to India typically requires jumping through several technological and bureaucratic hoops.

The real winner in this small sample size is the CPaaS Turnkey. Obviously, they have nailed their implementation as everything was delivered in a timely manner. The interesting observation is that out of the six attempts, it seems only one was delivered via the Alpha Sender ID registered in India. This could be a side-effect of the intelligent retry logic or simply Indian carrier mischief.

China Testing

For testing in China, we focused on two major metro areas, Shanghai and Beijing. You’ll also note that each phone used a completely different carrier. DNA denotes, did not arrive.

Data from Testing in China

China Conclusion

Testing with HeadSpin in China resulted in the complete opposite experience from India. In this case, 33% of Programmable SMS failed to arrive, and NO Verify messages sent to China arrive. Like most Voice or SMS API calls to phones terminating in China, misconfiguration or mischief by the Chinese firewall is surely to blame.

Interestingly enough, the message sent from China to the US arrived quicker than any other test.

Brazil Testing

Testing in Brazil was limited to a single carrier network. Overall, Brazil demonstrated the best performance across devices. The carrier behavior did not vary between phones and provided the fastest experience across all three test countries.

Data from Testing in Brazil

| Mobile Number | +55XXXXX | +55XXXXX | +55XXXXX |
| Device | iPhone XS | iPhone 6s | iPhone 7 |
| Location | São Paulo | São Paulo | São Paulo |
| Carrier | Vivo | Vivo | Vivo |
| India to US | No lag | No lag | No lag |
| CPaaS SMS to Brazil | No lag | No lag | No lag |
| Expected Body? | Yes | Yes | Yes |
| Unicode Issues? | No | No | No |
| Delivered By? | 27199 | 27199 | 27199 |
| Lag | No | No | No |
| CPaaS Turnkey > Brazil | No lag | No lag | No lag |
| Expected Body? | Yes | Yes | Yes |
| Unicode Issues? | No | No | No |
| Delivered By? | 27199 | 27199 | 27199 |
| Lag | No | No | No |

Brazil Conclusion

Testing in Brazil seems to have provided the best results out of the three countries. At the time of testing, there were limited Brazilian carriers to test with but on a positive note, the experience was exactly the same across each device.

Why Active Testing Matters to You!

Working with telecom is a fickle beast. When talking with customers, I frequently compare sending an SMS to sending a UDP packet. With both of these technologies, you initiate a send and the data is off! But, you never know if it has gotten to its destination. Any number of issues could affect the delivery.

Queueing (is there a football game in Munich, a cricket game in Lahore?)
Infrastructure Issues
Political Suppression
Grey Routes

Another way to think of SMS telephony is to consider hiring a container ship to deliver goods to a port of call. You can hire a cheaper company that takes the long route around South America or the more expensive company which goes through the Panama Canal. It is this route that has a far higher likelihood of arriving on time at the port of call. Your cheaper company may end up at the bottom of the sea while rounding Cape Horn.

As a mobile developer, you’ll have a clear understanding of how your mobile experience will play when leveraging HeadSpin’s robust suite of testing and performance monitoring solutions. You can take immediate control of a phone anywhere in the world with our Global Device Cloud and test to your heart’s content. Looking for a more robust long term solution? Leverage your own private cloud for automated testing or setup hourly, daily, weekly testing and benchmark reporting to identify trends and issues around your KPIs.

HeadSpin can help you automate functional, performance, and load testing across applications, devices, and networks to optimize digital experiences enabled by mobile, web, IoT, and 5G. Want to give HeadSpin a try? We have a Point of Presence in over 100 countries, 190 locations consisting of 30,000 devices. Reach out via our Contact Us form and we will coordinate a discussion and demo.

The post Active Testing SMS with HeadSpin appeared first on HeadSpin.