Can we help AT&T solve its mobile data problem?

I'm in midtown Manhattan, connected to the Net over my hotel's slow but costly wi-fi connection. Normally when I'm traveling — at least here in the U.S. — I avoid lame hotel connections by using AT&T's cellular data system, usually through my iPhone's "personal hotspot."

But that doesn't work here, except in the wee hours, I assume because demand on the system is lower. But I don't know. Maybe you do. If so, perhaps this fodder will stoke the problem-solving fires:

PING google.com (74.125.225.115): 56 data bytes
64 bytes from 74.125.225.115: icmp_seq=0 ttl=51 time=101.064 ms
64 bytes from 74.125.225.115: icmp_seq=1 ttl=51 time=92.423 ms
Request timeout for icmp_seq 2
(snip)
Request timeout for icmp_seq 32
64 bytes from 74.125.225.115: icmp_seq=2 ttl=51 time=31309.253 ms
64 bytes from 74.125.225.115: icmp_seq=3 ttl=51 time=30364.809 ms
64 bytes from 74.125.225.115: icmp_seq=5 ttl=51 time=28366.889 ms
64 bytes from 74.125.225.115: icmp_seq=7 ttl=51 time=26370.460 ms
64 bytes from 74.125.225.115: icmp_seq=10 ttl=51 time=23369.719 ms
64 bytes from 74.125.225.115: icmp_seq=12 ttl=51 time=21384.230 ms
64 bytes from 74.125.225.115: icmp_seq=14 ttl=51 time=19385.376 ms
64 bytes from 74.125.225.115: icmp_seq=4 ttl=51 time=29390.279 ms
64 bytes from 74.125.225.115: icmp_seq=6 ttl=51 time=27393.178 ms
64 bytes from 74.125.225.115: icmp_seq=9 ttl=51 time=24401.894 ms
64 bytes from 74.125.225.115: icmp_seq=11 ttl=51 time=22405.324 ms
64 bytes from 74.125.225.115: icmp_seq=13 ttl=51 time=20404.648 ms
64 bytes from 74.125.225.115: icmp_seq=15 ttl=51 time=18448.794 ms
64 bytes from 74.125.225.115: icmp_seq=34 ttl=51 time=453.465 ms
Request timeout for icmp_seq 47
(snip)
Request timeout for icmp_seq 58
64 bytes from 74.125.225.115: icmp_seq=35 ttl=51 time=24054.439 ms
Request timeout for icmp_seq 60
(snip)
Request timeout for icmp_seq 87
^C
--- google.com ping statistics ---
89 packets transmitted, 17 packets received, 80.9% packet loss
round-trip min/avg/max/stddev = 92.423/20452.720/31309.253/10051.146 ms

There are two ways we can go with this information. One is to give crap to AT&T, or to me for using AT&T (and for using an iPhone... I also have an Android, btw & fwiw) — or to the futilities of trying to do anything serious over something so commercial and klugey as a cellular data system. The other is to help AT&T with a problem it clearly has, as technical folk. If we can.

That's the appeal here. What's going wrong? Inadequate provisioning of capacity? Bufferbloat? Something else?

There's another issue I want to explore with this exercise, and that's opening companies to help from the customer/user side. Companies like AT&T aren't set up for that. They're organized to heal themselves from the inside.This excludes more sources of help than it includes, especially when the problems are technical and there are technical people on the outside who have perspective and expertise, and can provide useful assistance.

It's easy to be cynical about the prospects of companies opening up to real outside help. It's harder to try breaking them open. But that's what I'm after in this case.

We've got a lot of technical readers here. Lots of those readers have mobile phones. Probably more than a few have the same problems (and not just with AT&T) that my phone is experiencing here. Why not help out?

Here's another factor to consider: it's still early. We've hardly begun to build out the infrastructure for what Bob Frankston calls "ambient connectivity." Chances are, once we have ambient connectivity, cellular telephony will not be what most of us are using at the lower layers of the stack. But we'll get to ambient connectivity (and nearer milestones) faster, methinks, if we help work kinks out of the systems we do have today, and mobile data over cellular connections is one of those systems.

______________________

Doc Searls is Senior Editor of Linux Journal

Comments

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.

Perhaps instead of trying to

Anonymous's picture

Perhaps instead of trying to help the big, monopolistic telecomm corporations, we should try instead to help small companies or new startups to solve these problems?

I realize there is a tendency to want to just get on with our lives as best we can and ignore some of the bigger issues, but I think in cases like this we might be better off letting ATT continue to shoot itself in the foot in the hopes that enough people will become angry that something will actually be done about this. I want to see the Bell System broken up again, and to make it stick this time. We've allowed ourselves to become serfs in a society ruled by large corporations.

Who we should help

Doc Searls's picture

Good point. One startup worth looking at is Ting. Still in beta. Run by the Tucows folks, and old friends of Linux Journal. If any of you are interested, I can make introductions.

Doc Searls is Senior Editor of Linux Journal

try a different tool...

Ron Lockard's picture

Doc,

Next time try a tool called mtr (available in most distro repos) rather than just ping. Think of it as a combination of traceroute and ping. It will show latency and also packet loss over time at each routing hop as it runs. It would be interesting to actually see where the issue in the path really occurs and that should show it.

try a different tool...

Doc Searls's picture

Thanks, Ron. On the road again right now, but will do.

For the rest of ya'll, here's Wikipedia on mtr.

Doc Searls is Senior Editor of Linux Journal

Opening Companies

Larry's picture

Doc, I've often thought about encouraging companies to use the talent in its customer base. The main obstacle is how to qualify the customer's expertise. I know from my work that users think they know the answer but, more often than not, don't. On the other hand, if we do have useful insight, how do you get past tier 1 support? I've often had conversations with comp sci faculty (I've mostly worked in higher ed) about this issue. When they need support it's rarely tier 1. On smaller campuses, ad hoc support channels develop. On larger campuses, this is more difficult. Expand this to a company the size of AT&T, remove obvious credentials and you can see the challenge.

I suppose adding intelligence to the support portal could help. Allowing the user to self-score their knowledge/skills might be a first pass. From there, we could assign a score to support calls: tier-level of the solution, value of user's info/insights, helpfulness of the user. Based on the cumulative score, perhaps coupled with a description of the problem, the support portal could direct you a more appropriate support tier.

Another possibility is creating an advanced users group. This group could have a different support system, bypassing the usual tier 1 channel. The company could also draw upon this group to gather intel. on problems reported by others, try out solutions and/or test new services.

Opening Companies

Doc Searls's picture

A couple years ago I had a problem at our house in Santa Barbara with Cox, the only choice for Internet connectivity there. You can read the saga here: http://blogs.law.harvard.edu/doc/tag/cox/

It ended with a good relationship between me and one senior Cox guy in Santa Barbara. But, I doubt any institutional change happened at all. But your ideas are good ones. Thanks.

Doc Searls is Senior Editor of Linux Journal

European/Asian type network

sanchiro's picture

Will we ever have a network for cellular in the US where we focus on improvement to services and hardware, as they have in Asia and Europe, instead of this fragmented system split between GSM/CDMA?

Well, that is good evidence,

Howard Greenstein's picture

Well, that is good evidence, unless the buildings move at night when we're not looking. I quit AT&T a while ago - would love to see them solve this, might bring people back.

Probably not but good luck

tacra's picture

I thought long and hard about how to answer this. I decided to give a short answer and a long answer. The short answer is AT&T is the only one who can doing something about the problem as they are the only one who can "crawl" into the equipment and see what's really going on.

The long answer is based on what I know about voice/video/data communication. Some of my knowledge maybe out of date. Some is speculation based on trends I see from the world of investing.

The long answer. You have been warned.

There could be any number of issues here. Try pinging something in the middle of the day like a dns server on AT&T's network. As many people don't understand, you have a device talking to a router of sorts talking to another router and another and another until you actually reach the host you pinged. If any network in between is having issues, the whole thing will slow down.

Assuming for a minute that the problem isn't outside of AT&T's network, it could be RFI, overloaded towers, temperature at the tower or your phone, old/worn out switches (yes they do wear out), not enough bandwidth, poorly prioritized traffic. I even read a paper once purported to be written by an engineer at IBM claiming some computer equipment fails when a neutron from the sun strikes the silicon wafer in a certain way.

Try pinging with a packet large enough to force a fragmentation say a 2048 bytes when things appear to be working in the middle of the night. Try different ping patterns. A larger packet or different pattern can often fail on a bad switch (or router) while a small one gets through just fine. Try changing your location relative to the towers in the area during the day.

RFI could be coming from unshielded electronics or even an arc welder or a TV or radio station.

Priority traffic probably needs some discussion. Voice and video "calls" can accept some lost packets but cannot stand much of a delay. Computer data "calls" (a smartphone is just a computer that also does priority so voice sounds smooth while you cruise the web on your phone) cannot stand lost packets but can deal with long delays. The tower probably uses an ATM based network or a sonet ring or some other technology that implements quality of service parameters. The tech is designed to make things run smoothly but if it's old or overloaded, it will have issues.

One thing that has been gaining a lot of acceptance is streaming media to the phone. Think listening to cellphone radio or watching tv on your phone. Those applications generate a continues stream of data that has to traverse the network. IP makes the situation worse as every user of the network can have their own dedicated "channel". Imagine 20 people within range of the tower you're near all watching the same TV show at the same time but not necessarily at the same point in the show. That's 20 copies of the show streaming through the cellphone network through "your" tower. By the way, this streaming issue is becoming more serious and it affects all forms of computer networks including cable, dsl, AT&T's uverse, satellite, T1 (DS1), T3 (DS3), frame-relay, ATM, packet switched, etc., etc.

With tablets becoming more popular and the massive push to cloud computing, the problem is going to get a lot worse fast and it's going to spread all over the world. I can just see people using a tablet who have their data sitting in the cloud complaining about the fact they can't call up their documents, emails, maps, to do lists, contacts, etc. Something they could do had the stuck with a traditional laptop.

I'm sure there are readers here who know more than I do about networking and the cellphone networks. I'm sure most will agree there's little you can do other than call AT&T and report the problem.

Any number of issues

tacra's picture

I thought long and hard about how to answer this. I decided to give a short answer and a long answer. The short answer is AT&T is the only one who can doing something about the problem as they are the only one who can "crawl" into the equipment and see what's really going on.

The long answer is based on what I know about voice/video/data communication. Some of my knowledge maybe out of date. Some is speculation based on trends I see from the world of investing.

The long answer. You have been warned.

There could be any number of issues here. Try pinging something in the middle of the day like a dns server on AT&T's network. As many people don't understand, you have a device talking to a router of sorts talking to another router and another and another until you actually reach the host you pinged. If any network in between is having issues, the whole thing will slow down.

Assuming for a minute that the problem isn't outside of AT&T's network, it could be RFI, overloaded towers, temperature at the tower or your phone, old/worn out switches (yes they do wear out), not enough bandwidth, poorly prioritized traffic. I even read a paper once purported to be written by an engineer at IBM claiming some computer equipment fails when a neutron from the sun strikes the silicon wafer in a certain way.

Try pinging with a packet large enough to force a fragmentation say a 2048 bytes when things appear to be working in the middle of the night. Try different ping patterns. A larger packet or different pattern can often fail on a bad switch (or router) while a small one gets through just fine. Try changing your location relative to the towers in the area during the day.

RFI could be coming from unshielded electronics or even an arc welder or a TV or radio station.

Priority traffic probably needs some discussion. Voice and video "calls" can accept some lost packets but cannot stand much of a delay. Computer data "calls" (a smartphone is just a computer that also does priority so voice sounds smooth while you cruise the web on your phone) cannot stand lost packets but can deal with long delays. The tower probably uses an ATM based network or a sonet ring or some other technology that implements quality of service parameters. The tech is designed to make things run smoothly but if it's old or overloaded, it will have issues.

One thing that has been gaining a lot of acceptance is streaming media to the phone. Think listening to cellphone radio or watching tv on your phone. Those applications generate a continues stream of data that has to traverse the network. IP makes the situation worse as every user of the network can have their own dedicated "channel". Imagine 20 people within range of the tower you're near all watching the same TV show at the same time but not necessarily at the same point in the show. That's 20 copies of the show streaming through the cellphone network through "your" tower. By the way, this streaming issue is becoming more serious and it affects all forms of computer networks including cable, dsl, AT&T's uverse, satellite, T1 (DS1), T3 (DS3), frame-relay, ATM, packet switched, etc., etc.

With tablets becoming more popular and the massive push to cloud computing, the problem is going to get a lot worse fast and it's going to spread all over the world. I can just see people using a tablet who have their data sitting in the cloud complaining about the fact they can't call up their documents, emails, maps, to do lists, contacts, etc. Something they could do had the stuck with a traditional laptop.

I'm sure there are readers here who know more than I do about networking and the cellphone networks. I'm sure most will agree there's little you can do other than call AT&T and report the problem.

Any number of issues

Doc Searls's picture

Thanks for a long and thoughtful comment.

I've moved on from that location now (and am writing this from another one on the AT&T cellular network), so I can't pursue a diagnostic course for this isolated case.

But your point about future stress on the network is an especially good one. More and more people will be using their mobile devices as radios and TVs. In the long run most of today's radio and TV, at least in urban and suburban areas, will migrate to the Net, mostly over wi-fi and cellular connections, neither of which are prepared today — much less optimized — for the inevitable.

I already do most of my radio listening over my phone. In fact, this morning I was impressed while listening to SiriusXM, which made a graceful transition from hotel wi-fi to cellular data and back again. And this was at the location that gave me so much trouble earlier.

Click on any of the regional radio market ratings pages here and note that station streams are starting to show up. In San Francisco, KDFC's stream shows up while its signal does not. This is a harbinger of the future.

I am also sure there are engineers at phone and cable companies thinking this through. But the changes required are immense, and not certain to work. But having the long tail, especially the technical one, involved will be a big help.

Doc Searls is Senior Editor of Linux Journal

Multiple paths?

Doc Searls's picture

Here is the same ping test now, at 12:26am:

PING google.com (74.125.225.144): 56 data bytes
64 bytes from 74.125.225.144: icmp_seq=0 ttl=51 time=87.485 ms
64 bytes from 74.125.225.144: icmp_seq=1 ttl=51 time=69.432 ms
64 bytes from 74.125.225.144: icmp_seq=2 ttl=51 time=74.919 ms
64 bytes from 74.125.225.144: icmp_seq=3 ttl=51 time=89.601 ms
64 bytes from 74.125.225.144: icmp_seq=4 ttl=51 time=116.852 ms
Request timeout for icmp_seq 5
64 bytes from 74.125.225.144: icmp_seq=6 ttl=51 time=71.244 ms
64 bytes from 74.125.225.144: icmp_seq=7 ttl=51 time=68.192 ms
64 bytes from 74.125.225.144: icmp_seq=8 ttl=51 time=68.774 ms
64 bytes from 74.125.225.144: icmp_seq=9 ttl=51 time=63.503 ms
64 bytes from 74.125.225.144: icmp_seq=10 ttl=51 time=71.114 ms
64 bytes from 74.125.225.144: icmp_seq=11 ttl=51 time=80.305 ms
64 bytes from 74.125.225.144: icmp_seq=12 ttl=51 time=81.234 ms
64 bytes from 74.125.225.144: icmp_seq=13 ttl=51 time=69.312 ms
^C
--- google.com ping statistics ---
14 packets transmitted, 13 packets received, 7.1% packet loss
round-trip min/avg/max/stddev = 63.503/77.844/116.852/13.566 ms

The pattern is the same every day here. Useless during working hours and evenings, better in the middle of the night. Is this multiple paths?

FWIW, the Speedtest.net results are here. Not bad. But the Pingtest.net failed utterly.

Doc Searls is Senior Editor of Linux Journal

Multi-pathing

Howard Greenstein's picture

Since you're in Midtown, it is very likely that the ATT signal is just being blocked and hitting your phone from multiple paths.
Verizon does a better job (in NYC) than AT&T in figuring this out. That said, I literally work across the street from the Empire State Building and everyone still has cell issue.

Webinar
One Click, Universal Protection: Implementing Centralized Security Policies on Linux Systems

As Linux continues to play an ever increasing role in corporate data centers and institutions, ensuring the integrity and protection of these systems must be a priority. With 60% of the world's websites and an increasing share of organization's mission-critical workloads running on Linux, failing to stop malware and other advanced threats on Linux can increasingly impact an organization's reputation and bottom line.

Learn More

Sponsored by Bit9

Webinar
Linux Backup and Recovery Webinar

Most companies incorporate backup procedures for critical data, which can be restored quickly if a loss occurs. However, fewer companies are prepared for catastrophic system failures, in which they lose all data, the entire operating system, applications, settings, patches and more, reducing their system(s) to “bare metal.” After all, before data can be restored to a system, there must be a system to restore it to.

In this one hour webinar, learn how to enhance your existing backup strategies for better disaster recovery preparedness using Storix System Backup Administrator (SBAdmin), a highly flexible bare-metal recovery solution for UNIX and Linux systems.

Learn More

Sponsored by Storix