The DevBlog documents all developments and promotions that relate to the TM4B Bulk SMS Gateway and its associated services 70589.INFO & T4ME.
Summary: |
An explanation regarding to today's down time. |
Category: |
Down Time |
Link: |
www.tm4bhelp.com/dev/p-100.php |
Details: |
Over the past few days, a handful of customers have experienced a lack of responsiveness from our servers and, whilst we have frantically tried to understand the root problem behind it, we were unable to. Then today, our services were unavailable for almost 2 hours. Following last month's downtime, it is indeed very embarrassing to experience a further case of down-time in such a short period and an explanation is necessary. To begin with, ollowing last month's unexpected downtime, we took many steps to increase the reliability of our services. Part of these steps involved relocating our servers to, what is considered to be, one of the best data-centres in the world. However, as it would happen, even the best have unexpected problems as described by the following 3 messages from the Chief Network Administrator:
The fiber provider for one of our inter-datacenter Cedar Falls to Chicago links has just informed us that they have scheduled a maintenance for the morning of Wednesday, June 27th. This will result in routing reconvergence as traffic is routed around the downed fiber. Customers may see sub-optimal path selection and increased ping times during a single 15 minute window in the maintenance period.
As many of our customers have noticed, our upstream's scheduled maintenance did not go as planned. Due to issues that are still under investigation at this hour, the outage period was longer than was expected, and the window was open longer than scheduled. We are still inquiring with the provider as to why they did not enter a predictable fail state that would have allowed traffic to fail over to another connection at this time, instead of accepting our traffic and then failing to deliver it. Additional updates regarding this issue should be forthcoming later today. We thank everyone for their continuing patience on this matter.
After extensive analysis this morning, it appears that multiple issues, instead of a single one related to this maintenance, resulted in prolonged degraded service for customers in the Cedar Falls datacenter. As mentioned previously, a monitoring / configuration error on the part of the provider performing the maintenance today resulted in a longer than expected maintenance window this morning. This resulted in degraded speeds and latency for some customers in the Cedar Falls datacenter, as traffic had to take the scenic route via backup links to its destination.
In addition to this, further investigation this morning revealed an entirely seperate issue with another of our upstream providers accepting traffic for a subset of remote destinations, and then dropping that traffic, resulting in complete unreachability for some customers on a small selection of remote networks. Since this was only a small subset of the traffic the upstream was handling for our Chicago and Cedar Falls datacenters, it went un-noticed until customers brought it to our attention. Once the issue was pin-pointed, the offending upstream was removed from our routing mix until they reported the issue was solved (which has occured at this time).
As of this morning, the network core and distribution portions are operating normally - if you are still experiencing an issue, please contact our support department immediately so our NOC staff can look further into the issue.
We can only but apologise for this loss in reliability and unexpected downtime. We chose to locate our servers in the Cedar Falls data-centre because it is one of, if not, the best in the world and will continue to maintain our servers there, taking this as a one-off occurrence. We also request customers to inform us should they experience any problems. |
Summary: |
We have resolved the errors surrounding our UK Numbers. |
Category: |
Down Time |
Link: |
www.tm4bhelp.com/dev/p-90.php |
Details: |
We would like customers to know that the recent problem surrounding the UK Receiving Numbers has been fixed and they are fully live again. This has taken the attention of our entire support team and we apologise (a) for the downtime caused to those customers using UK Receiving Numbers and (b) for those customers that could not receive a support levels as high as usual. One interesting point, for those customers that may (sometimes) feel unhappy about our decision not to provide tele-support, is that a time like this is an excellent example of when the attention of certain members of staff need to be directed towards issues which hold paramount importance over other (comparatively) less important issues. We are dedicated to the highest levels of support and the use of the Contact Centre for customer communications makes this possible. |
Summary: |
We are currently facing problems with our UK receiving numbers. |
Category: |
Down Time |
Link: |
www.tm4bhelp.com/dev/p-89.php |
Details: |
We would like to advise customers that, due to problems which we are still trying to resolve, we are unable to receive MO SMS through our UK receiving numbers. MO SMS are reaching the Network Operators and are being forwarded in our direction but we are, for reasons which we are not yet able to establish, not able to receive them. We can only but apologise about the inconvenience(s) which are being caused and would like customers to know that we are working around the clock to understand and resolve this matter. Unless otherwise requested by affected customers, all of the messages which should have been handled during this time will be handled when the numbers come back online. |
Summary: |
T-Mobile has removed their suspension on our 70589 program. |
Category: |
Down Time |
Link: |
www.tm4bhelp.com/dev/p-82.php |
Details: |
We are happy to inform customers using our shared 70589 program in the U.S. that T-Mobile has removed their suspension and the messages are now being sent/received through the T-Mobile network without any problems. Network carriers in the U.S. are very serious about ensuring that their mobile messaging industry is protected and, to that effect, they do not hesitate to suspend/terminate short code programs which they see go against their rules & regulations. Fortunately, in our case, the suspension was due to a misunderstanding which was resolved immediately, but occurrences like these should sound as a warning and remind us about the importance of providing mobile messaging services which are carrier compliant. We would also like to take this occasion to announce that we have increased the annual cost of subscription services on the 70589 program from $49.00 per year to $99.00. Whilst this new rate may not necessarily be in-line with our low cost policies, we hope that it will help ensure only serious content providers to make use of services which are shared with other users. |
Summary: |
Our shared short code program 70589 is under temporary suspension. |
Category: |
Down Time |
Link: |
www.tm4bhelp.com/dev/p-81.php |
Details: |
Customers using our shared short code program 70589 should be aware that it is under a temporary suspension placed by T-Mobile. For one reason or the other, which we are in the process of establishing, they have caught feeling that our program is non-compliant with their rules & regulations. We are currently discussing the issue with them and aim to resolve it as quickly as possible. Meanwhile, this means that we are unable to receive MO SMS from, or send MT SMS to, T-Mobile subscribers through the shared 70589 program. |
Summary: |
During the festive period, static originator registration will be interrupted. |
Category: |
Down Time |
Link: |
www.tm4bhelp.com/dev/p-74.php |
Details: |
In between December 21st 2006 and January 3rd 2007, the registration process which all static originators must go through will be temporarily interrupted. Unfortunately, this is due to the third parties which this process depends upon and we would like customers to know, in advance, that any static originators which are still pending at the beginning of this period will not be progressed until its end. Similarly, the registration process for any new static originators purchase during that period will not begin until January 3rd 2007. |
Summary: |
T-Mobile (USA) has again rescheduled their planned maintenance. |
Category: |
Down Time |
Link: |
www.tm4bhelp.com/dev/p-62.php |
Details: |
Annoyingly, T-Mobile (USA) has, once again, rescheduled their upcoming planned maintenance - the details on the original notice have been modified to reflect these. We apologise for the further inconvenience, but there is not much we can do except to let our customers know. |
Summary: |
T-Mobile (USA) has rescehduled their planned maintenance. |
Category: |
Down Time |
Link: |
www.tm4bhelp.com/dev/p-60.php |
Details: |
We would like to advise customers that T-Mobile has rescheduled the second phase of their planned maintenance from August 23rd to August 30th. The full (modified) details of the maintenance can be found here. |
Summary: |
T-Mobile (USA) will be suffering from some expected down-time. | ||||||||||
Category: |
Down Time | ||||||||||
Link: |
www.tm4bhelp.com/dev/p-57.php | ||||||||||
Details: |
We would like to forewarn customers that T-Mobile will be performing planned maintenance on their network on the dates and times noted below. We apologize for any inconvenience this may cause.
|
Summary: |
We will be upgrading our database servers tomorrow. | ||||||||||
Category: |
Down Time | ||||||||||
Link: |
www.tm4bhelp.com/dev/p-18.php | ||||||||||
Details: |
We do apologise for the short notice, but this upgrade was not due to take place for a few weeks. It has been brought forward out of necessity to maintain the integrity of our data and has been allocated the least busy time. |