DiSC Team: Tues, May 25th, 2004 -- Accra, Legon

A picture of Omar | A picture of RJ and collegues | Samir, RJ and Omar

Today was a national holiday (actually a pan-African holiday), so we couldn't work at the library. We had another productive talk with Mark Davies, and experienced the frustration of using the Internet in Ghana first hand when the "always-on" satelite connection had problems at BusyInternet.

Mark gave us a bunch of good contacts for Ronnie and Samir to pursue, and we talked with one of the other organizations located at BusyInternet which does a business innovation competition for young entrepreneurs who want to start up a business in Ghana.

Mark also made it clear that he feels (how do I put this politely?) there is a lack of innovation in the IT business in Ghana. He feels (if I may paraphrase) that while there is no lack of raw programming talent and technical talent, companies in Ghana have trouble managing software/technical projects, providing customer service and other core business functions. Furthermore, this problem stems from an lack of experience: developers simply haven't had the opportunity to learn by doing. He suggested that more partnerships take place where Ghanaians developers get mentored, either explicitly or simply by the structure of a relationship, by international companies with more experience. I tend to agree with Mark that business experience and software development experience is irreplacable and essential to the success of a company, especially a software company.

Perhaps because it is not terribly interesting from a technical standpoint, especially accademic, we have not focused on some of the low hanging fruit: poorly managed or architected networks. Now I realize that there is probably a lot of pride on the part of the network administrators here, that they have accomplished a lot with a little, and I certainly don't want to insult or degrade their considerable efforts. Furthemore, some of the problems may stem from lack of resources (ie. time) rather than lack of knowhow, as Benette suggested yesterday.

With that said, however, I must say that I am baffled by two observations that I have made about the networks topologies that I have encountered. Clearly I don't understand all of the issues at stake, and I intend to get to the source of these architectural decisions that have seen.

First, both BusyInternet and the Legon campus have their email servers located in the United States. Both networks use VSAT (a geo-syncronous orbit, meaning high latency, satelite technology), to connect "directly" to the Internet: there is no land line link to a Ghana ISP. Mark made Busy's goal in his email server network design clear: to avoid using precious VSAT bandwidth when users dial in to another ISP to check their email being hosted on the BusyInternet server.

Now, I suspect that Busy has this problem partly because they want to provide WebMail (hotmail-like) email access on the web. This is an easy paradigm to understand for users, and makes sense for users who don't have their own computers, and don't want to reconfigure the POP or IMAP email client on the computer they are using every time they go to an Internet cafe to check their email. The result of this decision, however, is that it makes an important common case slow: checking your email from BusyInternet, or within their network (Busy is an ISP, which provides Internet service to local businesses via fixed wireless or dialup).

Here are three important facts about email:

1. SMPT (the protocol used to transport email from one server to the next) is an asyncronous protocol. This means that when you send an email, it gets sent to your local email server. When that server gets the chance, it forwards your email to another email server, closer to the email's destination. This process continues until the data arrives at its destination. This is called a "store-and-forward" design, meaning that data is stored in the intermediate hops until it makes it to the next one.

   An important and desirable characteristic of this design is that in environments where there is a crappy network link, data can sit and wait until the link comes back up, at which time it gets transfered across the link.

2. SMTP is delay tollerant, but HTTP is not. What does this mean, and why do we care? This means that the delivery of an email can wait a few seconds without any bad consequences: mail delivery is a "background" process which does not take up any user's time. We, as users, are used to the fact that when we send an email, it may not arrive for several seconds, or even minutes, hours or days, in the recipient's inbox. Wep pages, however, are not so delay tollerant. When a web page takes several seconds or even minutes to load, we get very frustrated. Sometimes WAN (Wide Area Network) links can be down for hours or days, meaning that web pages on the far side of the link are inaccessible. Thus, as a network designer, I want to minimize HTTP (the protocol used to transfer web pages) delays, even at the expense of increasing SMPT delays.
3. Email is, essentially, "non-mutating," meaning that the content of email does not change once it has been sent (strictly this is not true since email servers tack on headers when they forward email, but it is true engough). This means that it is safe to replicate email data profusely.

This brings me to my next point: why are Busy and Legon at the edge of the network? The Internet was very explicitly designed without a central routing authority. The ISPs (and Legon) should be linking directly to each other in order to save bandwidth and avoid latency over their precious VSAT links. This can be accomplished by routing data between "peer" ISPs over a local link, wireless or otherwise. This is exactly how the Internet was designed.

Mark seemed to believe that in order to route data within Ghana, there must be what he called an "Internet exchange." It may be the case that if many ISPs decide to collaborate on creating an Internet backbone in Ghana, that a more centralized, star-like network topology would make sense (since the number of connections required in a fully connected network is n² with respect to the number of nodes in the network). But Mark also lemented the fact that the planned "Internet exchange" has been dead in the water for some time, because the ISPs and other stakeholders can't come to consensus about something or other.

In that case, BusyInternet, Legon et. al can't afford to wait for consensus to appear. Busy, in my opinion, should establish a direct link with one or more of the major ISPs in the area, and route data to and from those ISPs directly over the local link. This decision would provide a competetive advantage to those who participate, since participating ISPs would experience more responsive network performance, and also conserve bandwidth over the expensive VSAT links.

Similarly, Legon should focus on providing internal services. Forget about providing Internet access to all their students: provide an email account and "intranet" access for course related activites. Over their fast local network, this will be much easier than trying to manage their services remotely. For my part, I intend to discover the root causes of these network architectural problems: is it the lack of resources and manpower? Lack of technical knowhow to actually configure the relevant softare and hardware? Lack of usable software? Lack of fundamental understanding of these issues?

We have been invited to give talks at both Legon and Busy, and there is an upcoming ICT conference in Accra that we might be able to talk at too. Perhaps I will give a talk encouraging the following mantra: "keep processing close to data, keep data close to processing." This is essential in high latency, "challenged" networks like those at Legon and BusyInternet, and is really at the core of these architectural problems.