Backup Mail Exchangers

January 28, 2009

On Monday night, the power supply unit (PSU) in the server that hosts our mail server failed at around 2200 GMT. We don’t have physical access to the server out of hours, so I wasn’t able to replace it until about 1045 the next day, so our main email server was down for nearly 13 hours.

We didn’t have a backup MX because:

  • It usually can’t check whether recipients are valid or not, and therefore must accept mail that it can’t deliver;
  • It usually doesn’t have as good antispam checks as the primary, because it’s a hassle to keep it updated;
  • Spammers usually abuse backup MXes to send more spam, including Joe Jobs.

I thought that this was OK because people who send us mail also have mail servers with queues, which should hold the mail until our server comes back up. It’s normal for mail servers to go down sometimes and this should not cause mail to be lost or returned.

However, we had a report that one of our users did not receive a mail addressed to them, and was told by the sender that it had bounced. I saw the bounce messsage and suspected Exchange, so I decided to check how long Exchange holds messages before bouncing them. Turns out it’s only five hours by default. Most mail servers hold mail for far longer, for example five days, sending a warning message back to the sender after one day.

Bouncing messages looks bad on us. Apart from making our main mail server more reliable 🙂 we need a backup MX to accept mail when the master is down.

However I do still want to minimise the spam problem that this will cause. Therefore I configured our backup MX to only accept mail when the master is down. Otherwise it defers it, which will tell the sender to try sending it to the master (again).

How did I achieve this magic? With a little Exim configuration that took me a day and that I’m quite proud of. I set up a new virtual machine which just has Exim on it, nothing else. I configured it as an Internet host, and to relay for our most important domains. Then I created /etc/exim4/exim4.conf.localmacros with the following contents:

CHECK_RCPT_LOCAL_ACL_FILE=/etc/exim4/exim4.acl.conf
callout_positive_expire = 5m

This allows us to create a file called /etc/exim4/exim4.acl.conf which contains additional ACL (access control list) conditions. The other change, callout_positive_expire, I’ll describe in a minute.

I created /etc/exim4/exim4.acl.conf with the following contents:

# if we know that the primary MX rejects this address, we should too
deny
        ! verify = recipient/callout=30s,defer_ok
        message = Rejected by primary MX

# detect whether the callout is failing, without causing it to
# defer the message. only a warn verb can do this.
warn
        set acl_m_callout_deferred = true
        verify = recipient/callout=30s
        set acl_m_callout_deferred = false

# if the callout did not fail, and the primary mail server is not
# refusing  mail for this address, then it's accepting it, so tell
# our client to try again later
defer
        ! condition = $acl_m_callout_deferred
        message = The primary MX is working, please use it

# callout is failing, main server must be failing,
# accept everything
accept
        message = Accepting mail on behalf of primary MX

The first clause, which has a deny verb, does a callout to the recipient. A callout is an Exim feature which makes a test SMTP connection and starts the process of sending a mail, checking that the recipient would be accepted. This is designed to catch and block emails that the main server would reject. Our backup server has no idea what addresses are valid in our domains; only the primary knows that.

The callout response is cached for the default two hours if it returns a negative result (the recipient does not exist on the master) or five minutes (see callout_positive_expire above) if the address does exist. We use a defer_ok condition here so that if we fail to contact the master, we don’t defer the mail immediately, but instead assume that the address is OK and therefore continue to the next clause.

The second clause of the ACL, which has a warn verb, is what took me so long to work out. Normally, if a condition in a statement returns a result of defer, which means that it failed, the server will defer the whole message (tell the sender to come back later). In almost all cases this is the right thing to do, but it’s the exact opposite of what we want here. We want to accept mail if the callout is failing, not defer it, otherwise our backup MX is useless (it stops accepting mail if the primary goes down).

Because this is such an unusual thing to do, there is no configurable option for it in Exim. The only workaround that I found is that there is exactly one way to avoid a deferring condition causing the message to be deferred: a warn verb. The documentation for the warn verb says:

If any condition on a warn statement cannot be completed (that is, there is some sort of defer), the log line specified by log_message is not written… After a defer, no further conditions or modifiers in the warn statement are processed. The incident is logged, and the ACL continues to be processed, from the next statement onwards.

So what we do is:

  1. Set the local variable
    acl_m_callout_deferred to true;
  2. Try the callout. If it defers (cannot contact the primary server) then we stop processing the rest of the conditions in the warn statement, as described above;
  3. If we get to this point, we know that the callout did not defer, so we set acl_m_callout_deferred to false.

The third clause  of the ACL, which has a defer verb, simply checks the variable that we set above. If we get this far then the primary server is not rejecting this address; and if it’s not deferring either, then it must be accepting mail for the address. In that case, we defer the message, telling our SMTP client to try again later, at which point it will hopefully succeed in delivering directly to the primary.

Callout result caching becomes a problem here. If the master was not reachable, but a previous callout had verified that a particular address existed, and that callout result was cached for the default 24 hours, then the backup MX would defer subsequent mail to that address for the next 24 hours, even if the master went down. This is why we changed the positive callout result caching time to 5 minutes earlier.

The fourth clause  of the ACL, which has an accept verb, is even simpler. It accepts everything that was not denied or deferred earlier. We can only get this far if the master is not accepting or rejecting mail for that address.

So far the configuration appears to work fine and has blocked 14 spam attempts (abusing the backup MX) in 14 hours.

Offline Wikipedia part 2

December 1, 2008

Having decided on a local MediaWiki installation, I started working through the import process. I noticed a few things that may help others.

If one forgets to increase the MySQL max_packet_size, then the import breaks somewhere in the middle (around 3 million records) but the Java process keeps producing progress information, so it’s not at all clear that the import has failed. One sign is that the import process rate of progress, as reported by the import tool in pages per second, suddenly speeds up by a factor of 5-10. You may wish to look out for this and abort the import if it happens, and to monitor the import process with mysqladmin processlist to ensure that it’s still doing things.

Installing the MediaWiki ParserFunctions extension solves most of the problems with random program code appearing in articles.

The import will tend to slow down very badly over time. For example, on one system it started at a rate of 160 pages/second and dropped to 18 over a three-day period. At this rate, it would have taken around 5-6 days to import all 7.5 million pages. Using the MySQL disable keys command did not help much, but what did was to restructure the tables to remove all the indexes. You can even do this while the import is running (I did). The SQL commands are:

  • ALTER TABLE page MODIFY COLUMN page_id INT(10) UNSIGNED NOT NULL, DROP PRIMARY KEY, DROP INDEX name_title, DROP INDEX page_random, DROP INDEX page_len;
  • ALTER TABLE revision MODIFY COLUMN rev_id INT(10) UNSIGNED NOT NULL, DROP PRIMARY KEY, DROP INDEX rev_id, DROP INDEX rev_timestamp, DROP INDEX page_timestamp, DROP INDEX user_timestamp, DROP INDEX usertext_timestamp;
  • ALTER TABLE text MODIFY COLUMN old_id INT(10) UNSIGNED NOT NULL, DROP PRIMARY KEY;

The following SQL commands should restore the indexes after the import is complete. If you don’t do this, the MediaWiki site will be very slow in operation.

  • ALTER TABLE page MODIFY COLUMN page_id INT(10) UNSIGNED NOT NULL AUTO_INCREMENT PRIMARY KEY, ADD UNIQUE KEY name_title (page_namespace,page_title), ADD KEY page_random (page_random), ADD KEY page_len (page_len);
  • ALTER TABLE revision MODIFY COLUMN rev_id int(10) UNSIGNED NOT NULL AUTO_INCREMENT PRIMARY KEY, ADD UNIQUE KEY rev_id (rev_id), ADD KEY rev_timestamp (rev_timestamp),
    ADD KEY page_timestamp (rev_page,rev_timestamp), ADD KEY user_timestamp (rev_user,rev_timestamp), ADD KEY usertext_timestamp (rev_user_text,rev_timestamp);
  • ALTER TABLE text MODIFY COLUMN old_id int(10) UNSIGNED NOT NULL AUTO_INCREMENT PRIMARY KEY;

With these changes I was able to achieve import speeds around fifty times faster, or 1000 pages per second, which should make it possible to import the entire Wikipedia in about 2 hours.

Offline Wikipedia

November 21, 2008

I’m working on making Wikipedia, the (in)famous free encyclopaedia, available offline, for a project in a school in rural Zambia where Internet access will be slow, expensive and unreliable.

What I’m looking for is:

  • Completely offline operation
  • Runs on Linux
  • Reasonable selection of content from English Wikipedia, preferably with some images
  • Looks and feels like the Wikipedia website (e.g. accessed through a browser)
  • Keyword search like the Wikipedia website

Tools that have built-in search engines usually require that you download a pages and articles dump file from Wikipedia (about 3 GB download) and then generate a search index, which can take from half an hour to five days.

For an open source project that seems ideally suited to being used offline, and considering the amount of interest, there are surprisingly few options (already developed). They also took me a long time to find, so I’m collating the information here in the hope that it will help others. Here are my impressions of the solutions that I’ve tried so far, gathered from various sources including makeuseof.com.

The One True Wikipedia

The One True Wikipedia, for comparison

MediaWiki (the Wikipedia wiki software) can be downloaded and installed on a computer configured as an AMP server (Apache, MySQL, PHP). You can then import a Wikipedia database dump and use the wiki offline. This is quite a complex process, and importing takes a long time, about 4 hours for the articles themselves (on a 3 GHz P4). Apparently it takes days to build the search index (I’m testing this at the moment). This method does not include any images, as the image dump is apparently 75 GB, and no longer appears to be available, and it displays some odd template codes in the text (shown in red below) which may confuse users.

Mediawiki local installation

Mediawiki local installation

Wikipedia Selection for Schools is a static website, created by Wikimedia and SOS Childrens Villages, with a hand-chosen and checked selection of articles from the main Wikipedia, and images, that fit on a DVD or 3GB of disk space. It’s available for free download using BitTorrent, which is rather slow. Although it looks like Wikipedia, it’s a static website, so while it’s easy to install, it has no search feature. It also has only 5,500 articles compared to the 2 million in Wikipedia itself (about 0.25%). Another review is on the Speed of Creativity Blog. Older versions are available here. (thanks BBC)

Wikipedia Selection for Schools

Wikipedia Selection for Schools

Zipedia is a Firefox plugin which loads and indexes a Wikipedia dump file. It requires a different dump file, containing the latest metadata (8 GB) instead of the usual one (3 GB). You can then access Wikipedia offline in your browser by going to a URL such as wikipedia://wiki. It does not support images, and the search feature only searches article titles, not their contents. You can pass the indexed data between users as a Zip file to save time and bandwidth, and you may be able to share this file between multiple users on a computer or a network. (thanks Ghacks.net)

WikiTaxi is a free Windows application which also loads and indexes Wikipedia dump files. It has its own user interface, which displays Wikipedia formatting properly (e.g. tables). It looks very nice, but it’s a shame that it doesn’t run on Linux.

WikiTaxi screenshot (wikitaxi.org)

WikiTaxi screenshot (wikitaxi.org)

Moulin Wiki is a project to develop open source offline distributions of Wikipedia content, based on the Kiwix browser. They claim that their 150 MB Arabic version contains an impressive 70,000 articles, and that their 1.5 GB French version contains the entire French Wikipedia, more than 700,000 articles. Unfortunately they have not yet released an English version.

Kiwix itself can be used to read a downloaded dump file, thereby giving access to the whole English Wikipedia via the 3 GB download. It runs on Linux only (as far as I know) and the user interface is a customised version of the Firefox browser. Unfortunately I could not get it to build on Ubuntu Hardy due to an incompatible change in Xulrunner. (Kiwix developers told me that a new version would be released before the end of November 2008, but I wasn’t able to test it yet).

Kiwix (and probably MoulinWiki)

Kiwix (and probably MoulinWiki)

Wikipedia Dump Reader is a KDE application which browses Wikipedia dump files. It generates an index on the first run, which took 5 hours on a 3 GHz P4, and you can’t use it until it’s finished. It doesn’t require extracting or uncompressing the dump file, so it’s efficient on disk space, and you can copy or share the index between computers. The display is in plain text, so it looks nothing like Wikipedia, and it includes some odd system codes in the output which could confuse users.

Wikipedia Dump Reader

Wikipedia Dump Reader

Thanassis Tsiodras has created a set of scripts to extract Wikipedia article titles from the compressed dump, index them, parse and display them with a search engine. It’s a clever hack but the user interface is quite rough, it doesn’t always work, requires about two times the dump file size in additional data, it was a pain to figure out how to use it and get it working, and it looks nothing like Wikipedia, but better than the Dump Reader above.

Thanassis Tsiodras' Fast Wiki with Search

Thanassis Tsiodras' Fast Wiki with Search

Pocket Wikipedia is designed for PDAs, but apparently runs on Linux and Windows as well. The interface looks a bit rough, and I haven’t tested the keyword search yet. It doesn’t say exactly how many articles it contains, but my guess is that it’s about 3% of Wikipedia. Unfortunately it’s closed source, and as it comes from Romania, I don’t trust it enough to run it. (thanks makeuseof.com)

Pocket Wikipedia on Linux

Pocket Wikipedia on Linux (makeuseof.com)

Wikislice allows users to download part of Wikipedia and view it using the free Webaroo client. Unfortunately this client appears only to work on Windows. (thanks makeuseof.com)

WikiSlice (makeuseof.com)

WikiSlice (makeuseof.com)

Encyclopodia puts the open source project on an iPod, but I want to use it on Linux.

Encyclopodia

Encyclopodia

It appears that if you need search and Linux compatibility, then running a real Wikipedia (MediaWiki) server is probably the best option, despite the time taken.

One of the reasons that using a Content Management System (or CMS) is so powerful is that they allow people with little to no knowledge of building web pages to set up and start publishing their very own site with hardly any fuss. Back in the day when I first learned how to design basic websites it was through grubbing around finding quick and dirty fixes to html and css in order to get to roughly where I wanted to be. Anyone starting out today could create a much funkier site in far less time and with far, far less effort.

Essentially CMS opens up online publishing to those with very limited web literacy allowing them to get cracking on their content. That’s very empowering.

But there’s a catch

However, one of the repercussions of this very welcome deskilling has been the fact that we are left in the hands of the CMS providers. Without any understanding of how to create your own sleek, bandwidth friendly site you’re often left with the clunky scripts, unnecessarily large graphics and general baggage that comes with your template design.

Aptivate created a set of Guidelines to help people understand that whilst in the developed world it’s tempting to believe that bandwidth is infinite if you don’t clear out the junk (that you may not even notice) from your websites you may unintentionally be preventing those in developing nations from using your site at all.

Unfortunately, for those who are using a CMS to create their websites they are far less likely to have the know how or the confidence to get into the bowels of their site in order to make something usable for those on low bandwidth. Which means, that for this particular audience, we need to provide solutions for them, rather than attempting to train up the whole world up from scratch on how to create a low bandwidth CMS template.

I’ve long argued that something that would be a useful addendum to our guidelines would be a set of good looking and functional low-bandwidth templates for the main CMS providers. Providing a way to strip down drupal, blogger, wordpress, et al would be phenomenally useful for those without the skills to do this for themselves.

Effectively this would be a low bandwidth website in a box which even current users of these CMS’s could transfer over to without much fuss. Currently the best alternative I’ve found at my regular (blogger) blog The Daily (Maybe) is to provide a link to a loband provided version which is certainly faster but is a bit of a hastse and doesn’t allow me any real flexibility over layout.

Maneno: a blogging platform for Africa

So imagine my joy when I came across Maneno last week. A CMS blogging platform designed specifically with low bandwidth in mind and provided from servers in Africa, cutting down on slow internal connections. As the blurb says “Maneno strives to provide a communication and development platform for Sub-Saharan Africa.”

Good looking and providing all the functionality you need in a decent website, the online feedback I’ve seen so far has been universally positive, particularly around download times, which can massively increase the expense of browsing the net in the very places where this service needs to be as cheap as possible. That is really important. In the words of blogger White African “The site absolutely flies.”

Although Maneno is still in a beta version it works like a dream and looks very impressive. It seems just the ticket if you are setting up a new site with little knowledge of design and want to ensure potential readers in Africa actually get the opportunity to read what you have to say.

I apologise for not writing another article sooner. I’d like to respond to some of the comments on my previous article, IT in the field, by some of the people I mentioned, Jeff Allen and Jon Thompson. I’ll include their comments and my responses inline.

Jon writes:

1) Almost no one uses PDA’s. Mobiles – yes. (See my post on Nokia Data Gathering.)

I would argue that I’m coming from a different position, not on what’s currently in the field and being used, but what could be. I think that Jon is referring to a similar kind of future possibilities in his follow-on article. But I’m also thinking in terms of what an organisation (NGO or government) can do to avoid these problems in the first place, by changing the way that they deploy technology.

I accept completely that PDAs are not commonly used in the field at the moment. However, I’d suggest that organisations take a much closer look at deploying PDAs (and mobile phones) instead of general-purpose PCs in current and future projects, as it would avoid most of the problems that Jeff reports and that we are discussing here.

2) Is anyone buying and deploying Inveneo? Who is Aleutia? What is their penetration. Odds are Jeff’s guy in Congo will never know about either.

I’m not intending to put the onus on Jeff’s guy in Congo, or any other end user, to select appropriate technology for their circumstances. I don’t think they have the technical skills or buying power to do so. The choice was already made for them by the organisation that supplied a PC running Windows without antivirus updates. I believe that this was a bad choice, driven by the factors that I gave in my previous article, and that different choices could be made in future to avoid such problems.

3) Webmail will stay the same until the cows come home. Sure, someone can write tricky code but will the guy in Congo ever see it? Probably not.

Actually, I have seen comments asking why Google doesn’t use their own Gears library for their own webmail service, particularly in the context of recent Gmail service outages. If they did decide to do so, the benefits should translate immediately to users with low-bandwidth or intermittent connections, in the Congo as elsewhere.

4) Mac OSX does not exist in the rest of the world and barely in the Balkans, some of which are even slated for EU membership.

I don’t see that as a strong argument against deploying alternatives such as MacOS X in new systems. We have a problem: poor maintainability of traditional IT systems in the field. We have to find solutions to those problems, which may involve deploying different technology. The same argument applies to Ubuntu.

5) So you do support Ubuntu adoption as an alternative? I think you are getting my point. You could just install Ubuntu and forget about the AVG argument.

But the end user will not deploy Ubuntu. The providers of the equipment will have to do it for them. That’s my point, that the organisations have to change, not the end users.

Vertical programs that distribute hardware almost never follow-up so anything they put in the field is usually toast within a few months.

That’s precisely my point in the last article, even if I rambled a bit in the process of getting there. But I think it would be great to have hard numbers that we could present as evidence to the organisations to get them to recognise the problems in what they’re doing, and to change their ways.

For example, let us just say that something causes the OS to hiccup so the well meaning local IT guy steps in and recommends reinstalling the OS. Of course the owner doesn’t have one so the IT guy offers one of his own. Bootleg install. Story over.

That is certainly true for general-purpose computers, but even a minimal amount of BIOS lock-down can prevent the reinstallation of the software, or make it significantly harder. There’s also the question of why the man in the Congo is turning to his “local IT guy” for technical support? Why isn’t the organisation that provided the computer providing support for it as well?

9) Forget about NGO’s as they are not the problem. Remember, this guy worked for the Congolese Gov’t so he gets whatever trickles down (a few odd machines from international agencies)

These “international agencies” are precisely who I meant by NGOs. Perhaps I should have included IOs as well, but all this jargon is going to get very confusing to anyone who’s not an expert in the field. Is there any reason not to lump NGOs, IOs and government programmes into the same basket as “organisations” all?

His support team? The local DVD vendor and the well meaning IT tech. Therefore, educate the local health official, the DVD vendor and the local IT tech…

And why are they his only (or primary) means of support? MSF sent Jeff out to deal with a problem that wasn’t even to do with any equipment that they provided (as far as we know), but it was stopping the man in the Congo from interacting with MSF.

Had MSF not sent Jeff to the Congo, you, me and Jeff would be none the wiser.

Indeed, but had the original organisation not sent out the wrong equipment (a Windows computer with no support and no antivirus updates), MSF would not have had to waste a lot of valuable time and money and resources on sending Jeff out in the first place. Also, had MSF provided their own, more suitable equipment, for the man in the Congo to use, Jeff would not have had to make an on-site visit either.

I’d like to respond to Jeff’s comments as well, but this article is already getting rather long and ranty, so I’ll leave that for another day.

With deregulation very much on people’s minds due to the imminent demise of capitalism I thought now might be a good time to discuss the international consensus on the “liberalisation” of public utilities in the developing world. By liberalisation I, of course, mean private companies delivering public services like water and electricity.

Whilst there is clearly a debate in regard to publicly run utilities in the West, with some for and some against, there seems to be an unparalleled consensus when it comes to the governments of the developing world and their (in)ability to provide public services without private help.

Partly this is historical. In the West there has been long term infrastructure investment so a break up usually involves pawning the family silver and a shift away from a proven model. In much of the developing world this infrastructure is in a far more uneven state, and poorer governments have difficulty finding the funds to roll out new pipelines (for example) to rural areas. Who better then than private companies to do it for them? After all 51 of the 100 largest economies in the world are corporations who are only too willing to make short term investments for long term returns.

A failing state whose government doesn’t care one way or the other whether the people are well served is unlikely to be able to deliver essentials like a decent telecommunications network to isolated communities. So when it came to handing Tanzania’s water provision over to a UK company no one was prepared to argue that this wasn’t going to work, after all, they’re the experts. Except it was a disaster (see here).

The same argument applies there as it does here. Private companies provide these services in order to make a profit. So the service will be the minimum required and the cost will need to include the director’s new private jet. You get less for more in an NHS hospital run by private finance – why would it be any different under an African sun?

The only real difference is that when private water companies stuff up in the West it’s just a case of more leaky pipes and higher bills – in developing regions whole communities run dry. Whilst good financial management is obligatory, the aim has to be the human benefit in country not the bank balances of rich Westerners.

The very point that makes liberalisation seem more acceptable in developing nations (weak, ineffective states) also means these states have real difficulties regulating the behavior of private companies. The greater the liberalisation, the weaker these states become until there is barely any democratic control over the waytilities are run at all. It feeds an addictive cycle that means the deeper the liberalisation, the harder it is for a nation state to provide public services direct.

The free market consensus has meant that the very thing that undermines the ability of developing world governments to become effective managers over essential services is seen as the thing that is the savior of these services.

In Bolivia the gas industry went for years without making any significant contribution to the national coffers, and the jobs it provided were almost entirely staffed by non-Bolivians – which meant it was only when the Morales government introduced proper taxation (under the misnomer nationalisation) did this significant industry begin to fund a program of reforms that had a direct impact on the lives of the poorest Bolivians.

Even the WTO admits that “There is no causal link between foreign investment and poverty reduction. 80% of FDI is in the form of mergers and acquisitions, little in the form of productive investment that creates jobs and exports.”

During a largely unobjectionable speech a few days ago WTO Director-General Pascal Lamy argued for “the reduction of barriers to trade in critical services, such as banking, energy, and environmental services”. This whilst the rest of the world is arguing for an increase in regulation to the banking industry. So even when the tried and tested methods of the modern market are being rethought for the West poorer, less influential nations still get fed the same old diet of the state being a barrier to growth – and they aren’t in a position to underwrite entire industries if they get into trouble.

Lars Thunell of the World Bank’s International Finance Corporation (IFC) let the cat out of the bag when he said in August that “We believe that providing clean water and sanitation services is a real business opportunity.” Well, that’s great, but first and foremost it’s a human necessity.

In practice, in places like Zambia, “the twin policy goals of ensuring commercial viability and meeting social objectives have been shown to be incompatible, if not contradictory, under the new system.” The WTO says that “Water privatization, almost everywhere else that it has applied, has meant more expensive and lower quality water for poorer communities, or even, as in Puerto Rico, no water at all for the poor. (The rich in these places benefit from fine water.)”

Shripad Dharmadhikary talks about how “The [World] Bank’s knowledge is frequently created by highly paid, often international, consultants, who have little knowledge of local conditions. The knowledge creation is mostly directed towards arriving at a pre-determined set of policies – privatisation and globalisation. This knowledge creation is often selective, in that information, evidence or experiences that do not support these pre-determined outcomes are ignored.” In other words people in poor nations are disempowered by ‘experts’ in rich nations when it comes to running their own economy and essential services.

International mechanisms are used to bulldoze opposition rather than facilitate meeting targets like the Millenium Development Goals. So, for example, “the benefit of GATS [are seen as] as helping to overcome domestic resistance to change, to render domestic protest against privatization futile. It is time we examined whether this is really the appropriate tactic to raising living standards, or whether listening to the concerns of the poor is in fact necessary.”

A bit of a trawl though the WTO archives revealed that even organisations responsible for this mess recognise that their past behavior has not been entirely helpful. “In almost all countries that have undertaken rapid trade liberalisation, wage inequality has increased [seeing a] 20-30% fall in wages in some Latin American countries. Trade liberalization is negatively correlated with income growth among the poorest 40 per cent of the population, but positively correlated with income growth among higher income groups. In other words, it helps the rich get richer and the poor get poorer.”

But whilst it’s important to recognise that the market does not provide magical solutions we still aren’t out of the problem that very poor countries often do not have the skills nor funds to invest in major infrastructure projects. Global envision express this idea most clearly when they said “The fact is that most developing-country markets are too small to support infant-industry promotion and their states are too weak, incompetent and corrupt to efficiently administer the complex instruments required. As for WTO rules, it makes sense for developing-country governments to voluntarily enter into commitments with other WTO members that bind in sensible policies (for example, to restrict subsidies and performance requirements), and provide external discipline against silly and harmful government intervention.”

In other words, if you play by their rules you don’t have real options. Without the ability to deliver upfront investment there is no way to avoid the inevitable corruption that comes from working with Western corporations. Alternatively you could substantially tax their profits to fund better infrastructure, health and education – which benefits those corporations in the long run anyway.

A fairer deal takes countries out of the indignity of waiting for aid packages that don’t deliver and having political ‘reform’ foisted on you that is just a one sided set of rules that lock those nations into further dependency.

You don’t build effective states by systematically undermining their ability to deliver the essential services that their populations demand. Building functioning democracies relies upon local people having direct control over their economy and public utilities. Whilst liberalisation seems like a good go round for today’s problems it breeds its own future difficulties and dependencies so needs to be deployed on a case by case basis rather than as a matter of ideology.

Fibre for Africa

September 26, 2008

The consensus seems to be that Africa needs more land and submarine links to provide enough bandwidth for its long-term growth, and bring down satellite Internet costs.

East Africa currently depends completely on expensive satellite bandwidth. There are several projects in progress to lay submarine cables down the east and west coasts, and it can be difficult to keep track of them all.

Luckily, Steve Song has drawn an excellent map here. I’m linking it here to help people find it (including myself, next time I need that map).

We also need overland connections to help bring that bandwidth to landlocked countries, to help them share and compete with each other, and to network rural areas. Fibre, copper and microwave are the traditional and expensive options, O3B wants to provide a satellite alternative, but the TIER group‘s WiLDNet project has the most disruptive potential in my view, potentially replacing microwave links with something that’s a hundred times cheaper and can be bought off the shelf.

TIER also wants to see their technology used to provide international bandwidth and compete with the undersea cables:

The vision is to connect Gilbraltar, which has low-cost world-class bandwidth and hosting, overland via long-distance Wifi through Morocco/Algerian, Mali, Burkina Faso, to Ghana.  This means crossing the Sahara, which is certainly not trivial.  (Timbuktu is roughly on this path.)  The article said 6 Mb/s, but I am thinking something much higher.  Although this is a crazy idea, I think it is much cheaper than many proposed projects, and if it worked you could grow the network over time and also increase BW for busy links, even moving to fiber once you have the traffic to pay for it.

(reference, video)

Google Broadband

September 9, 2008

According to FT and other sources, Google has announced their support for a new initiative called O3B to “bring internet access to 3bn people in Africa and other emerging markets by launching at least 16 satellites to bring its services to the unconnected” in 2010.

They will… order 16 low-earth orbit satellites… as the first stage in a $750m project to connect mobile masts in a swath of countries within 45 degrees of the equator to fast broadband networks… the project could bring the cost of bandwidth in such markets down by 95 per cent.

This will probably be the largest single investment in developing country network infrastructure in history. The TEAMS submarine cable for East Africa (which is not yet active) will cost $82m to lay according to Wikipedia, while SAT-3 (West Africa) cost $280m. However, the comparable but larger Iridium satellite service, with 77 LEO satellites, was estimated to have cost US$6 billion and filed for bankruptcy in 1999.

According to the FT, “wireless spectrum required for the service had been secured through the ITU“, but this seems unlikely as the ITU works by consensus and not coercion, and it has no power to override local governments’ license demands.

In Kenya for example, a deregulated market, an international gateway operator must pay Ksh 15m ($210,000) for a 15-year license, plus up to $70,000 per year, and probably the same again for VSAT licenses. If this is scaled up from Kenya’s 37 million population to the whole of Africa’s 955 million, O3B might have to pay US$ 18m per year for licenses alone, adding $360m to the cost of the project over the 20-year lifespan of the satellites.

The technical model is interesting. By not dealing directly with end users, but being a bandwidth provider for communications providers, O3B enters a market with Intelsat, Iridium and other satellite operators, providing expensive bandwidth in places where none is available. Their service should have lower latency than the usual geostationary satellites, as their satellites orbit closer to the Earth in MEO (NYT claims only 120 milliseconds, compared to 500 for a geostationary satellite). However:

  • latency will still be worse than land-based connections (I don’t buy NYT’s claim that 120 ms is “close to fiber”)
  • bandwidth will still be limited by available frequency space
  • bandwidth will have to be shared (to some extent) between users of a single satellite
  • ground stations communicating with moving satellites have spectrum efficiency problems (due to the Doppler effect)
  • ground stations with moving (tracking) antennae must switch from one satellite to another every 10 minutes, causing a short dropout
  • satellite communications require more power than microwave links, and in this case are likely to be located in rural areas with no electricity grid
  • the hidden node problem makes satellite more suitable for leased line substitution than Internet access

Latency is likely to remain a problem for voice customers, both mobile networks and Internet access, for which bandwidth demand is largely being driven by VoIP (estimated at 70% of bandwidth use in some African countries). Satellite VoIP providers apparently use different codecs with lower bitrates than standard VoIP, so popular services like Skype may not work well over satellite.

Google and other partners have so far invested $65m of the total $750m sought, and venture capital is being raised for the rest. This proves just how much this bottom of the pyramid Internet market is worth to Google.

Satellite bandwidth is already highly commoditised, and O3B’s plan to reduce this from US$4,000 per megabit per month to $500 requires compelling evidence. In any case this is wholesale bandwidth, not for end users, who will still pay whatever the telco wishes to charge.

I also think that their US$ 750 million investment in satellites will be useless within 20 years of launch due to degradation (NYT claims a 10-15 year life for MEO satellites), whereas building more land capacity in Africa would have much longer-lasting benefits.

Om Malik says: “I’m intrigued by this startup because it does make sense to offer connectivity in remote areas. It also makes sense because Africa is one of the booming cellular markets and one where there is a need for cellular backhaul infrastructure. In remote areas, voice is going to be the killer app for a long, long time. The problem is that this company will always compete with fiber networks in terms of pricing, and that might put them on the back foot.

Google clearly wishes to use this project to enable broadband Internet access in developing regions, but many other things must be in place, including fixed power infrastructure, PCs or OLPCs, technical support and skills, and demand and useful content and services for areas with lower literacy, before that can happen.

Hopefully this will at least increase the spread of mobile and broadband networks by adding backhaul options (competition) and reducing the financial barriers that telcos have to overcome to deploy these networks, but this is far from certain, especially as the telcos would like to offer such services themselves, and may view O3B as a competitor more than a supplier.

See also: