The sad tale of clownfish
This is a story about the little software that could - a tiny application for archiving government reports by CC-ing an email address. It taught me that the biggest challengesin development work are human, not technical. There isn't a real moral here, just a tale from the software and development field.
In my previous employment, I was responsible for overseeing a number of software and data teams maintaining the data collection infrastructure for internal use and for the government of the DRC. Among these tools was the national health information system powered by DHIS2, which records public health data from health facilities nationwide. The data originates as paper-based records in health facilities that are transcribed by data managers at regional health zone offices on a monthly basis.
In an effort to improve data quality, the Ministry of Public Health introduced the idea of "Feedback Reports" (Rapport de Retroinformation). A data analyst at the provincial health office was responsible for reviewing the monthly submissions in DHIS2 and preparing reports that would be sent to the health zone offices, commenting on the overall health trends, data quality, and suggesting improvements. By aligning the consumers and producers of the data, the health zone data managers would learn what to look for in their data, take greater ownership of their data, and investigate and correct mistakes pointed out in these feedback reports. It's also just nice to know that someone cares. These reports were sent by email to the health of the health zone office.
Since my team was responsible for much of the public health data, we were asked to develop a system to archive these electronic reports. In order to reduce the burden on users, my idea was to introduce a new email address (e.g. reports@moh.cd) that would receive the message, download the attachment, file it away on a Google Drive / OneDrive site, and then share the report with all parties copied on the original email. So was born Clownfish.
Over time, I added a few simple features, including a plain HTML dashboard that added a log of the emails received and allowed users to modify the file names. Similarly, instead of "sharing" the file via some API, clownfish just replied with a public link to the file. This would allow users to to access the files from whatever account they wanted, rather than getting hit with a "your account no longer has access" if they changed accounts.
The uphill battle of development
The challenge of building clownfish wasn't software development. I had whipped up a prototype over the weekend, and spent a few months refining it the user base grew. The real challenge was internal politics.
During the initial design and implementation of clownfish, I used Google Drive as the backend for storing files. This was handy since the vast majority of my users had gmail addresses and were familiar with Google Drive. However, my employer was in the midst of a merger, and the new IT management only knew how to work Office 365. So, we all needed to transition to Office 365, my little application included. Turns out, there wasn't a way to upload a file to OneDdrive and share with users outside of the organization, according to the new head of IT. I lobbied extensively to allow this restriction for my little clownfish, since this was the explicit purpose of the application. However, after six weeks I was no closer to getting authorization, and I quietly spun up my own NextCloud instance on DigitalOcean and rewrote the application to do use that as the backend. It wasn't as nice - fewer users were familiar with it, and added an additional maintenance burden to my small team.
With the data storage solved, the next challenge came from a very reasonable user request. Rather than using the "clownfish" name for the email address, a data manager in the Kasai suggested the email handle should be "retro" and it should be sent from an official mail server instead of a gmail handle. This was a really good idea, and I was glad to get feedback from users to improve the application. So, I kicked the request up the chain to set up the dedicated email address in Outlook and worked on the code changes to make this possible.
Oh, how naive was I! For a similar reason as the OneDrive roadblocks, my organization wasn't able to set up an email address for my application. I do not remember the rationale fully, but I believe it was against the organization's policy to create an address for someone who wasn't an employee. Or maybe they didn't want the server to speak SMTP. Whatever the case, this was not a feature that they would support. So what happened? You guessed it. After another several weeks of emails back and forth to the head of IT, explaining this was in our development contract, explaining the use case, arguing for an exception, I got tired. My software team had a mail server we maintained for a different application, and we spun up an address on that mail server, working around our orgs IT policies to suite our users. They were happy, I'm assuming our IT was happy, and I was frustrated.
I wish I could say that was the end of the email story. Despite our attempts to whitelist the mail server with places like SpamHaus, we could almost never deliver emails to our own internal MS Exchange email addresses. Our organization paid for some fancy tool that detected and blocked spam, and nearly always marked out messages as such, meaning the project managers who needed insight into these feedback reports, almost never got them. Our external users, on yahoo, gmail, and the like, didn't have any issue, but we were hard pressed to get the internal users onboarded. We raised tickets with our IT to whitelist the domain, and that would work for a few weeks, but afterwards we would find ourselves blocklisted again. We never did come up with a good solution for this issue.
Epilogue
With its own mail server and data storage solution completely outside of our internal IT department, clownfish ran happily for a few years. I left the company in 2021, and provided handover notes to my successor, Bruce. Later, the organization underwent budget cuts and Bruce reached out to me for advice about reducing hosting costs and mothballing software projects. I logged in to clownfish to discover five years of detailed reports, faithfully sent by the provincial health office in the Kasai, all archived in their proper folders. It's amazing when something just keeps working, long after you leave it.
A few months later, as part of my dissertation work, I decided to check in on clownfish again to see if I could request access to the reports for qualitative analysis. It had been shut down during the budget cuts, so I reached out to Bruce to see if I could at least get access to the NextCloud dataset. To my dismay, I learned it had been deleted - all the carefully crafted reports were gone.
I've asked myself over the years what lessons I could learn from clownfish.
It is clear, through the clownfish experience and other experiences, there was a disconnected between the head office in DC that set IT policy, and those of us in the field that responded to grant proposals, implemented projects, and designed technical solutions. This is a tension that exists in many international development organisations. The intervention, designers thereof, implementers, technicians, governmental partners, and the populations participating in the intervention are all located far from the home office, so what is the role of the home office? In a normal organisation, it is top management that dictates the direction of the company, but in the case of international development, it is often the opposite. Technicians on the ground will determine what is technically feasible, country office management, in collaboration with the Ministry partners, will decide what is political feasible, and those closer to the populations in theory benefiting will determine what is socially feasible. The organisation is then left in a position where upper management is being told by those lower on the totem pole what to do (e.g. "source these medicines", or "use this technical solution", or "make this exception to our standard policy for XYZ reason"). I cannot imagine that this is a comfortable position for the home office.
In the case of clownfish, the disconnect between the home office and the field teams led to the field team working away from the home office team. Rather than the home office viewing itself as a facilitator of the work, they instead sought to control the work by imposing restrictions and creating hurdles to jump through. I believe they were well-intentioned. The move to O365 might have made sense from a corporate perspective, and the security policies might have been rationale in a vacuum. However, the raison d'etre of the organisation was not to use a particular office product, it was to respond to the needs of the Congolese population, and clownfish's successes in this regard were in spite of the organisation, rather than facilitated by the organisation.
There isn't really a clean moral of the story to end on. Could I have done something different? Maybe I should have traveled to the home office and articulated the needs and goals of the project. Maybe the organisation should have worked in the institutional culture to ensure that the Congolese population was always seen as the key stakeholder. I'm not sure. All I know is that clownfish served its role faithfully until the day the lights were turned out on its server rack. I guess that was the best it could have ever hoped for.
