MakeSense was meant to test feedback loops from “citizen-led sensor monitoring of environmental factors” in the Brazilian Amazon, providing structured, accurate and reliable data to compare against government measurements and news stories in the Amazon basin. The project centered around developing, manufacturing and field testing DustDuino sensors already prototyped by Internews, and developing a dedicated site to display the results at OpenDustMap.
It may seem obvious that it was too ambitious to try to create a mass-produced hardware prototype with two types of connectivity, a documenting website, do actual community engagement and testing (in the Amazon) AND do further business development, all for $60,000, not to mention the coordination required. But, it is also true that typical funds available for innovation lend themselves to this kind of overreach.
Good practice would be to support innovators throughout the process, including (reasonable) investment in team process (while still requiring real-life testing and results), and opportunities for further fundraising based on “lessons” and redesign from a first phase. As well, an expectation that the team be reconfigured, perhaps losing some members and gaining others between stages, plus defining a clear leadership process.
Supportive and intensive incubation, with honest assessment built in through funding for evaluations such as the one we published for this project would go a long way toward better innovation results.
Funders should also require transparency and honest evaluation throughout. If a sponsored project or product cannot find any problems or obstacles to share about publicly, they’re simply not being honest. Funders could go a long way toward making this kind of transparency the norm instead of the exception. In spite of an apparent “Fail Fair”-influenced acculturation toward embracing failure and learning, the vast majority of projects still do not subject themselves to any public discussion that goes beyond salesmanship. This is often in fear of causing donors to abandon the project. Instead, donors could find ways to reward such honest self-evaluation and agile redirection.
Introducing new technology in international development is hard. And all too often, the key details of what actually happened in a project are hidden — especially when the project doesn’t quite go as planned. As part of the MakeSense project team, we are practicing transparency by sharing all the twists, turns and lessons of our work. We hope it is useful for others working with sensors and other technology, and inspires greater transparency overall in development practice.
The MakeSense project was supported by Feedback Labs and the project team included GroundTruth, Internews, InfoAmazonia, Frontline SMS/ SIMLab, and Development Seed. MakeSense was meant to test feedback loops from “citizen-led sensor monitoring of environmental factors” in the Brazilian Amazon, providing structured, accurate and reliable data to compare against government measurements and news stories in the Amazon basin. Over the course of the project, DustDuino air quality sensor devices were manufactured and sent to Brazil. However, the team made several detours from the initial plan, and ultimately we were not able to fulfill our ambitious goals. We did succeed in drawing some important learnings from the work.
- Technical Difficulties are to be expected
Setting up a new hardware is not like setting up software: when something goes wrong, the entire device may have to go back to the drawing board. Delays are common and costly. This should be expected and understood, and even built into the project design, with adequate developer time to work out bugs in the software as well as hardware. At the same time, software problems also require attention and resources to work out which became an issue for this project as well, which often relied upon volunteer backup technical assistance.
- Simplify Technical Know-how Required for Your Device
The project demonstrated that it is important to aim for the everyday potential user as soon as possible. The prototype, while mass-produced, still required assembly and a slight learning curve for those not familiar with its components, and also needed some systems maintenance in each location. Internews plans for the DustDuino’s next stage to be more “Plug-and-play” — most people don’t have the ability to build or troubleshoot a device themselves.
- Consider Data Systems in Depth
This project suffered from a less well-thought-out data and pipeline system, which required much more investment than initially considered. For instance, the sensor was intended to send signals over either Wi-Fi or GSM, but the required code for the device itself, and the destination of the data shifted throughout the project. Having a working data pipeline and display online consumed a great deal of project budget and ultimately stalled.
- Prioritize Data Quality
The production of reliable data, and scientifically valid data, also needs to be well planned for. This pilot showed how challenging it can be to get enough data, and to correct issues in hardware that may interfere with readings. Without this very strong data, it is nearly impossible to successfully promote the prototype, much less provide journalists and the general public with a tool for accountability.
It is important to be intentional about technical vs programmatic allocation, and not underestimate the need for implementation funding. It is often the case that software and hardware development use up the majority of a grant budget, while programmatic and implementation or field-based design “with” processes get short shrift in the inception phase. Decision making about whether to front-load the technology development or to develop quick but rough in order to get prototypes to the field quickly, as referenced in the narrative, should be made intentionally and consciously. Non-technical partners or team members should be aware of the incentives present for technical team members to emphasize hardware/software development over often equally critical local engagement and field testing processes, and ideally have an understanding of the basic technical project requirements and operations. This project suffered from different understandings of this prioritization and timeline.
The anticipation of a need for future funding dominated early conversations, highlighting a typical bind: funding available tends to skew to piloting with no follow-up opportunities for successful pilots. This means that before the pilot even produces its results, organizations must begin to source other funds. So, they must allocate time to business development as well, which can be difficult if not impossible, and face pressure to create marketing materials and other public relations pieces. This can also in some cases (although not with this pilot) lead to very premature claims of success and a lack of transparency. During this project, there was some disagreement among team members about how much to use this pilot fund to support the search for further investment — almost as a proposal development fund — and how much to spend on the actual proof of concept through hardware/software development and field testing.
This is a lesson for donors especially: when looking for innovative and experimental work, include opportunities for scale-up and growth funding or have a plan in mind for supporting your most successful pilots.
A consortium project is never easy. A great deal of time is required simply to bring everyone to the same basic understanding of the project. This time should be adequately budgeted for from the start. Managing such a team is a challenge, and experienced and very highly organized leadership helps the process. FrontlineSMS (which received and managed the funding from Feedback Labs) specifically indicated they did not sufficiently anticipate this extensive requirement. Also, implementing a flat structure to decision making was a huge challenge for this team. Though it was in the collective interest to achieve major goals, like follow-on funding, community engagement, and a working prototype, there were no resources devoted to coordinating the consortium nor any special authority to make decisions, sometimes leading to members operating at cross purposes. Consistent leadership was lacking, while decision-making and operational coordination were very hard given quite divergent expectations for the project and kinds of skills and experience. This is not to say that consortium projects are a poor model or teams should not use a flat structure, but that leading or guiding such a team is a specialty role which should be well considered and resourced.
Part of the challenge in this case was that the lead grantee role in the consortium actually shifted in 2015 from FrontlineSMS to SIMLab, its parent company, when the FrontlineSMS team were spun out with their software at the end of 2014. The consortium members were largely autonomous, without regular meetings and coordination until July 2015, when SIMLab instituted monthly meetings and more consistent use of Basecamp.
Set up clear communications frameworks in advance, including bug reporting mechanisms as well as correction responsibilities. Delays in reporting bugs with Development Seed and FrontlineSMS APIs contributed significantly to the instability of the sensors in the field. Strong information flow about problems, and speedy remote decision-making, was never really achieved. At the same time, efficiency in such consortia is paramount, so that time isn’t taken from operational matters with coordination meetings — so a balance must be struck. This project eventually successfully incorporated the use of BaseCamp.
I’ll confess, I’ve never been a huge fan of M&E. While I absolutely love the data and statistics and numbers and fascinating insights of a good evaluation, as the founder of a nonprofit in Kenya, Map Kibera, doing a quality job on monitoring and much less evaluation was daunting.
Not only because we were under-resourced, and lacked high staff capacity (our members all coming from the Kibera slum) – but Map Kibera was actually set up in part to counter a problem obvious to any Kibera resident. NGOs and researchers were constantly collecting data, but then were usually never seen or heard from again.
Where Does the Data Go?
Was that data even seen again by the collecting organization after their project reports were turned in? What good was all that time and energy spent – on the part of the organization, but more importantly on the part of the good citizens of Kibera, who were tired of answering questions about their income and toilet habits three times in one week? And didn’t they have a right to access the resulting information as well?
Now many organizations, including Map Kibera, our organization GroundTruth Initiative, and others such as the panelists who joined me at the M&E Tech Feedback Loops Plenary: Labor Link, Global Giving, and even the World Bank have put an emphasis on citizen feedback as the core of a new way of doing development.
It’s possible to imagine a world where some of the main reasons for doing monitoring and evaluation are shifted over to citizens themselves – because they want to hold to account both governmental and non-governmental organizations so that the services not only get to the right people, but those people can drive the agenda for what’s needed where.
While a complicated study on, say, the school system and education needs of Kibera people might provide insight, if it sits on a shelf and doesn’t inspire grassroots pressure to shift priorities and improve education, what is the use of it? Why not instead invest in efforts to collect open data on education jointly with citizens, like our Open Schools Kenya initiative?
The benefit here is that the information is open and collectively tended – meaning kept up to date, shared, made comparable with other data sets (like Kenya’s Open Data releases from the government), and used for more than just one isolated study. It’s a way for the community to assess the status of local education itself. In our own neighborhoods and school systems, we wouldn’t have it any other way.
A Citizen Led Future
As noted by Britt Lake of Global Giving on our panel discussion, taking this concept to its logical conclusion there would be a loss of control by development agencies. Is this the real hurdle to citizen-led data collection? To what extent can aid systems be devolved to the people who are meant to primarily benefit?
A truly forward-thinking organization would embrace this shift, because with better information being collected and used at the grassroots, there will be more aid transparency overall and less opportunity for gaming the system. If you think that no small CBO has ever submitted a bogus progress report which went up the chain at USAID, think again.
Nowhere in this system is there incentive to give an accurate account of failures or document intelligent but unpredicted programmatic adaptations and detours that were made. Yet if there’s one thing I know the Kibera people want, it’s for the many organizations they see around them in the slum to be held accountable for all the funds they receive, and for delivering on their years of promises for improving the slum.
In fact, transparency around aid at this hyperlocal level is something we should even feel an ethical obligation to provide. Here’s hoping that in 20 years, the impact of the open data and feedback movements will mean that public information about projects done in the public good is reflexively open and responsive, and M&E as a separate and often neglected discipline is a thing of the past.
This post was originally featured on ICT Works as a Guest Post.
How can all the information about Kenyan schools, including data released by the Kenyan government, and citizen mapping, have a greater impact on education? We’ve been working for the past few months on a project to make information about schools much more available and useful in Kenya. It’s a joint operation between GroundTruth Initiative, Map Kibera, Development Gateway, Feedback Labs, and the Gates Foundation among others.
Many people collect information about education – and they sometimes make it open and free to use. So, why isn’t it easy to find information about a particular school – for a parent, or for an education researcher? Much of the information that’s out there isn’t connected to the other data – and especially when it comes to informal schools, which provide a great deal of the education services in places like informal settlements.
Citizen data – like mapping schools using OpenStreetMap – should also be easy to combine and compare with official education data. And finally, all this info could be accessible and useful to everyone from parents to policymakers.
So, we’ve started with Kibera as a test location for the Open Schools Kenya project.
Over the past few months, the Map Kibera team has engaged parents, school leaders, and education officials in Kibera to find out how the informal school sector can be more visible, and to assess the demand for information on education. Now, a widespread effort is underway by Map Kibera to make sure that the schools data that the team collected a few years ago is still accurate, and to add new info as well. We’re also collecting photos of each school, no matter how small. Every one will have a page on the website, really bringing the informal school sector to light. Formal schools in Kibera will be there too.
Much of the work so far has been around engaging important leaders in the community, who care about local kids getting the best education. Mikel Maron of GroundTruth was recently in Nairobi working on the project and will be updating in a separate post about this busy trip. Ultimately, the community wants to know more about its schools, and to improve them. So do education supporters throughout Kenya.
But beyond this important mission of organizing and making interoperable many data sets across the vast education sector in Kenya, we’re also working on an ambitious hypothesis: that parents and community leaders in education will want to provide feedback on schools, which in turn will inform policy and improve individual schools. Ultimately, our platform will be a place where people can not only be consumers of information, but will provide their own opinions and suggestions on schools, and, importantly, submit corrections and updates to the data on the site. Given the early positive response to these ideas, we’re optimistic that this will be possible in Kibera and also Kenya-wide.
The project is not just about education, either. It has far-reaching potential in other sectors as well. We hope to demonstrate that citizen data, official data, academic research and more can come together and be part of a conversation with those on the ground who feel the impact most of government policy in every sector – ordinary citizens. And, that this kind of conversation means that people “own” their own information, and we can see the beginnings of a true “feedback loop” or dialogue between citizens and government, through the medium of shared data.
This article was originally posted on the Map Kibera blog, July 3, 2014.
On a recent post, I talked about how effective feedback loops are close to being realized in Dar es Salaam.
In this post, I discuss the challenges facing this type of work today – specifically, the serious problems with the current landscape of funding in technology for development, and how we hinder progress in getting from flash-in-the-pan pilots to meaningful change.
In Dar es Salaam, GroundTruth began as a consultant to a pilot initially conceptualized and supported by the World Bank. We’ve supported it to an extent now independently (and voluntarily) for three years. In spite of having very engaged community members, having met with prominent members of government who have a strong interest in the information, and getting some local notoriety, as well as a good amount of international attention, the ultimate work of creating a strong feedback loop has yet to be done.
What’s going on?
Well, here’s how funding tends to operate when it comes to technology for development or feedback projects:
- A tech pilot concept is developed and funded by a large organization or institution, lasting no more than a couple of months. The concept can be initiated by a smaller partner NGO or by the big organization or institution, or even by a funder.
- OR: a project contest, hackathon or app contest is initiated, sponsored and publicized by a funder or large agency. The contest may come with some funding as a reward.
- The winner of the contest or the implementer of the pilot works on the idea, and if there is enough funding, tries it out in the field. This involves – or should involve – real people, and real communities. It’s possible that they get some results, usually in the form of uptake by citizens – there are reports posted to their online reporting tool, SMS sent in, apps tried out, used and maybe even tinkered with based on feedback.
- Blog posts are written about the nascent success, and a conversation is started about what this can potentially contribute to the feedback loop or target social issue. Publicity helps raise awareness of the pilot. Social media lights up, conferences are attended and lightening talks are made.
- That’s it! The funding fades and the world moves on to the next new thing. (Here is a humorous take on this published today by ICT Works).
Of course, there are some attempts to provide sustained funding for important ideas – there’s the Grand Challenge model, for instance. But more commonly, ideas that are proven to be good languish in a post-hype slump, while backers search for the Next Big Thing (or, Next New Thing). In some ways this is a chronic issue in development funding. But when it comes to technology, it’s much worse, simply because the focus tends to be on the technology itself – not on the program design, context or thornier issues in the society which created the problem in the first place. And, as we all know from our own lives, technology is indeed a quick fix and changes almost daily. Suddenly we can communicate instantly with thousands of people or book plane tickets in a few seconds from our phone. Why should it be any different for efficiently solving a social problem? And, shouldn’t any project be almost immediately “scaleable” – taken to a huge number of people or places very quickly – just like an iPhone 5 or Pinterest?
But creating actual social impact with the help of a technology is, clearly, a completely different ballgame. While we should know that, many are blinded by the potential for continuous (and cheap) experimentation which continuously boosts the profile of the associated agencies – simply because the news cycle highlights the “new” and “buzzworthy”.
The temptation of quick and inexpensive (if superficial) impact and great PR is proving to be too much to resist.
Unfortunately, lost in the storm are not only the potentially transformative projects, but the people who took part in the pilot phase in the first place. Those people are the citizens, the residents, the community members, the real people who hoped they had something to gain from putting effort into association with a promising pilot. Is it really responsible – or ethical – to ditch such an effort before it has time to bear fruit? No – which is why many participants in this funding cycle keep trying to serve their public in spite of such immense resource challenges.
And the final blow is that the same funders continue to serve the cycle of the new, while tending to blame the initial developers and implementers for not creating something that’s going to operate on its own sans grant funding. That is, something marketable. Or, that proves its own worth in a matter of months and thereby becomes something the public will pay for, or requires no money because of extensive volunteerism. In my view, this fallacy is akin to saying that public libraries don’t deserve public funding because people should pay to access books and information if they truly value them (or they should manage and run their own free-braries). But that’s another post.
Many technology interventions can indeed create an attractive output that appears online quickly and relatively easily, whether or not the ground reality has changed at all. This is different from most development areas – health, education – whose challenges resist even the illusion of a quick impact.
Where we need to concentrate resources now is on those organizations and individuals who have gotten past the first three, four, five iterations of a technology intervention and that attractive output – the projects where a constituency of support has already been built up at the grassroots level.
This isn’t just us – I’ve met countless inspiring people often working in their own backyards on real feedback loops and real impact from collective citizen voices amplified by technology. Most of the time, what I hear is this same story – their potential is severely limited because after the initial buzz, there wasn’t any more funding. While one might think that a good project will somehow manage to attract the support it needs, that’s just not always the case. There are indeed resources out there which are being spent on technology and development, but they are not being directed toward those people already making a real difference, nor are they targeted at the post-pilot phase – which is not the same as the “scaling up” phase. I would call it the “impact phase” – putting in the hard work needed to create a tangible effect and close the loop of feedback, leaving a real mark on society. This might happen in year 2, 3, or 4 of a project, not year one.
We’ve had the surreal experience of watching a presentation in Washington which happened to show our Dar es Salaam pilot as an example of a great success, while we were chatting online with a project participant and learning that our entire stock of computer equipment had been destroyed in a flood, and therefore any chance of the pilot reaching its potential was nearly eliminated unless we could somehow get them new equipment. We started asking around for donations, and thankfully friends at a small tech company contributed two laptops. The project has now been kept going on just those two laptops for over a year now. This is great – but clearly limited. I think it shows that the nitty gritty of getting any of these ideas to work is ALWAYS much more difficult than it might initially appear. It requires a stronger commitment than anyone anticipates when they first realize that technology could become a game-changer in development.
It’s complicated: in our case, there has indeed been interest to push forward from this pilot on the part of those we started out with at the World Bank. But there are often problems with having large institutions involved with small experimental pilots, which require strong adaptability and agility – within the organization as well as externally. For one reason or another and after various attempts, there hasn’t yet been a successful channel for taking it forward to the impact phase.
Unfortunately, it’s quite common for pilots that have great potential and international fanfare to stall and not go much further than that. I understand the real need for serious thinking and research on how to create that elusive impact – how to complete the loop – and for much, much better evidence and stronger theories of change. But that is the level at which we should be experimenting and piloting by now. And that is where funding needs to be directed. This field is no longer new – there are many, many pilots which need to take the experimental mindset to the next level of closing the loop of impact. Let’s give them our support.