The Trials, Tribulations, and Triumphs of Choosing an M&E Platform

Republished with permission from ICTWorks.org

At the recent MERL Tech conference, Tania Lee (Caktus Group), Tom Walker (Engine Room), Laura Walker McDonald (SIMLab), and Lynnae Day (Oxfam America) led a session called, “The Trials, Tribulations, and Triumphs of Choosing an M&E Platform.” They’ve written up their reflections and learning from that session focusing on project design, tool design/research, and getting things off the ground, whether that means finding external help or building a custom solution.

Choosing a tool: where to start?

Many organizations come to a procurement process with a favourite platform in mind, either from prior experience or because they’ve heard compelling marketing or stories. Starting from the tool means that you are not being true to what really matters – your needs and those of your users.

SIMLab and Caktus Group use an Agile methodology that prioritizes a general sense of direction and strong knowledge of the user, and uses that to get to a prototype that actual users can test as early as possible (whether you’re building a tool or configuring an existing one). Getting feedback on a system’s capabilities while you’re developing it lets you respond as your idea meets reality.

Understand your direction of travel, through workshops with your team, potential users and other stakeholders

• Create and understand the theory of change for the tool thoroughly.
• Write ‘user personas’ – a concise, clear picture of the people you are building for, including their habits, levels of technical skill and working environment.
• Write or draw a process map that shows exactly what information is being exchanged, where it comes from, where it goes and how. This helps you to see potential blockers or impractical elements of your plan.
• From the process map and user personas, develop user stories (individual tasks that the system should be able to perform). Prioritize those, and develop the first few.
• As soon as you have something you can test, show a prototype system to real users.
• When you can, use it ‘live’ in a pilot and continue to build and release new versions as you go.

Things to consider while going through this process:

• Project timeframes: is this system constrained by a particular project timeframe, or does it need to be sustainable beyond the end of your project? Is it an internal operational system – in which case there may be no end date for its use?
• What level of security do you need the system to have? What kind of data will it handle, and what are the legal requirements in the country (or countries) where it will be used?
• What level of user support and documentation do you need?
• Will users need training?
• What level of system analytics are you hoping for?
• How tolerant of risk are you/your organization? For example, do you need to choose an established vendor or product that you know will still be there five years from now?

Challenges

The session acknowledged some of the challenges with this kind of work:

• In large organizations, working on internal projects requires consultation with a wide range of stakeholders.
• Project managers are often trying to bridge gaps in knowledge (and even language) between technical and non-technical people. They may themselves not be technologists.
• Donor funding may not allow the kind of flexibility inherent in an Agile development process. The proposal process requires you to have done much of the design work – or to presuppose its outcome – before you get the funds to support it.
• During the development process, the environment and needs themselves may change.
• No product or system is ever truly ‘finished’.
• Systems development is rarely funded well enough to make something robust that has all non-functional requirements, and allows for maintenance and updates.

What’s already out there?

If you don’t already have a tool in mind, the next step is to conduct a market analysis. A market analysis helps you determine what tools are available and decide whether to use or customize an existing tool, or develop your own custom solution.

There are several common challenges organizations face in their search for M&E tools, including:

• Finding a tool that’s an exact fit for all of your needs, particularly for larger projects; you’re likely to have to pick and choose.
• Finding all of the relevant options quickly and easily, or…
• …when you find a host of similar tools, quickly and easily understanding their specific differences and value-adds.
• Developing user needs and tool requirements in an often-changing context (internal or external).

There are also several common approaches organizations take to navigating these challenges. Some of these include:

• Staff that are familiar with existing tools or have experience with a specific tool that fits the requirements, can do some simple testing and make recommendations.
• Taking to peer organizations to see what tools they’ve used (or not used, and why) for similar processes or projects.
• Hiring consultants or working with IT departments to support the search, selection, and contracting.
• Referencing some of the tool guides that exist (like NetHope or Kopernik), but without relying completely on them for decision-making, as they’re not always up-to-date or have all of the information you need.

Finally, there are some key questions to ask when assessing your final pool of solutions, besides whether they fit the bulk of your minimum user requirements, including:

• How will the user interact with the system? What kind of software/hardware requirements and training will your users need to get the most out of their experience?
• What kind of tech support is available – end user support? customization support? break-fixes? anything at all? And how much will varying levels of support cost?
• What are the up-front costs (for software and hardware)? What are the on-going costs at the current project size? What are the on-going costs at scale (whatever the size of your “scale” is)?
• If you’re planning to scale, how well does the tool adapt to different environments? For example, does it work offline and online? Does the interface support localization in the language that you need? Does it have the capacity to integrate with other tools?

You’ve done the groundwork: what next?

Once you’ve taken the time to understand your needs and look into what tools are already out there, you might find you still need features that an off-the-shelf tool can’t provide. If you don’t have the necessary skills within your organisation, you might need to consider getting technical support from outside your organisation, or building a custom solution.

Finding the right external help, if you need it

During the discussion, many people said that they found it difficult to know where to find good sources of external support. This seems to be a struggle in many contexts: it was also a common theme in the engine room’s research project (supported by Making All Voices Count) into how transparency and accountability initiatives in Kenya and South Africa choose tools.

Sometimes, this is just a question of networks and knowing who to ask. The discussion threw up a collection of useful places to start (such as Pelican Initiative) for this, and we’re currently collecting more. But there are some things that you can do to help pick the right support provider.

Choose criteria that make sense for your organisation. In the discussion, a lot of people said that the most important factor determining whether they could work well with a technical provider was how well the provider understood their organisation’s long-term goals. Providers of technology tools can sometimes focus on the tool itself rather than the organisation and its goals. For a relationship to be successful, there needs to be a clear understanding of how a tool fits into an organisation.

Be clear with your needs

This makes it critically important to present what you need to support providers in a clear, comprehensive way. Whether this takes the form of a formal request for proposals, a short brief or a specification document, communicating your needs to people outside your organisation is actually a good way of narrowing down which features are really essential – and which you can do without.

Writing this in language that technical providers can understand often takes a lot of time and effort. You may already have a lot of the raw material for a good brief if you’ve already documented your needs and got a good idea of existing products, including elements like user personas and user stories (see the Understanding your needs blog, above).

At this point, several members of the discussion said that they had found it helpful to work with a consultant who could translate their needs into technical specifications – which they could then distribute to actual providers. It’s worth looking at existing templates (like this one by Aspiration) and guides on the request-for-proposal process (like this one by TechSoup).

Different organisations will have different criteria for making a selection. Some might want having knowledge of a particular country context, experience in a particular sector or a more general sense of how the organisation was likely to grow and change over time. Others found it more helpful when providers were physically located in the same city as them, or available to answer queries at specific times. Several people in the discussions mentioned times when they hadn’t considered factors like these – and came to regret them later.

Building a custom solution

Building a custom solution, in this context, means building a tool (or integrating a set of tools) that meet needs that no other tools in the marketplace can. Deciding to build a custom tool may require weighing tradeoffs: short-term needs vs. long-term flexibility, specialized usability features vs. time to market, or unique program needs vs. organizational standards.

One of the most often repeated challenges identified by session participants was securing the necessary funding to properly build a custom solution. It can be difficult to convince donors to pay for technical resourcing of a software development project and/or maintenance and support of the system.

Sometimes, organizations will find themselves cobbling together a few different grants to fund an initial prototype of a system. This approach can help to generate additional organizational support, as more programs have a stake in the custom system’s success. It’s also important to consider how and where a custom solution may fit into the organization’s overall strategy – if there is alignment, additional support across various teams could also help to ensure the project’s success.

Custom solutions require custom skills

Managing the custom solution project requires technical skillsets and a good understanding of the software development cycle. Without someone to line up necessary resources, make day-to-day decisions, and provide strategic guidance and technical oversight, a project can easily lose direction and the end result may not meet your needs. It may be helpful to collaborate with other teams within your organization, or to hire a vendor or consultant who can provide project management services.

Session participants also recommended building custom solutions using an iterative or “Agile” approach; this means that smaller sets of functions are tested at various points in the process to collect more useful and direct feedback. An Agile software development approach can help to identify the highest priority features so that a team can spend their time and energy working on what’s important. If this is your organization’s desired approach, it’s important to work with a team that has the right experience working in this way.

Some very helpful guidelines can be found in the Principles for Digital Development, which describes principles for building ICT tools and is a great way to approach building any custom solution.