Oct 20, 2024
Making Impact Evaluation Accessible - Research & Scope Document
WakeUp Labs conducted in-depth research and interviews with Badgeholders as part of Milestone 1 for the "Making Impact Evaluation Accessible" Mission, aiming to improve evaluation frameworks for Optimism’s Retro Funding process. The study identified challenges in the current system, such as high workloads and lack of standardization, and proposed an iterative, experimentation-based methodology to streamline evaluations and increase effectiveness. Findings will inform a new framework and platform that centralizes data, highlights best practices, and supports future improvements to RetroPGF rounds.
Summary:
This document aims to summarize and outline the work completed in the first Milestone by WakeUp Labs as part of the Mission Request “Making Impact Evaluation Accessible,” proposed by LauNaMu on January 19, 2024, and awarded to the WakeUp Labs Team during Cycle 19 in late March 2024.
Milestone 1 consisted of conducting online research and interviewing Badgeholders to analyze their voting methodologies in RetroPGF — such as the one proposed by LauNaMu — with the goal of proposing an improved impact evaluation framework aligned with the objectives of the current season. Additionally, we aimed to assess whether these methodologies had been effectively considered in the new RetroPGF.
This approach allows us to fully understand the current workflow, identify opportunities for improvement, and find ways to streamline and optimize tasks by focusing only on the most impactful ones for evaluators.
The goal is to eliminate processes that do not add value to project evaluation, and to propose a new methodology. We also investigated similar solutions that could inspire or complement our proposal.
This article with key findings and best practices will be shared through a blog, LinkedIn, and X (Twitter), and will be proactively distributed to the community to maximize outreach.
This document can also serve as a valuable resource for future RetroFunding applicants, helping them achieve better results or avoid unnecessary applications.
Moreover, the OP Collective has shown interest in initiatives like this on several occasions, both through Mission Requests driven by the Foundation (1), (2), and through the DAO, with funding from Retro Grants or by aligning Intents in this direction.
This topic will also be addressed in our conclusion.
Some key data points were considered to carry out this research and achieve meaningful results:
At least 25 Badgeholders were contacted for this investigation.
Out of a total of 146 delegates for RetroPGF3, which is not expected to increase in the short term, this represents 17% of the total delegates.
Ideally, we aim to interview 30 Badgeholders, keeping in mind the Central Limit Theorem.
As of the publication of this document, 6 interviews have been conducted, representing 5% of the total Badgeholders.
However, this research may evolve to include more interviews, so these numbers are subject to change, and anyone within the OP Collective is invited to participate.
In any case, the articles reviewed also provide significant insights that contribute to the impact of this research.
In addition to specific research articles, official OP Foundation documentation and forum posts were considered. Among the most relevant sources, we highlight:
Before we begin, it is important to provide a brief clarification on how the Optimism Collective is organized, as this research may be useful for users unfamiliar with the ecosystem.
The Optimism Collective is a coalition of companies, communities, and citizens working together to reward public goods and build a sustainable future for Ethereum.
The Optimism Collective is bound by a mutually beneficial pact, a vision that can be summed up by the axiom Impact = Profit.
It is essential to understand that:
Governance of the Optimism Collective began with the launch of the OP token and the Token House.
As members of the Token House, OP holders are responsible for submitting, deliberating, and voting on various types of governance proposals. In carrying out these functions, OP holders may either vote directly or delegate their OP voting power to someone else.
In simple terms, the Token House is managed by OP Holders and Delegates.
The Token House consists of OP token delegates. Anyone can become a delegate, although the most influential are the top 100 delegates, who are the most active in the forums and are well-informed about all governance proposals. They vote on Agora and participate in shaping Mission Requests, determining grant funding, or deciding how much OP is allocated in Retroactive Public Goods Funding (Retro Funding) rounds. While expertise is valuable, delegates can also express their opinions on the forum without being governance experts, simply to provide input on key decisions.
The Citizens' House is a large-scale experiment in reputation-based, one person = one vote governance, and is responsible for Retroactive Public Goods Funding (Retro Funding).
Retro Funding is the primary economic engine of the Collective, designed to reward those who have created positive impact across the Collective and the Superchain. The concept behind Retro Funding is simple: it’s easier to agree on what has been useful in the past than to predict what might be useful in the future.
The individuals who participate in the Citizens' House are known as Badgeholders.
The Citizens' House is exclusively responsible for Retro Funding. It operates almost like a private group of key contributors within the Optimism ecosystem. Members of the Citizens' House are responsible for selecting new Citizens, though the selection process is currently closed. Unlike Token House delegates, Citizens do not need to be highly active in forums. Their primary responsibility is to stay informed about key developments relevant to their role, such as new voting mechanisms, Retro Funding schedules, and proposal formats.
Within the Optimism Collective, two important organizations provide operational support:
OP Labs, which is responsible for the core software development.
The OP Foundation, a Cayman Islands foundation that supports the establishment of the Optimism Collective, the development of the Optimism ecosystem, and the technology that powers it.
About the Retro Funding:
Since the introduction of Badgeholders and the corresponding mechanisms for funding projects, efforts have been made to improve these processes through experimentation. Each Retro Funding cycle sees changes in parameters, point allocation methods, categories, and the selection of individuals responsible for administering the process.
This experimentation has the potential to yield significantly better results than simple iteration or incremental improvements—it’s like aiming for a 10x improvement instead of just 1.25x. However, it can also lead to less effective outcomes or the removal of previously successful methods. Additionally, it often results in overwork for participants, as they must also understand these new concepts before they can effectively engage. This is why retrospective periods are so important; they allow for reflection and learning, helping to build a stronger foundation for future cycles.
Interview and Outreach Process:
To conduct this research, we compiled a list of Badgeholders known by the WakeUp Labs team, consisting of around 10 individuals.
Naturally, our selection was closely tied to the Optimism community in Argentina and Latin America, primarily due to geographical proximity.
After reaching out to these individuals and informing them about the project, we confirmed their availability and conducted interviews lasting around 45 minutes. These interviews included questions we deemed relevant, aiming to spark discussions on topics we were interested in exploring. Gonzacolo.eth, co-founder of WakeUp Labs lead the interviews.
Given the diverse backgrounds of the participants, we allowed each interviewee to steer the conversation toward areas where they could provide the most value.
As a result, not everyone answered all the questions, as the questions were designed to serve as conversation starters rather than a rigid script. This approach made our analysis and conclusions more efficient.
Some of the reference questions used in the interviews were as follows:
Research Questions:
How long have you been a Badgeholder, and when did you start actively participating in the Retroactive Grants process?
Have you formed groups with other Badgeholders to divide the voting workload?
Where do you see the greatest challenges in the current process?
Changes in OP's Current Paradigm Through Experiments:
What’s your opinion on this?
What do you foresee for the future?
Where do you get information about updates for the next Retro or new working methods?
Do you have any insights into key factors for quickly evaluating a high-quality project? This can also include negative aspects that might lead you to reject a project.
Do you consider specific user behaviors when adding or subtracting points? For example:
Is project participation in the forum part of your evaluation?
Do you value projects that actively engage in governance?
Do you consider the project’s geolocation?
Do you assess whether the project deploys only on OP or across multiple chains?
Do you take into account if the project is women-led?
Do you follow any particular work methodology?
Do you think the current number of Badgeholders is adequate, or should the roles be expanded?
These questions were adjusted depending on the interviewee, and not all of them were necessarily asked.
As we conducted more interviews and refined the project scope, we decided to add the following question:
A question more focused on the framework's conclusion: Do you think the approach to conducting new retrospectives, along with their iterations, is well-executed? What do you think could be improved?
This allowed us to follow a north star that we will address in the conclusions and next steps after completing the interviews.
It’s important to mention that some Badgeholders were unavailable for interviews.
However, we successfully conducted 6.
We also asked interviewees if they knew other Badgeholders outside our immediate circle to expand our sample.
As a result, we completed a list of 33 individuals and interviewed 6 of them.
We can now move on to some conclusions, considerations, and the current challenges faced by these individuals:
Current Challenges Faced by Badgeholders:
Badgeholders (citizens of OP) face significant challenges when voting, largely due to the dynamic and ever-changing context of each Retro Funding cycle, coupled with a high workload.
As the ecosystem continues to grow, with the constant addition of both new and returning participants, the task of categorizing, scoring, and providing feedback becomes increasingly complex.
While performing a minimal evaluation is straightforward, doing it correctly requires thoroughly analyzing each proposal, understanding who is applying, and evaluating them appropriately.
Therefore, dedicating the necessary time to this task involves a high degree of workload, a deep understanding of the overall ecosystem context, and the ability to evaluate projects accurately.
Moreover, Badgeholders often hold demanding jobs within the Web3 space, and in many cases, they are involved in more than one role or project.
Conclusion 1 - Diversity of Opinions:
Since Badgeholders are designed to be unique individuals who do not need to dedicate all their time to Optimism and also follow their own projects and initiatives, it was fascinating to gather their perspectives and see how they differ. Initially, we thought the group would be much more homogeneous, but in reality, their opinions, ways of thinking, and decision-making methods turned out to be quite diverse. If the Citizens were intended to be individuals with their own decision-making power and reasoning, the system is functioning exactly as intended.
This also allows each Badgeholder to consider specific factors at the end of the rounds, evaluating applications and determining a specific amount for each project. We would not standardize this decisive feature and would leave it up to the Badgeholders to determine the final value through voting.
Conclusion 2 - Experimentation Challenges:
As several Badgeholders stated in their interviews, while experimentation is positive and there is a “Reflection” period similar to a "Retro" in agile methodologies, arriving at a first conclusive opinion can be complicated.
We believe the “Retro” period must be well-structured and carefully worked through to reach key insights and improvements.
This becomes even more critical given the current experimentation mechanism implemented by the Optimism Collective.
Experimentation allows for the creation of unique cases, entirely different from what has been done before, with the potential for 10x results instead of simple incremental improvements.
However, as some Badgeholders noted, a final, stable version has not yet emerged. We believe it's not worth discarding the good elements of a previous version just for the sake of experimentation, so that each "Round" isn't an entirely new process requiring a fresh preparation.
An iterative mechanism that allows for experimentation could be:
Retro “T-1,” the version before implementing this progressive structure.
Test with Sample A, which is a new experiment, entirely different from the previous version.
Test with Sample B, which is an iteration of previous practices, building on the lessons learned and positive results from the previous Retro.
In the reflection period, form Sample C, which combines the results from A and B.
And will be used as the Sample B.
In this way, they will achieve a more advanced iterative process that yields better results; this theory will be our proposed framework.
What's most interesting is that as the number of trained BadgeHolders increases, you can easily run several experiments simultaneously and evaluate results even faster.
Conclusion 3 - Filtering and Categorization Processes:
Almost all Badgeholders follow a two-stage process before starting their evaluations:
First, they purge "junk" projects, "scammers," or those irrelevant to Optimism’s goals, allowing them to focus on projects that truly add value to the network. Many see this as the most critical step—filtering out applications that are merely trying to obtain OP without delivering real impact. This initial filter requires reviewing all applicants.
Currently, there are Badgeholders specifically dedicated to this filtering process.
However, there have been errors in both directions:
Projects miscategorized as scams or low-quality.
Meme tokens awarded for generating gas fees for the network, rather than supporting impactful projects.
The second step involves categorizing projects/types of projects. This is something the Collective is already doing with recent versions of the Retro.
Projects are segmented not only by category—such as DeFi, Wallets, Governance, DevTooling, etc.—but also by subcategories and differentiated by metrics.
This is essential because narrowing the scope helps reduce the number of irrelevant projects.
This is how the 'Retroactive Grants 5' will be segmented by categories.
Important Note: As the crypto ecosystem grows, generic categories are becoming too broad. One thing Retro is doing well is further refining these categories. For example:
DeFi → Too broad.
DeFi → Swaps Protocols → Quite broad.
DeFi → Lending Protocols → Quite broad.
DeFi → Lending Protocols → Fixed Rate → Narrow.
Narrowing the scope by both category and sub-type allows for more focused evaluations. While it may exclude some projects, it helps in refining the quality of assessments.
Conclusion 4 - Alignment with the Foundation and the "Collective Feedback Commission":
While this Mission Request is still in development, Badgeholders made it clear that new tools and formats for evaluating projects that could be considered for RetroActive Grants are important. The challenge we see is that for this to be truly worthwhile, there must be greater alignment with the Foundation, through a Foundation Mission Request, and full immersion with the key players (Badgeholders). The context is so dynamic that these concepts can quickly become outdated unless they are 100% aligned with the aforementioned stakeholders. This should be a more collaborative process rather than an isolated effort.
Complications with Parallel Platforms: It’s important to be cautious of unofficial platforms that may complicate the process. It would be advisable for verification and review tools to be approved by OP Labs or the Foundation to avoid duplicating efforts or using unofficial platforms that could cause confusion.
We don't believe it's worth funding projects, even if they are public goods, that do not meet item 1.
Conclusion 5 - Focusing on Median-Based Weighting:
Different Badgeholders use various strategies to minimize outliers. Some rely on tools like ChatGPT for input, while others tend to adjust their points based on projects they like the most or least.
This can be visualized through a weighted curve approach, where outliers at both the top and bottom are adjusted:
A. Reduce the benefits given to projects at the top. B. Provide a bit more support to those at the bottom, or exclude them completely.
Conclusión 6 - Valuing Positive and Meaningful Participation in Governance and Technical Contributions:
While active participation in the community is not a determining factor, most Badgeholders consider it a plus. Its relevance depends on the specific topic. However, suspicious or low-value participation that simply generates noise or interaction without contributing value is heavily penalized.
Future Considerations Based on the Current State of Retro Funding:
Many of the suggestions mentioned here have already been implemented.
After reviewing the platform used for Retro 5, several mechanisms seem well-implemented and moving in the right direction, such as:
The option for Badgeholders to opt out.
Improved filtering mechanisms.
The reduction of categories.
A more streamlined token distribution process.
Introducing on-chain ranking and point systems to determine new Badgeholders, or using a “Guest” logic for temporary participants, seems like a positive step forward.
The ability to quickly evaluate projects with a ranking from “Very High” to “Low,” as is now possible and was suggested by LauNaMu in their article, makes a lot of sense.
It could be interesting to consider allowing inactive Badgeholders to delegate their voting power to others, potentially with a small multiplier. While this could lead to some centralization of power, it would be particularly useful for complex topics that not all Badgeholders may fully understand, such as highly technical research or specific elements of the OP Stack.
Given these developments, our focus in the Mission has shifted more towards the ideas outlined in Conclusion 3, specifically the development of a new framework that we believe is important to consider for the future. This includes showcasing data from past Retros and proposing a framework to implement this new methodology.
Although the platform is yet to be built, the core objectives for a first screening tool would be:
Filtering out applicants, with the option to hear other Badgeholders' reasons for such decisions.
Reorganizing applicants by categories.
Normalizing or applying Winsorization to address extreme cases and outliers.
However, since the official "voting" platforms already address these needs and have access to project information via API/forms without requiring manual data entry, our platform will focus on:
Displaying all relevant information from previous Retros, up to Retro 5.
Centralizing details on how to vote, what to vote for, and sharing relevant articles, research, and best practices.
Offering a page to propose our new mechanism and framework.
Including an interactive section for feedback where users can share their conclusions from previous Retros, helping consolidate information that can be used in future iterations.
Building a platform to streamline feedback and retrospectives in this manner would be valuable, though it may exceed the current scope of our Mission Request
New Iterative Improvement and Experimentation Framework
This process is designed to achieve excellence in a specific activity (let’s call it A) through an approach that combines continuous improvement and experimentation. It draws on elements from the Lean Startup method and the explore-exploit trade-off theory.
Objective:
To become the best at A through an iterative cycle that combines continuous learning (exploiting what is known) and the exploration of new possibilities (experimenting with the unknown).
Process Steps:
Create the Basic Initial Version (Version I - Minimum Viable Product)
Develop a basic version of A. This first version captures the essential elements needed to start working in this area.
This acts as the Minimum Viable Product (MVP)—not perfect, but sufficient to gather data and experiences.
Iterate and Improve on What is Known (Version II - Exploit Learning)
Analyze and improve Version I using the feedback and data gathered.
Make incremental improvements to optimize the process, fixing errors and enhancing what works.
This is the exploit phase, akin to the Lean Startup method, where rapid iterations refine the current version.
Experiment with Something New (Version III - Explore New Options)
Simultaneously, develop a new version of the activity, entirely different from Version II.
This version is based on novel ideas or gut feelings and may appear random, but the goal is to explore new ways of doing things that might yield better results.
Experimentation allows for innovation beyond the boundaries of the known.
Compare and Combine the Best (Version IV - Synthesis and Evolution)
After testing both Version II and Version III, compare the two to identify the most effective aspects.
Synthesize the best elements into a new version (Version IV), combining the lessons learned from the exploit phase and the best ideas from the explore phase.
Repeat the Process: Continuous Iteration and Parallel Experimentation
The cycle repeats, creating new versions based on:
Iterating on the previous version to continue improving (exploiting what has been learned).
Running new experimental versions to test hypotheses and explore new possibilities (exploring what is unknown).
By conducting multiple experiments in parallel, the process accelerates, enabling faster learning and closer alignment with the goal of becoming the best at A.
Importance of the Explore-Exploit Trade-off
This process balances the need to exploit what is known to work with the necessity of exploring new possibilities that could lead to significant breakthroughs.
Exploit (Versions II, IV, VI, etc.): Use existing knowledge to optimize and improve efficiency.
Explore (Versions III, V, VII, etc.): Introduce new ideas or approaches that could bring transformative value or new perspectives to activity A.
Conclusion:
This iterative process of continuous improvement and experimentation blends the logic of Lean Startup (rapid iteration and improvement) with the explore vs. exploit approach (balancing known improvements with novel ideas). By evolving through both incremental iterations and disruptive experiments, we move closer to excellence in activity A.
Final Notes
This concludes our research for this milestone. We hope it provides valuable insights, and we are open to feedback and suggestions.
If you would like to participate in an interview or share your perspective, please feel free to contact us. This is an iterative document, so we are happy to make updates as needed. You can reach us via social media or directly on my personal Telegram: @gonzacolo.
This is only the first milestone of the project: Making Impact Evaluation Accessible, and we look forward to achieving more milestones soon.
All updates will be published in the relevant Update Thread on the Optimism forum.
Thank you for reading.
Summary:
This document aims to summarize and outline the work completed in the first Milestone by WakeUp Labs as part of the Mission Request “Making Impact Evaluation Accessible,” proposed by LauNaMu on January 19, 2024, and awarded to the WakeUp Labs Team during Cycle 19 in late March 2024.
Milestone 1 consisted of conducting online research and interviewing Badgeholders to analyze their voting methodologies in RetroPGF — such as the one proposed by LauNaMu — with the goal of proposing an improved impact evaluation framework aligned with the objectives of the current season. Additionally, we aimed to assess whether these methodologies had been effectively considered in the new RetroPGF.
This approach allows us to fully understand the current workflow, identify opportunities for improvement, and find ways to streamline and optimize tasks by focusing only on the most impactful ones for evaluators.
The goal is to eliminate processes that do not add value to project evaluation, and to propose a new methodology. We also investigated similar solutions that could inspire or complement our proposal.
This article with key findings and best practices will be shared through a blog, LinkedIn, and X (Twitter), and will be proactively distributed to the community to maximize outreach.
This document can also serve as a valuable resource for future RetroFunding applicants, helping them achieve better results or avoid unnecessary applications.
Moreover, the OP Collective has shown interest in initiatives like this on several occasions, both through Mission Requests driven by the Foundation (1), (2), and through the DAO, with funding from Retro Grants or by aligning Intents in this direction.
This topic will also be addressed in our conclusion.
Some key data points were considered to carry out this research and achieve meaningful results:
At least 25 Badgeholders were contacted for this investigation.
Out of a total of 146 delegates for RetroPGF3, which is not expected to increase in the short term, this represents 17% of the total delegates.
Ideally, we aim to interview 30 Badgeholders, keeping in mind the Central Limit Theorem.
As of the publication of this document, 6 interviews have been conducted, representing 5% of the total Badgeholders.
However, this research may evolve to include more interviews, so these numbers are subject to change, and anyone within the OP Collective is invited to participate.
In any case, the articles reviewed also provide significant insights that contribute to the impact of this research.
In addition to specific research articles, official OP Foundation documentation and forum posts were considered. Among the most relevant sources, we highlight:
Before we begin, it is important to provide a brief clarification on how the Optimism Collective is organized, as this research may be useful for users unfamiliar with the ecosystem.
The Optimism Collective is a coalition of companies, communities, and citizens working together to reward public goods and build a sustainable future for Ethereum.
The Optimism Collective is bound by a mutually beneficial pact, a vision that can be summed up by the axiom Impact = Profit.
It is essential to understand that:
Governance of the Optimism Collective began with the launch of the OP token and the Token House.
As members of the Token House, OP holders are responsible for submitting, deliberating, and voting on various types of governance proposals. In carrying out these functions, OP holders may either vote directly or delegate their OP voting power to someone else.
In simple terms, the Token House is managed by OP Holders and Delegates.
The Token House consists of OP token delegates. Anyone can become a delegate, although the most influential are the top 100 delegates, who are the most active in the forums and are well-informed about all governance proposals. They vote on Agora and participate in shaping Mission Requests, determining grant funding, or deciding how much OP is allocated in Retroactive Public Goods Funding (Retro Funding) rounds. While expertise is valuable, delegates can also express their opinions on the forum without being governance experts, simply to provide input on key decisions.
The Citizens' House is a large-scale experiment in reputation-based, one person = one vote governance, and is responsible for Retroactive Public Goods Funding (Retro Funding).
Retro Funding is the primary economic engine of the Collective, designed to reward those who have created positive impact across the Collective and the Superchain. The concept behind Retro Funding is simple: it’s easier to agree on what has been useful in the past than to predict what might be useful in the future.
The individuals who participate in the Citizens' House are known as Badgeholders.
The Citizens' House is exclusively responsible for Retro Funding. It operates almost like a private group of key contributors within the Optimism ecosystem. Members of the Citizens' House are responsible for selecting new Citizens, though the selection process is currently closed. Unlike Token House delegates, Citizens do not need to be highly active in forums. Their primary responsibility is to stay informed about key developments relevant to their role, such as new voting mechanisms, Retro Funding schedules, and proposal formats.
Within the Optimism Collective, two important organizations provide operational support:
OP Labs, which is responsible for the core software development.
The OP Foundation, a Cayman Islands foundation that supports the establishment of the Optimism Collective, the development of the Optimism ecosystem, and the technology that powers it.
About the Retro Funding:
Since the introduction of Badgeholders and the corresponding mechanisms for funding projects, efforts have been made to improve these processes through experimentation. Each Retro Funding cycle sees changes in parameters, point allocation methods, categories, and the selection of individuals responsible for administering the process.
This experimentation has the potential to yield significantly better results than simple iteration or incremental improvements—it’s like aiming for a 10x improvement instead of just 1.25x. However, it can also lead to less effective outcomes or the removal of previously successful methods. Additionally, it often results in overwork for participants, as they must also understand these new concepts before they can effectively engage. This is why retrospective periods are so important; they allow for reflection and learning, helping to build a stronger foundation for future cycles.
Interview and Outreach Process:
To conduct this research, we compiled a list of Badgeholders known by the WakeUp Labs team, consisting of around 10 individuals.
Naturally, our selection was closely tied to the Optimism community in Argentina and Latin America, primarily due to geographical proximity.
After reaching out to these individuals and informing them about the project, we confirmed their availability and conducted interviews lasting around 45 minutes. These interviews included questions we deemed relevant, aiming to spark discussions on topics we were interested in exploring. Gonzacolo.eth, co-founder of WakeUp Labs lead the interviews.
Given the diverse backgrounds of the participants, we allowed each interviewee to steer the conversation toward areas where they could provide the most value.
As a result, not everyone answered all the questions, as the questions were designed to serve as conversation starters rather than a rigid script. This approach made our analysis and conclusions more efficient.
Some of the reference questions used in the interviews were as follows:
Research Questions:
How long have you been a Badgeholder, and when did you start actively participating in the Retroactive Grants process?
Have you formed groups with other Badgeholders to divide the voting workload?
Where do you see the greatest challenges in the current process?
Changes in OP's Current Paradigm Through Experiments:
What’s your opinion on this?
What do you foresee for the future?
Where do you get information about updates for the next Retro or new working methods?
Do you have any insights into key factors for quickly evaluating a high-quality project? This can also include negative aspects that might lead you to reject a project.
Do you consider specific user behaviors when adding or subtracting points? For example:
Is project participation in the forum part of your evaluation?
Do you value projects that actively engage in governance?
Do you consider the project’s geolocation?
Do you assess whether the project deploys only on OP or across multiple chains?
Do you take into account if the project is women-led?
Do you follow any particular work methodology?
Do you think the current number of Badgeholders is adequate, or should the roles be expanded?
These questions were adjusted depending on the interviewee, and not all of them were necessarily asked.
As we conducted more interviews and refined the project scope, we decided to add the following question:
A question more focused on the framework's conclusion: Do you think the approach to conducting new retrospectives, along with their iterations, is well-executed? What do you think could be improved?
This allowed us to follow a north star that we will address in the conclusions and next steps after completing the interviews.
It’s important to mention that some Badgeholders were unavailable for interviews.
However, we successfully conducted 6.
We also asked interviewees if they knew other Badgeholders outside our immediate circle to expand our sample.
As a result, we completed a list of 33 individuals and interviewed 6 of them.
We can now move on to some conclusions, considerations, and the current challenges faced by these individuals:
Current Challenges Faced by Badgeholders:
Badgeholders (citizens of OP) face significant challenges when voting, largely due to the dynamic and ever-changing context of each Retro Funding cycle, coupled with a high workload.
As the ecosystem continues to grow, with the constant addition of both new and returning participants, the task of categorizing, scoring, and providing feedback becomes increasingly complex.
While performing a minimal evaluation is straightforward, doing it correctly requires thoroughly analyzing each proposal, understanding who is applying, and evaluating them appropriately.
Therefore, dedicating the necessary time to this task involves a high degree of workload, a deep understanding of the overall ecosystem context, and the ability to evaluate projects accurately.
Moreover, Badgeholders often hold demanding jobs within the Web3 space, and in many cases, they are involved in more than one role or project.
Conclusion 1 - Diversity of Opinions:
Since Badgeholders are designed to be unique individuals who do not need to dedicate all their time to Optimism and also follow their own projects and initiatives, it was fascinating to gather their perspectives and see how they differ. Initially, we thought the group would be much more homogeneous, but in reality, their opinions, ways of thinking, and decision-making methods turned out to be quite diverse. If the Citizens were intended to be individuals with their own decision-making power and reasoning, the system is functioning exactly as intended.
This also allows each Badgeholder to consider specific factors at the end of the rounds, evaluating applications and determining a specific amount for each project. We would not standardize this decisive feature and would leave it up to the Badgeholders to determine the final value through voting.
Conclusion 2 - Experimentation Challenges:
As several Badgeholders stated in their interviews, while experimentation is positive and there is a “Reflection” period similar to a "Retro" in agile methodologies, arriving at a first conclusive opinion can be complicated.
We believe the “Retro” period must be well-structured and carefully worked through to reach key insights and improvements.
This becomes even more critical given the current experimentation mechanism implemented by the Optimism Collective.
Experimentation allows for the creation of unique cases, entirely different from what has been done before, with the potential for 10x results instead of simple incremental improvements.
However, as some Badgeholders noted, a final, stable version has not yet emerged. We believe it's not worth discarding the good elements of a previous version just for the sake of experimentation, so that each "Round" isn't an entirely new process requiring a fresh preparation.
An iterative mechanism that allows for experimentation could be:
Retro “T-1,” the version before implementing this progressive structure.
Test with Sample A, which is a new experiment, entirely different from the previous version.
Test with Sample B, which is an iteration of previous practices, building on the lessons learned and positive results from the previous Retro.
In the reflection period, form Sample C, which combines the results from A and B.
And will be used as the Sample B.
In this way, they will achieve a more advanced iterative process that yields better results; this theory will be our proposed framework.
What's most interesting is that as the number of trained BadgeHolders increases, you can easily run several experiments simultaneously and evaluate results even faster.
Conclusion 3 - Filtering and Categorization Processes:
Almost all Badgeholders follow a two-stage process before starting their evaluations:
First, they purge "junk" projects, "scammers," or those irrelevant to Optimism’s goals, allowing them to focus on projects that truly add value to the network. Many see this as the most critical step—filtering out applications that are merely trying to obtain OP without delivering real impact. This initial filter requires reviewing all applicants.
Currently, there are Badgeholders specifically dedicated to this filtering process.
However, there have been errors in both directions:
Projects miscategorized as scams or low-quality.
Meme tokens awarded for generating gas fees for the network, rather than supporting impactful projects.
The second step involves categorizing projects/types of projects. This is something the Collective is already doing with recent versions of the Retro.
Projects are segmented not only by category—such as DeFi, Wallets, Governance, DevTooling, etc.—but also by subcategories and differentiated by metrics.
This is essential because narrowing the scope helps reduce the number of irrelevant projects.
This is how the 'Retroactive Grants 5' will be segmented by categories.
Important Note: As the crypto ecosystem grows, generic categories are becoming too broad. One thing Retro is doing well is further refining these categories. For example:
DeFi → Too broad.
DeFi → Swaps Protocols → Quite broad.
DeFi → Lending Protocols → Quite broad.
DeFi → Lending Protocols → Fixed Rate → Narrow.
Narrowing the scope by both category and sub-type allows for more focused evaluations. While it may exclude some projects, it helps in refining the quality of assessments.
Conclusion 4 - Alignment with the Foundation and the "Collective Feedback Commission":
While this Mission Request is still in development, Badgeholders made it clear that new tools and formats for evaluating projects that could be considered for RetroActive Grants are important. The challenge we see is that for this to be truly worthwhile, there must be greater alignment with the Foundation, through a Foundation Mission Request, and full immersion with the key players (Badgeholders). The context is so dynamic that these concepts can quickly become outdated unless they are 100% aligned with the aforementioned stakeholders. This should be a more collaborative process rather than an isolated effort.
Complications with Parallel Platforms: It’s important to be cautious of unofficial platforms that may complicate the process. It would be advisable for verification and review tools to be approved by OP Labs or the Foundation to avoid duplicating efforts or using unofficial platforms that could cause confusion.
We don't believe it's worth funding projects, even if they are public goods, that do not meet item 1.
Conclusion 5 - Focusing on Median-Based Weighting:
Different Badgeholders use various strategies to minimize outliers. Some rely on tools like ChatGPT for input, while others tend to adjust their points based on projects they like the most or least.
This can be visualized through a weighted curve approach, where outliers at both the top and bottom are adjusted:
A. Reduce the benefits given to projects at the top. B. Provide a bit more support to those at the bottom, or exclude them completely.
Conclusión 6 - Valuing Positive and Meaningful Participation in Governance and Technical Contributions:
While active participation in the community is not a determining factor, most Badgeholders consider it a plus. Its relevance depends on the specific topic. However, suspicious or low-value participation that simply generates noise or interaction without contributing value is heavily penalized.
Future Considerations Based on the Current State of Retro Funding:
Many of the suggestions mentioned here have already been implemented.
After reviewing the platform used for Retro 5, several mechanisms seem well-implemented and moving in the right direction, such as:
The option for Badgeholders to opt out.
Improved filtering mechanisms.
The reduction of categories.
A more streamlined token distribution process.
Introducing on-chain ranking and point systems to determine new Badgeholders, or using a “Guest” logic for temporary participants, seems like a positive step forward.
The ability to quickly evaluate projects with a ranking from “Very High” to “Low,” as is now possible and was suggested by LauNaMu in their article, makes a lot of sense.
It could be interesting to consider allowing inactive Badgeholders to delegate their voting power to others, potentially with a small multiplier. While this could lead to some centralization of power, it would be particularly useful for complex topics that not all Badgeholders may fully understand, such as highly technical research or specific elements of the OP Stack.
Given these developments, our focus in the Mission has shifted more towards the ideas outlined in Conclusion 3, specifically the development of a new framework that we believe is important to consider for the future. This includes showcasing data from past Retros and proposing a framework to implement this new methodology.
Although the platform is yet to be built, the core objectives for a first screening tool would be:
Filtering out applicants, with the option to hear other Badgeholders' reasons for such decisions.
Reorganizing applicants by categories.
Normalizing or applying Winsorization to address extreme cases and outliers.
However, since the official "voting" platforms already address these needs and have access to project information via API/forms without requiring manual data entry, our platform will focus on:
Displaying all relevant information from previous Retros, up to Retro 5.
Centralizing details on how to vote, what to vote for, and sharing relevant articles, research, and best practices.
Offering a page to propose our new mechanism and framework.
Including an interactive section for feedback where users can share their conclusions from previous Retros, helping consolidate information that can be used in future iterations.
Building a platform to streamline feedback and retrospectives in this manner would be valuable, though it may exceed the current scope of our Mission Request
New Iterative Improvement and Experimentation Framework
This process is designed to achieve excellence in a specific activity (let’s call it A) through an approach that combines continuous improvement and experimentation. It draws on elements from the Lean Startup method and the explore-exploit trade-off theory.
Objective:
To become the best at A through an iterative cycle that combines continuous learning (exploiting what is known) and the exploration of new possibilities (experimenting with the unknown).
Process Steps:
Create the Basic Initial Version (Version I - Minimum Viable Product)
Develop a basic version of A. This first version captures the essential elements needed to start working in this area.
This acts as the Minimum Viable Product (MVP)—not perfect, but sufficient to gather data and experiences.
Iterate and Improve on What is Known (Version II - Exploit Learning)
Analyze and improve Version I using the feedback and data gathered.
Make incremental improvements to optimize the process, fixing errors and enhancing what works.
This is the exploit phase, akin to the Lean Startup method, where rapid iterations refine the current version.
Experiment with Something New (Version III - Explore New Options)
Simultaneously, develop a new version of the activity, entirely different from Version II.
This version is based on novel ideas or gut feelings and may appear random, but the goal is to explore new ways of doing things that might yield better results.
Experimentation allows for innovation beyond the boundaries of the known.
Compare and Combine the Best (Version IV - Synthesis and Evolution)
After testing both Version II and Version III, compare the two to identify the most effective aspects.
Synthesize the best elements into a new version (Version IV), combining the lessons learned from the exploit phase and the best ideas from the explore phase.
Repeat the Process: Continuous Iteration and Parallel Experimentation
The cycle repeats, creating new versions based on:
Iterating on the previous version to continue improving (exploiting what has been learned).
Running new experimental versions to test hypotheses and explore new possibilities (exploring what is unknown).
By conducting multiple experiments in parallel, the process accelerates, enabling faster learning and closer alignment with the goal of becoming the best at A.
Importance of the Explore-Exploit Trade-off
This process balances the need to exploit what is known to work with the necessity of exploring new possibilities that could lead to significant breakthroughs.
Exploit (Versions II, IV, VI, etc.): Use existing knowledge to optimize and improve efficiency.
Explore (Versions III, V, VII, etc.): Introduce new ideas or approaches that could bring transformative value or new perspectives to activity A.
Conclusion:
This iterative process of continuous improvement and experimentation blends the logic of Lean Startup (rapid iteration and improvement) with the explore vs. exploit approach (balancing known improvements with novel ideas). By evolving through both incremental iterations and disruptive experiments, we move closer to excellence in activity A.
Final Notes
This concludes our research for this milestone. We hope it provides valuable insights, and we are open to feedback and suggestions.
If you would like to participate in an interview or share your perspective, please feel free to contact us. This is an iterative document, so we are happy to make updates as needed. You can reach us via social media or directly on my personal Telegram: @gonzacolo.
This is only the first milestone of the project: Making Impact Evaluation Accessible, and we look forward to achieving more milestones soon.
All updates will be published in the relevant Update Thread on the Optimism forum.
Thank you for reading.
