Diffusion of an Innovation: Lessons from the Multistate Longitudinal Data Exchange

Written by Heather McKay, Sara Haviland, and Suzanne Michael, Rutgers, The State University of New Jersey, Education and Employment Research Center and Sarah Leibrandt, Western Interstate Commission for Higher Education

 

Introduction

Why do some innovations in policy or practice take off, while others fizzle out? Social scientists and policy experts have grappled with this question for decades, studying the diffusion of new ideas and examining how and why states, organizations, and institutions adopt innovations to their policies or practices.[2] What motivates these groups to take on new ways of doing things? And why are some more willing and able than others to try something new? What factors facilitate some to become “first movers,” or early adopters in innovative practices, and what factors prevent others from doing so?[3] These questions are especially pressing in the introduction of new collaborative, multistate data-sharing systems, where the main benefits of joining a system are access to other partners, their data, or resources with respect to the “mobility of human capital within, into, and out of states.”[4]

The mechanics of novel collaborative systems are particularly compelling in an era of big data, as states recognize both the limits of isolated data systems and their potential when they are combined with one another. In recent years, states have begun adopting innovations in data use and application, including accessing new data sources, combining data sets, and using different analytic methods. These efforts are designed to help states provide more comprehensive answers to policy and practice questions and to better inform their work. The adoption of a new practice, however, does not happen all at once. Rather, it is a process some states and organizations will take on and adjust to faster than others, and some will choose not to engage in at all.[5]

This brief examines the diffusion of a data innovation from 2014 to 2019: the introduction of the Multistate Longitudinal Data Exchange (MLDE), a collaborative system both within and between state data agencies. It is based on research and evaluation activities conducted over the past five years by Rutgers’ Education & Employment Research Center (EERC) including data collected from over 40 interviews with state leaders and Western Interstate Commission for Higher Education (WICHE) representatives, observations of user group meetings, surveys, and MLDE document analysis. This brief presents the salient factors that contributed to the emergence of early adopters of the new data exchange. Specifically, what made these groups want to join? What concerns did they have? How might future exchange organizers build on the interests of early adopters as well as respond to their own issues of concern?

The Decision to Join a New System

As with all new innovations and collaborations, the decision for states to join the MLDE involved not only a complex assessment of incentives and costs but also a great deal of trust. States had to make decisions about joining without knowing whether the project would ultimately be successful. They did not know who would be partnering with them or what, if anything, their counterparts would contribute to the collaboration. Many, if not all, must have wondered, Would there be a problem with free riders? Joining the MLDE exchange also involved the identification and commitment of staff resources. Data agencies had to think about security considerations – issues such as client privacy and potential use of data – and carefully weigh the risks and benefits of exchanging data. State stakeholders interviewed by EERC identified several key questions they had from the beginning: Will enough states and state agencies participate to make the MLDE information useful? Will those future partners provide enough data, and will the quality of that data be high enough, to make joining the MLDE worthwhile? Will all exchange partners use the provided data in a responsible and secure manner? Will they commit state resources to future updates and maintenance to keep the system going?

As the exchange began work, there were general feelings that a system developed by and for the states provided more comfort and guarantees than a federally developed and operated system. The MLDE was rolled out in two phases, each of which required different considerations on the part of the actors.

Phase I: The Innovators

The MLDE began with a four-year pilot project (2010–2014) in four states: Washington, Oregon, Hawai‘i, and Idaho.[6] These states can be described as the “first-movers” or “innovators,” as they fostered the diffusion of the new idea – in this case, the MLDE.[7] By joining the MLDE at this early stage, they affirmed an interest in trying something new and a willingness to take a risk and engage in the work necessary to make the idea a reality. Joining the project at the start meant that these states had no guarantee of success or even of a broader launch, and no model for participation. Yet, they were all able to secure the buy-in of key stakeholders within their states’ education and workforce agencies to actively participate in the pilot phase of the project.

Through qualitative interviews with representatives from some of these states, EERC sought to understand what made these state innovators so amenable to working with the MLDE from its inception, a time when its benefits were neither fully known nor guaranteed.

Our key informants shared that their states joined the pilot because they saw value in the MLDE’s goals. They believed the MLDE would give them the data they needed to answer important questions posed by policymakers and agencies, help them evaluate state programs and practices, facilitate funding allocations, and improve consumer services. One representative described conceiving of the work of the MLDE pilot, and the practice of sharing data between states, as “valuable but difficult,” – that getting to the end point of sharing data would not be an easy task. This respondent recalled noting that there would be political, legal, and structural challenges to overcome. Nevertheless, they perceived great potential and significant value in participating. First-mover states also spoke about the inherent value of experiencing the learning process in a pilot like the MLDE.

Phase II: The Early Adopters

At the conclusion of the four-year pilot, WICHE received a grant from the Bill & Melinda Gates Foundation to further develop the MLDE and expand its membership up to ten states. This new phase launched with the buy-in momentum of the four pilot innovator states and promising results from the pilot. These factors provided some assurance to the next group of participants – the “early adopters” – both that the MLDE was being run by a team that had proven its capability in the area and that a multistate data exchange could work. At the same time, while the pilot had established the MLDE’s value as a regional data set, it was not clear if it would also be valuable as a national-level data exchange. Thus, like the innovators before them, these early adopter states had to assess the benefits and risks of participation. In the section below, we highlight the most prominent issues and key considerations that emerged as new data agencies and states were joining, waiting to join, or choosing not to join the MLDE; and some of the factors that helped mitigate challenges or hesitations.

Regional Considerations

For most states, the MLDE is only useful if other states – preferably those nearby – are also actively participating. For example, having both Washington and Idaho participate was important, since they experience a good deal of cross-state economic and residential mobility. Nonetheless, some states with no identified regional counterpart were willing to take the risk and be early adopters, hoping that neighboring states would follow their lead. Many of the initial state participants joined because they had an immediate need for the data or because they had the resources available to do so and hoped that by making a move, they would prompt proximal states to do the same.

The states of particular attraction for the first movers and early adopters, and the MLDE as a whole, were: 1) economically and regionally relevant states – those with geographical proximity, economic pathways, and mutually beneficial exchange interests, and/or 2) “big” states such as California and Texas. Many saw the involvement of one or more states that fit into at least one of these categories as the main condition for their own participation. This dynamic of states wanting other states to join first created a challenge for WICHE as it tried to expand the MLDE. What factors played a role in moderating this risk?

Establishing Regional Footholds.

One way to counteract this dynamic was to get one early adopter to sign on in each geographic region who might then share their MLDE experiences and its data resources with their counterparts in contiguous states. WICHE hoped that this would stimulate proximate states to become interested in learning from the first movers and early adopters, and maybe even inspire those states to emulate them by joining the exchange themselves, thereby expanding the network outward. There was even some thought that as states benefitted from the use of data culled from the exchange, participation in the MLDE could come to be seen as a competitive advantage by surrounding states, prompting those states to join so they too could benefit from the exchange.

While this strategy has been successful in expanding membership in some regions, the efficacy of regional first-movers or early-adopter states as recruiters varied. Unsurprisingly, there often is an imbalance of power between states within the same region because of differences in area, population size and mobility, location relative to others, and economies. Some states find themselves in a position of relative advantage that allows them to choose with whom to share their data and on what terms. On the opposite end, there are states that are interested in the data and are ready to make great use of it but are perceived as not having enough to offer in return. As such, they cannot spark the interest of or commitment from the data-privileged states. Although this imbalance was expected, the extent to which it affected the states was greater than expected. Individuals from data-rich states reported to EERC that they were less motivated to speak with states that were smaller in size or relatively isolated.

The tendency to rely on other states to make the first move and join, if left unaddressed, can turn into a loop of action paralysis in which states wait for each other indefinitely, each unwilling to risk being the only one from the area to join. WICHE attempted to confront this challenge by holding regional meetings during which the states already participating in the MLDE and other states could talk with one another about the exchange and its potential benefits to them. Creating regional coalitions was another way to address the challenge. It was thought that regional groups working together to resolve individual states’ concerns, and then joining the MLDE together, could reduce anxiety and mistrust. WICHE has also considered working through the challenge of membership through regional pilots, allowing states to learn more about the exchange, test its utility for their needs, and experience the potential for value added. While no regional pilots have been launched to date, there is ongoing discussion about moving in this direction.

State Precedent and Data Resources.

Another factor that seemed to influence a given state’s participation in and enthusiasm for the MLDE was the state’s previous experience with developing a State Longitudinal Data System (SLDS). States with little or no exposure to an SLDS seemed to be more reluctant to join the MLDE, in many cases due to a limited understanding of the legalities, technology, and logistics involved, as well as an inability to see the long-term utility of such a project. On the other hand, states with backgrounds in data aggregation, matching, and exchange seemed to be more willing to join. These states seemed to understand both the risks and the payoffs; they recognized that the future potential for scaling such systems to the nationwide level may outweigh the risks they faced as early adopters.

The availability of resources such as staffing, time, and money was both a reason for some states to join and a reason for some not to join. For states that were understaffed in data and IT roles, this was a major concern. To help alleviate this, WICHE provided member states with funds to help them build infrastructure or to support other aspects of implementation.

 An Investment in Relationships and the Greater Good.

Some saw the value of the MLDE to be the network it created among participating states. The networks and committees convened to develop and implement the MLDE encouraged states to develop strong relationships with each other by providing a venue to communicate about their data practices. Many were thankful for these connections and the information they gained from them. Such relationships have important “spillover” benefits for additional collaboration between states.

One reason some states were able to overcome this first-mover and early-adopter challenge was simply altruism. A number of state leaders reported that their reason for joining was for the greater good – the development of a nationwide longitudinal data exchange that provided better data to consumers, even if it may be years before it truly benefited their own state or agency. These actors were putting in the time and thought needed to lay the groundwork for a future with better data.

Within-State Challenges

In addition to first-mover and early-adopter challenges for the states, there were also challenges within states. In some cases, not all state agencies (K-12, higher education, and workforce) within a state signed on to the MLDE or chose to contribute data to the exchange. This often occurred because of time, perceived value, or because there were concerns about security or potential misperceptions by their constituents. The data structure in each state varied, which also had an impact on states’ ability to join: some state data structures were more centralized while others less so; some states had mature data systems while others were much less mature.

Responding with Flexibility.

WICHE originally planned to require all state agencies related to K-12, higher education, and workforce to join the exchange in order for a state to participate. However, as with its strategic use of regional footholds, WICHE shifted its within-state strategy and responded with flexibility, aiming to include all agencies while acknowledging that data exchange might not be immediately possible. One committed state leader expressed relief and gratitude that the MLDE leadership was flexible enough to allow for segmental, partial enrollment of the state into the project. In fact, this ultimately made the difference in this state’s decision to join.

The Use Case Challenge

The usefulness of the MLDE’s data was an important factor in the decision for some states to join the exchange. Some state officials quickly saw how to make use of the MLDE’s data to study the cross-state migration patterns of individuals transitioning from high school or college into the labor force. Representatives from those states felt that being able to share and compare data with other states through a central data-sharing network would fill an enormous gap in understanding the mobility of talent across the nation. Many of these respondents noted that the value of the MLDE would increase over time as work and migration patterns continue to change. Other states saw value in the exchange of data but lacked a pressing sense of urgency to join.

In some cases, respondents’ assumptions about how the MLDE’s cross-state data would benefit their state were not fully borne out – cross-state analyses were not actually happening in those respondents’ organizations. This finding aligns with the results of a recent study by Pew Charitable Trusts about how states use data to inform decisions. Pew found that innovative uses of administrative data in states are generally rare.[8]

Not all states, or in some cases agencies, felt that the data they could get through the MLDE would be useful or valuable to them or that what the MLDE offered was worth investing their changing and often limited resources in. Several respondents were concerned about what kind of data they would receive from other states, and its consistency and quality over time. They worried about how the breadth and quality of data from other states would compare to their own as well as how partners in other states might use their data. They also worried about the changing demands on state agencies. For many states, concerns ebbed and flowed depending on workloads, funding, the current political and policy climate, and other factors.

For some states and/or state agencies, whether a perception or a reality, migration of students and workers across state lines was not considered to be a prominent issue. Some state representatives felt that the information already available to them was good enough, or they had access to other sources to get needed data. Others noted that joining the exchange would require approval from state policymakers and expressed concerns about how to present the value of a nascent exchange to elected officials and the public at large. As a result of the above concerns, a number of states that indicated some interest, and even participated in various meetings, had not yet signed a memorandum of understanding.

Demonstrations of Value.

To counteract some of these perceptions and challenges, WICHE chose to use information as their “marketing” tool. They released materials on use cases, shared stories, and provided information about the movement and migration of talent gleaned from studies done during the pilot phase of the project. These materials not only helped state data agency leaders see how the MLDE could be deployed but also gave those leaders a resource to use when making the case for participation with policymakers and the public.

Recognizing Efficiencies.

Many saw the MLDE’s potential to reduce the burden of paperwork and data processing in cross-state work as a major appeal, which helped to make the use case. One interview respondent described, “instead of running [data sharing agreements] separately for [State A] and [State B], now I could do it only once. This is very efficient. And now I have to upload the data once, I don’t have to do it twice. So, there is real potential in this.” Some respondents said membership in the MLDE would save staff time and effort in both reporting and executing administrative tasks. States were excited about the possibility of having one data set that could provide the information they needed for state and federal reporting, as well as help them answer questions from their state legislatures.

Data Security

Data safety was the paramount issue for most of the respondents we interviewed. State users wanted to make sure that the system would protect their constituents’ information and that data would not “fall into the wrong hands.” When asked whether they believed the MLDE was secure, most respondents answered affirmatively but admitted having some concerns. Establishing trust is an important factor in alleviating safety concerns for those involved in the MLDE. Many respondents talked about the trust they had developed with MLDE states and WICHE over time through working together and spending time together at WICHE-led meetings. Some expressed concern that as the network expands, these relationships will not stay as strong, and the trust they had built up may begin to erode. One respondent said, “The biggest thing I can see is the security and level of trust among states, because [the MLDE network] is a black box, in a way, as opposed to working directly state-to-state.”

Security Precautions.

WICHE has taken every precaution to ensure that the MLDE is a safe and secure system with multiple layers of protection to ensure that personal information is safeguarded at every step of the exchange. Only states that have signed detailed data-sharing agreements that protect individual-level data may participate in the exchange of data; data use is restricted and subject to state approval; data access is limited to just a few individuals, all of whom have signed a nondisclosure statement prohibiting them from using the data for any purpose other than those defined in the MLDE governance rules; data are protected by multiple layers of encryption; the design of the MLDE exchange is undoubtedly complicated and the process in place to link data across states enhances security and privacy by limiting the amount of data stored in one place. For more information on the protections WICHE and the states have put in place regarding the MLDE, please see the one-page document titled, 10 Layers of Protection in the MLDE.

Developing Trust

WICHE worked hard to foster trust among network members throughout the MLDE expansion process. They employed multiple strategies to build this “culture of trust,”[9] the most important of which was the creation of opportunities for inclusive collaboration among states in the development of the MLDE. This helped state representatives become familiar with each other and learn how other states stored and used data. This relationship building among state partners helped to ease some of the worries states had about security and data use. For more information on trust, please see this brief in our series. [10]

Sustainability and Building a Model for the Long Term

Finally, a common concern about joining the MLDE was the potential for a national-level version of the MLDE to be developed and sustained over time. This concern became especially salient with respect to collaborative efforts, especially those involving local and state agencies.[11] As noted above, joining a national data exchange, particularly in its earliest stages, involves a significant commitment of time and resources. As states considered investing the effort and expense involved in joining the exchange, they wondered: Would they continue to see benefits in the long run? Would a national MLDE be able to support its work without the ongoing infusion of philanthropic dollars?

From the beginning, WICHE built the MLDE with an eye to the future. The initial costs of building the MLDE were necessarily high, as WICHE built the system and involved the necessary contractors to ensure a secure and stable technical process and to develop the collaborative culture and infrastructure required to make the exchange attractive to state data agencies. At the same time, WICHE also worked to establish a model that would ultimately charge a lower fee to join, and projected that once the MLDE was established, the annual maintenance costs for members would be in the five-figure area.

Even with this strategic plan, some state representatives expressed concerns about their ability to secure ongoing funds from state legislatures to maintain their membership in the exchange. One respondent noted having observed an irony in the state budgeting process that often made it easier to acquire funds to build new systems than to maintain systems once they are put in place. Such sustainability concerns echo the literature; scholars have found that “limited availability of financial resources is one of the most apparent reasons for failure in information sharing initiatives.”[12]

To address some of these challenges, WICHE developed tools and materials intended to facilitate and ease state-level implementation. These included ideas for use cases and various business plans designed to help secure operating dollars and foster the further expansion of the MLDE.

Conclusion

Understanding why some stakeholders decide to test or adopt a new policy or practice is complex. There are often multiple reasons informing such a decision, including getting a benefit, being the first, and sometimes even altruism. States and agencies that decide to become early adopters are taking a risk in many ways. It can be helpful for organizations working on developing and putting into place new policies and practices to consider these risks and benefits and to try to mitigate them through both information sharing and the development of structures, policies, and procedures that support the innovation. Helping to create relationships between partners and developing a community of trust can also be useful.

 

***** 

For More Information

For further information about the MLDE, please contact Patrick Lane, WICHE vice president of policy analysis and research, at plane@wiche.edu, or visit the project website: https://www.wiche.edu/key-initiatives/multistate-longitudinal-data-exchange/. The other briefs in this series include: Building Trust for Inter-Organizational Data Sharing: The Case of the MLDE, Diffusion of an Innovation: Lessons from the Multistate Longitudinal Data Exchange, Designing the Architecture of a Multistate Data Sharing Model, and Documenting the Value of Non-Degree Credentials: The Potential Role of the Multistate Longitudinal Data Exchange.

For further information about evaluation research in data sharing or workforce development, please contact Heather McKay, director of Rutgers’ Education & Employment Research Center, at hmckay@rutgers.edu, or visit the Center’s website: http://smlr.rutgers.edu/eerc.

Endnotes

[1] The other briefs in this series include: Building a Multistate Governance System, Building Trust for Inter-Organizational Data Sharing: The Case of the MLDE, Designing the Architecture of a Multistate Data Sharing Model, and Documenting the Value of Non-Degree Credentials: The Potential Role of the Multi-State Longitudinal Data Exchange. Available from the WICHE website at https://www.wiche.edu/key-initiatives/multistate-longitudinal-data-exchange/.

[2] Everett Rogers, Diffusion of Innovation (New York: Free Press, 1962/2003).

[3] Todd Makse and Craig Volden, “The Role of Policy Attributes in the Diffusion of Innovations,” The Journal of Politics 73, no. 1 (2011), 108-124. Frederick J Boehmke, Abigail Matthews Rury, Bruce A Desmarais, and Jeffrey J Harden, “The Seeds of Policy Change: Leveraging Diffusion to Disseminate Policy Innovations,” Journal of Health Politics, Policy and Law 42, no.2 (2016), 285–307.

[4] Brian T. Prescott, Beyond Borders: Understanding the Development and Mobility of Human Capital in an Age of Data-Driven Accountability. A Report on WICHE’s Multistate Longitudinal Data Exchange Pilot Project (Boulder, Colorado: Western Interstate Commission for Higher Education, 2014), accessed on 10 June 2020 at https://www.wiche.edu/longitudinalDataExchange.

[5] Sallyann Bergh, Alyssa Davis, Amber Ivey, Dan Kitson, and Jennifer Thornton, How States Use Data to Inform Decisions (Philadelphia, PA: PEW Charitable Trusts, 2018), accessed on 10 June 2020 at https://www.pewtrusts.org/en/research-and-analysis/reports/2018/02/how-states-use-data-to-inform-decisions.

[6] Prescott, Beyond Borders.

[7] Rogers, Diffusion.

[8] Bergh, Davis, Ivey, Kitson, and Thornton, How States Use Data.

[9] J. Ramon Gil-Garcia and Djoko Sigit Sayogo, “Government Inter-Organizational Information Sharing Initiatives: Understanding the Main Determinants for Success,” Government Information Quarterly 33 (2016), 572–582. Wioleta Kurshaska, “Relationships Between Trust and Collaborative Culture in the Context of Tacit Knowledge Sharing,” Journal of Entrepreneurship, Management and Innovation 13, no.4 (2017), 61–78.

[10] Heather McKay, Sara Haviland, and Suzanne Michael, Building Trust for Inter-Organizational Data Sharing: The Case of the MLDE (Boulder, CO: Western Interstate Commission for Higher Education, 2020), 1-3.

[11] Gil-Garcia and Sayogo, “Government Inter-Organizational.”

[12] Gil-Garcia and Sayogo, “Government Inter-Organizational,” 574.

References:

Bergh, Sallyann, Alyssa Davis, Amber Ivey, Dan Kitson, and Jennifer Thornton. How States Use Data to Inform Decisions. Philadelphia, PA: PEW Charitable Trusts, 2018. Accessed on 10 June 2020 at https://www.pewtrusts.org/en/research-and-analysis/reports/2018/02/how-states-use-data-to-inform-decisions.

Boehmke, Frederick J, Abigail Matthews Rury, Bruce A Desmarais, and Jeffrey J Harden. “The Seeds of Policy Change: Leveraging Diffusion to Disseminate Policy Innovations.” Journal of Health Politics, Policy and Law 42, no.2 (2016), 285–307.

Gil-Garcia, J. Ramon and Djoko Sigit Sayogo. “Government Inter-Organizational Information Sharing Initiatives: Understanding the Main Determinants for Success.” Government Information Quarterly 33 (2016), 572–582.

Kurshaska, Wioleta. “Relationships Between Trust and Collaborative Culture in the Context of Tacit Knowledge Sharing.” Journal of Entrepreneurship, Management and Innovation 13, no.4 (2017), 61–78.

Maske, Todd and Craig Volden. “The Role of Policy Attributes in the Diffusion of Innovations.” The Journal of Politics 73, no. 1 (2011), 108-124.

McKay, Heather, Sara Haviland, and Suzanne Michael, Building Trust for Inter-Organizational Data Sharing: The Case of the MLDE (Boulder, CO: Western Interstate Commission for Higher Education, 2020), 1-3.

Prescott, Brian T. Beyond Borders: Understanding the Development and Mobility of Human Capital in an Age of Data-Driven Accountability. A Report on WICHE’s Multistate Longitudinal Data Exchange Pilot Project. Boulder, Colorado: Western Interstate Commission for Higher Education, 2014. Accessed on 10 June 2020 at https://www.wiche.edu/longitudinalDataExchange.

Rogers, Everett. Diffusion of innovation. New York: Free Press, 1962/2003.