Etikk i praksis. Nordic Journal of Applied Ethics (2019), 13(1), 55-80
http://dx.doi.org/10.5324/eip.v13i1.2525


Responsibility for Assistive Technologies: Product Assessment Frameworks and Responsible Research and Innovation


Erik Thorstensen

Work Research Institute, Oslo and Akershus University College of Applied Sciences, erikth@oslomet.no

The approach to innovations known as Responsible research and innovation (RRI) aims to move the innovation system towards creating products that strive to realize social values along with economic benefits. This paper discusses the systematic assessment of assistive technologies in order for them to meet the aims expressed in RRI. A central issue in the discussion is how to facilitate an integration of insights from the discourse on RRI with more established assessment approaches such as Health Technology Assessment (HTA). Based on the literature on existing socio-ethical assessment tools, I investigate how these tools can be combined with HTA and how they can add perspectives from RRI that might increase the socio-ethical value of assistive technologies. Through a discussion on how to understand RRI, HTA, assessment and integration, I suggest a list of four possible approaches that have the potential to be applied as assessment approaches that integrate insights from RRI and HTA. These are then evaluated on their ability to address issues that have emerged from a literature review on RRI and assistive technologies, on empirical studies in this technology field and on their product focus. In conclusion, I argue that the Ethical Impact Assessment, the Socratic approach, the Ethical Matrix, and the HTA Core Model seem to be the most promising methodologies, but that these need adjustments to cover substantive themes from RRI.

Keywords: energy scenarios, justice, future generations, capabilities approach
 

Introduction

Worldwide we see a proliferation of assistive technologies in hospitals, institutions and homes, ranging from high-tech robots to low-tech analogue alarms. There are high hopes for assistive technologies in policy circles as part of an approach that might empower people with disabilities of all ages to reside at home longer, as opposed to living in different kinds of institutions. In the last 25 years, researchers, policy-makers and health professionals have especially focused on older adults in order to accommodate smart homes for this group (Thygesen 2009). The vision behind this practice has been to increase older adults’ independence and self-esteem, while reducing both health care expenses and an increased number of employees in health care delivery. In addition, several technology firms are launching solutions promoted as meeting these policy objectives. In the field of assistive technologies, we then encounter both policy pull and technology push, as von Schomberg (2011) calls them. Seen from a socio-ethical perspective, concerns over assistive technologies relate to issues such as how care is perceived, experienced and given (Roberts & Mort 2009), the possible medicalization of homes (Hofmann 2013), focus on caregivers at the expense of users (Topo 2008), and possible conflicts between empowerment of older adults and the larger goal of addressing ageing as a demographic challenge (Pols & Willems 2011).


The Assisted Living project is a research project using insights from Responsible Research and Innovation (RRI) in order to research and develop assistive technologies. The project consists of a facility for independent living, an SME for developing assistive technologies and smart home solutions, technology assessors, and researchers in technology assessment, ethics, ICT, nursing, and occupational therapy. We compare the assistive technology developed in the RRI-based project to an existing assistive technology developed in a project that is not based on RRI. The intention is to obtain information on the possible influence of process on product.


In the health technology field, Health Technology Assessment (HTA) is a common denominator for different policy advice formats on novel products or solutions. Consequently, this approach has high legitimacy and high policy relevance. In order to make RRI relevant and legitimate for decisions regarding health policy, I will explore how different assessment frameworks used for socio-ethical assessments of health technologies reflect the core components of RRI. A number of such assessment dimensions have been identified in the current context in a previous paper (Thorstensen 2017b). I will then focus on evaluating assessment approaches, but this study does not claim to exhaust all the possible socio-ethical approaches that might be combined with HTA and also resonate well with RRI.


This paper argues that Ethical Impact Assessment (ETIA) (Wright 2011),1 the Socratic approach (Hofmann 2005b; Hofmann, Droste, Oortwijn, Cleemput, & Sacchini 2014), the HTA Core Model (Lampe et al. 2009), and the Ethical Matrix (Kaiser & Forsberg 2001) are relevant candidates for assessing assistive technologies in line with HTA and the central thinking in RRI. In order to reach this conclusion, this paper begins with a general introduction into RRI and product assessment, followed by an explication of what a product assessment methodology of assistive technologies in line with RRI should contain. Based on literature that assesses health technologies and RRI and assistive technologies, I present the possible candidate methodologies and relate these to the aforementioned criteria.


 Responsible Research and Innovation

Responsible Research and Innovation (RRI) is a term used for research policy, for a cross-sectorial approach to science and innovation governance, and for an intended practical approach to research and innovation (Ribeiro, Smith, & Millar 2016). For the Assisted Living project, the last term of the phrase RRI applies. When conceived of as a method for innovation, Assisted Living project utilizes a conception of RRI as containing:

 

  1. A specific focus on addressing significant societal needs and challenges
  2. A research and development process that actively engages and responds to a range of stakeholders
  3. A concerted effort to anticipate potential problems, identify alternatives, and reflect on underlying values, and
  4. A willingness from relevant actors to act and adapt according to 1–3 (Wickson & Forsberg 2015: 1164)

 

The Assisted Living project understands itself as responding to the “Ageing society” as a “societal need or challenge.” The justification for using RRI in assistive technologies lies in the connection between RRI and these societal challenges (von Schomberg 2013).2 In addition, all four points above constitute the normative foundation for establishing how to assess a product in line with RRI.


Part of the novelty in the Assisted Living project, is the use of RRI as a guiding principle for doing research and innovation in practice. RRI has a novel normative approach to research and innovation in its study of the moral purpose of innovations. This approach aims to combine central value aspects in societies with the social and economic benefits of research and innovation through close links with industry (Pacifico Silva, Lehoux, Miller, & Denis 2018). Different approaches and RRI both include affected parties, users, stakeholders and policy-makers. However, they differ somewhat in that von Schomberg (2011) has tried to explicate how to conceptualize these values whereas Owen, Macnaghten, and Stilgoe (2012) tend to emphasize that these values should be sought among the participants, and operationalized towards application in research (Owen 2014). Wickson and Forsberg (2015) strive to combine these two approaches.3 In the European Union, RRI has been presented in relation to the research policy areas (the so-called “keys”) of public engagement, open access, gender, ethics, and science education (and earlier also governance).


Product assessment

A more specific meaning of the term “product assessment” is conditioned by background and discipline. In an early contribution to RRI, Stilgoe et al. mention some questions that are relevant for a product assessment:

 

·     How will the risks and benefits be distributed?

·     What other impacts can we anticipate?

·     How might these change in the future?

·     What don’t we know about?

·     What might we never know about? (Stilgoe, Owen, & Macnaghten 2013: 1570)


von Schomberg (2011) argues from a different perspective that in RRI, products should be evaluated through their ethical acceptability, sustainability and social desirability. When it comes to assistive technologies for the elderly, products must have acceptable functioning and design in line with the users’ and central stakeholders’ values and preferences, how they relate to quality of life, and general ethical principles. Central to the overall idea of RRI is to reflect on the purpose of research and innovation (Owen et al. 2012; Stahl & Coeckelbergh 2016).
  
The concept of a “product” needs further clarification.4 My understanding follows Brey (2012), who suggests three relevant levels for an ethical analysis of technology: 1) the technology level, 2) the artifact level and 3) the application level. With solutions at the application level, Brey refers to “the concrete use of a technological artifact or procedure for a particular purpose or in a particular context, or a specific configuration of an artifact to enable it to be used in a certain way” (Brey 2012: 311). In the present case, “application” would thus refer to an artifact being used by elderly individuals with diminished cognitive capacity in their homes in order to perform certain tasks or sets of tasks. Brey’s distinction between technology, artifact, and application makes it possible to distinguish different aspects of product assessment and to differentiate it from Technology Assessment – which has as its focus Brey’s “technology” level. I follow Brey in this article, and by “product” I refer to Brey’s application level that includes a specific purpose and the configuration surrounding the application in order for it to achieve this purpose.
  
Recently, researchers from the Responsible Industry project presented an approach where they argue that effective RRI should be assessed according to “certain identifiable consequences” (Stahl et al. 2017). These consequences are derived from the RRI “keys” of the European Commission (2012) as well as from an expert group on suitable policy indicators for RRI (Strand et al. 2015). Stahl et al. have employed the keys together with the process criteria from RRI, as listed above, to create a project self-assessment tool to monitor the RRI quality of a project or an organisation (ORBIT 2017). The ORBIT tool and similar RRI process evaluation tools fall outside the scope of the meaning of product assessment in this article.
 

Health Technology Assessment
The historical background for HTA is similar to that of the different forms of Technology Assessment (TA). They both originated in and through the same impulses that created the U.S. Congressional Office of Technology Assessment (OTA) in 1972 and with the same focus on efficiency, understood as the cost-effectiveness of a given health intervention. Traditionally, pharmaceuticals, vaccines and medical equipment have been easier to assess through HTA, while different types of health care practices have been more challenging to assess due to their interlinked and complex nature (Banta 2003). Adding to the complexity, studies reveal a lack of clear definitions in the literature on how value is or should be defined (Antoñanzas, Terkola, & Postma 2016).


Currently, the INAHTA (International Network of Agencies for Health Technology Assessment) defines HTA as follows: “Technology assessment in health care is a multidisciplinary field of policy analysis. It studies the medical, social, ethical, and economic implications of development, diffusion, and use of health technology” (INAHTA 2016). This expands the scope of the original use to include social and ethical aspects in a systematic manner. HTA still has as its main purpose to assist in decisions related to policy. Now, the policy context can be at any level, including at the low level of a hospital or the higher level of a municipality assessing new interventions.


With respect to RRI and HTA, Cuijpers and van Lente (2015) present a strong case for how HTAs do not allow for different sets of logic or values. In a similar vein, Moors and Peine (2016) argue that HTA assumes a perspective that downplays solutions enhancing individual agency and promotes medicalization. The integration of the social and ethical dimensions has suffered from being analyzed independently of the other epistemological dimensions in the HTA. As a remedy to this dissociation, suggestions have been put forth that mainly serve to add complexity to the procedure while subsuming social and ethical issues under general effectiveness research. (Refolo et al. 2016). Another dimension that has rendered this integration problematic has been that the specific methods have been immature (Hofmann 2005a). The EU-funded project Integrate HTA suggests including assessments of socio-cultural dimensions into HTA, in addition to or in parallel with an ethical assessment (Lysdahl et al. 2016; Mozygemba et al. 2016). Including other normative issues seems to be in line with the move from ELSA (Ethical, legal and social aspects) to RRI. Since HTA is by definition an interdisciplinary endeavour and since RRI revolves around continuously discussing and reflecting on assumptions, inherent values and responses, the preferable mode of integration in the product assessment approach and the HTA would seem to be those of coordination and of interaction, as defined by Hofmann et al.:

 

·     Coordination: Ethics is still an independent part of the HTA, but the role and weight might differ depending on contextual factors. The results from the other parts of the HTA will influence the input to the ethical parts, but not the methodological choices of the ethicists.

·     Interaction: Ethics and other disciplines will be in a continuous exchange of viewpoints and results with possible redefinitions of the policy question, methods, and relevant comparative cases for all disciplines. (Hofmann et al. 2015: 131-132)

 

This means that there should be some form of interaction between an assessment method based on socio-ethical concerns and monetary and health concerns in an HTA.
Indicators for choosing an approach

In an earlier study based on a review of the RRI and assistive technologies literature as well as dialogues with stakeholders in the current project, I documented that a range of values ought to be taken into account when developing assistive technologies (Thorstensen 2017b).5 I will now recapitulate the central findings from this article. A central concern is that a product assessment approach used in an RRI project should be able to incorporate these points.


The good life:

A central issue for the assessment of assistive technologies is that the stakeholders are included in thinking about and reflecting upon what constitutes a benefit in the particular case. These benefits should not be reduced to utilitarian concerns but should also include social relations, values such as exercise and amusement, and reflect modifications to the lived environment through devices and a novel type of care. Including the stakeholders’ own understanding of benefit can be carried out as a part of the assessment (when interviews or other qualitative approaches are used) or in phases prior to the actual assessment.


Risks and benefits before use:

The above concerns should be used in a solid and complete pre-trial (i.e. ex ante) testing of a product. Such testing should also encompass privacy, safety, security, information collection and sharing as well as a thorough review of the evidence base of the product. Assessments of the product’s economic, social and ecological sustainability ought to constitute a part of this evidence base. However, this criterion will be disregarded for the present purpose, since we are striving for an assessment method that is applicable both to products made in an RRI process and other products (without any possibility to conduct an ex ante assessment), where an assessor or an assessing body lacks information about the process.


Distribution of risks and benefits:
An assessment should include considerations about the distribution of risks and benefits. In particular, one should be aware that assistive technologies might entail a transfer of the risks towards the elderly. If such a transfer seems to be the case, alternatives to the solution should be explored or the transfer needs to be justified in terms of other types of benefits.

Distribution of responsibilities:

As with the distribution of risks and benefits, the distribution of responsibilities regarding the product should be well understood, but also open for discussion. This relates particularly to the division of responsibility among the different operators in the service provisions connected to a product.


Training:

A final concern is that the assessment also investigate how users are introduced to the product, since health professionals and users alike identify didactical approaches (or their absence) as key aspects in the process.

  
These criteria overlap to some extent with the previously referenced product assessment question posed by Stilgoe et al. (2013), who mention the distribution of risks and benefits. The dissimilarities, however, have to do with future impacts and changes in such impacts. As Cuijpers, van Lente, Boenink, and Moors (2014) point out, the future orientation inherent to RRI demands a responsibility towards an imagined or assumed future. In order to fulfill this demand, I believe that an assessment approach would also need to include an exploration into contingency, in order to assess plausible changes (Stilgoe et al. 2013). Recently, Pacifico Silva et al. (2018) investigated Responsible innovation in Health (RiH) in a policy setting, and they argue that this should encompass five value domains consisting of nine dimensions.6 Their addition to the themes above are the health relevance, the organizational dimension and frugality. Health relevance relates to the innovation’s intent to address neglected or under-prioritized disease or adding to incremental benefits for the more affluent. The organizational dimension studies how or if the producer – inspired by the literature on social entrepreneurship – seeks to provide additional values to society. The frugality dimension is rooted in the literature on health economics showing the pressures on the health system by novel technologies and focuses on whether greater value is produced through fewer resources. Regarding the RRI qualities of the frameworks, I use frugality, social entrepreneurship and health relevance as parameters based on the consideration of the frameworks suggested by Pacifico Silva et al. (2018)


The following criteria are central in analyzing how to address the quality dimension of ethical assessment approaches: comprehensiveness and a broad inclusion of values, user-friendliness and transparency (Beekman et al. 2006; Forsberg, Shelley-Egan, Thorstensen, Landeweerd, & Hofmann 2017; Kaiser, Millar, Thorstensen, & Tomkins 2007). In practical terms, this translates to an assessment that includes a broad range of relevant values and other ethical aspects. The ability to practically operationalize these criteria is key, so that other persons (users, stakeholders, decision-makers or the public) can understand how conclusions follow from the relevant values and other ethical aspects.7
  
The values proposed by the desired assessment approaches should at least encompass a good life, risks and benefits before use, risks and benefits in use, distribution of risks and benefits, distribution of responsibilities, and training within a fixed and transparent framework that allows for case-relevant and socio-ethically relevant aspects and arguments. Stilgoe et al. (2013) supply the criteria of other possible impacts, future changes of impacts and systematic inquiry into ignorance, as well the purpose of the innovation. From Stahl et al.’s (2017) substantive list of product assessment criteria I retain gender/equality and diversity, open access, social justice/inclusion, sustainability and science education – and here ethics should be added for systematic purposes. Based on Pacifico Silva et al. (2018), I use health relevance, frugality and social entrepreneurship as criteria. In terms of the process criteria of governance and public engagement from the European Commission’s six “keys,” I believe that governance is covered by distribution of risks and benefits and distribution of responsibilities, while public engagement is covered by comprehensiveness and a broad inclusion of values, since these presume the inclusion of a variety of perspectives and viewpoints.


Discussion of assessment approaches

Several scholars have recently studied in detail some of the possible candidate frameworks (Forsberg et al. 2017) and their relation to HTA (Hofmann et al. 2015; Lysdahl et al. 2016). I will use their work as a point of departure in addition to my work in the literature review for the Assisted Living project to present the possible frameworks that may be suitable to apply as an RRI product assessment framework. I have selected the frameworks and methodologies discussed here based on whether they appeared in the reviewed literature or if they have been discussed in relation to HTA and socio-ethical issues.8 Some of the approaches reviewed by Assasi et al. (2014) propose a variety of a literature reviews, but these will not be considered as they constitute what Hofmann calls an “add-on approach.”9


According to Hofmann et al. (2015) and Lysdahl (2016), the possible interactive and coordinated approaches between HTA and socio-ethical assessments are Social Shaping of Technology (SST), Axiological (Socratic, EUnetHTA), Constructive Technology Assessment, Interactive Technology Assessment (iTA), and the Ethical Matrix. These authors also mention utilitarianism, discourse ethics and wide reflective equilibrium as interactive ethical approaches. However, these approaches would not qualify as frameworks in the sense that they provide practical guidelines for addressing ethical quandaries. Since utilitarianism is a theory of normative ethics, discourse ethics is a way of justifying and legitimizing moral judgements, while wide reflective equilibrium is an approach that might well be used in order to analyze or to develop coherence between principles and judgements (Forsberg 2007c). Of the methods or frameworks discussed in the mentioned literature review (Thorstensen 2017b), only the Ethical Impact Assessment has the potential to focus on an application (Forsberg et al. 2017), whereas the remaining ones are mainly used in planning and/or interventions.10


Constructive Technology Assessment (CTA) is not a form of product assessment, but an approach to identify societal impacts of technology developments early, in order to make these impacts as beneficial as possible (Schot & Rip 1997). Interactive Technology Assessment (iTA) shares the same purpose as CTA and does not include interaction between stakeholders, but is limited to interviews and analysis (Reuzel 2004). Social shaping of technology (SST) is explained by Lysdahl et al. to be a position that sees

 

technology as the product of societal processes within industry, research institutes, governmental bodies, and society at large, rather than an independent artefact that has a certain, measurable impact on its target. Therefore it is important to understand the engagement and strategies of various actors, and the way various problems are defined and resolved (2016: 64).

 

SST’s strongest asset is its ability to create novel insights as to the interaction between society and technology, which is invaluable. The challenge of this approach is that it lacks structured resources to guide an assessment towards specific topics, which limits its comprehensiveness and transparency (Forsberg et al. 2017). Consequently, the approaches I will address in more detail in this paper are the HTA Core model, the Socratic Approach, Ethical Impact Assessment and the Ethical Matrix.

 The HTA Core model

The HTA Core model is in itself a synthesis of a range of HTA practices developed by the European network for Health Technology Assessment (EUnetHTA), and it is currently online in its third version.11 Ethical analysis is a separate chapter of the HTA Core model. The nineteen isolated ethical questions cover six topics, seemingly derived from Beauchamp and Childress’ principlist approach and expanded to include legislation and ethical consequences of conducting an HTA.12 Since the HTA Core model is structured by questions in the form of a checklist, it provides many opportunities for different types of comparison between health technologies. However, limiting the selection of questions to only these nineteen socio-ethical elements seem unnecessary, since the full HTA version of the HTA Core model contains several highly important normative questions that are likely to affect the uptake of health technologies, such as, “G0001 How does the technology affect the current work processes?” This theme is highlighted as important by Wouters, Weijers and Finch (2017) and is reinforced by actual practice where the EUnetHTA (2017b) has investigated potential organizational changes. Even though the list of ethical questions might be found lacking, the total HTA Core model rectifies these shortcomings to some extent. The total HTA Core model further examines resource uses and social aspects such as health relevance, but does not directly include a value-enhancing business model despite addressing market conditions and aspects of ownership.


An investigation of the practices in EUnetHTA’s published reports shows that patient involvement does not seem to be an absolute criterion, since they in one instance write “no response from patients has been received” (EUnetHTA 2017c: 49), but in other instances conducted focus groups with patients (EUnetHTA 2016b). This point is also raised by Lysdahl et al. (2016). Evidence of the quality assurance of the possible ethical impacts is also variable (EUnetHTA 2017d). In one instance the ethical aspects are addressed by the first author but not reviewed, but they are checked in another instance (EUnetHTA 2016b). Good practice of HTA further depends on a range of extra-disciplinary factors that relate to the practice of assessment rather than science. These variations in application practices imply that there is room for adjusting and improving actual practices. The HTA Core model is highly transparent due to its specificity.


Since the HTA Core model accounts for ethical, cultural, social, legal, and normative issues in the form of a list, it has a large degree of transparency Furthermore, the compatibility with HTA is a given. The HTA Core model is reflective in its normative valuations because questions allow for discussion of central normative assumptions in the framework. However, the practical application of the full HTA Core model does not appear to include the more reflective questions, and how socio-ethical issues are integrated varies greatly. However, the openness of the framework to including such elements is a strength in this context.


The Socratic approach

Hofmann et al. (2015) group the Socratic approach and the HTA Core model together in terms of how they can be integrated with HTA. Hofmann (2005b) presented the Socratic approach as a methodology to systematically include ethical issues in HTA without giving preference to any ethical theory. In 2014, Hofmann et al. revised the 2005 approach, because the 2005 version was not exhaustive, lacked issues related to screening, lacked a method for balancing harms and benefits, had shortcomings in the treatment of distributive justice, and included some questions that were unclear and needed modification.


In what follows, I primarily address the 2005 version and view the 2014 amendments and additional modifications as indications of the flexibility of the approach (see for example Hofmann (2017b)). Procedurally, the Socratic approach identifies the purpose, the stakeholders and the relevant moral questions; investigates, analyzes and discusses the identified moral questions; and finally summarizes the central findings (Hofmann et al. 2014).


The original 33 questions are listed and approached from different perspectives: the technology (i.e. application) itself, the user group, human dignity or personhood, social aspects, consequential aspects, other stakeholders, the implementation, the assessment process and governance issues (Hofmann 2005b, 2017b). This form of perspectivation is one of the appealing features of the approach since it addresses the contextual reality of the technology in use. The Socratic approach seems to be unique in asking the question, “What are the moral consequences of the HTA?” which allows for reflection on the purposes of the assessment. Question 17 addresses health relevance, while frugality and enhanced social benefits seem to be lacking.


Hofmann (2017a) provides an application of the modified Socratic approach. Here, the modified Socratic approach shows itself to be reflective in its normative analysis, with questions related to the normative assumptions in the technology assessment methodology. Hofmann’s mastery of his own method impresses a reader. However, the recent application of the Socratic approach to smart houses as assistive technology (Sánchez et al. 2017) lacks such reflective discussions on the promissory nature of assistive technologies. Such a difference in use opens up questions regarding the skills necessary to apply the method. Hofmann (2013, 2017a, 2017b) makes extensive use of literature reviews and provides rich presentations, but more participatory approaches could be useful to receive input in areas where the literature remains silent or where there are controversies.


Ethical Impact Assessment

In the FP7 Prescient project, Venier et al. (2013) present an Ethical Impact Assessment (ETIA). The approach is structured around Beauchamp and Childress’ (2013) principlism approach with an additional section on privacy and data protection. Wright (2011) identifies a range of possible Beauchamp and Childress principles relevant to ICT assessments.13 In the first paper on the ETIA, Wright (2011) specifies questions on several themes related directly to health and technology.14 As with the modified Socratic approach, the ETIA is a stepwise approach starting out with a threshold analysis (European Committee for Standardization 2017), but the ETIA steps move in time whereas the modified Socratic approach moves along themes.15 The central procedural steps in the ETIA are the involvement of stakeholders in identifying risks and solutions, the production of the report and the independent audit of the report (Wright 2015).


ETIA draws on the development of ethical tools as well as the Constructive Technology Assessment (CTA), and the specifications of the questions are derived from studies of CTA and other assessment methodologies (Wright 2011). The ETIA does have a clear function as an ex ante assessment approach, like the CTA, but additionally has lists of questions that can be used to address the application and artifact levels of technologies. Brey (2012) supports ETIA’s substantial list of questions but finds that Wright lacks a clear method for forecasting. Since the scope of this article is to address previously developed products, Brey’s criticism is valuable but does not in any way undermine the use of the ETIA as a possible product assessment framework.


ETIA provides the transparent procedure that Forsberg et al. (2017) found lacking in principlism. However, Reijers and colleagues characterize ETIA as an “intra assessment method,” which is used “at the stage of an R&I in which conceptual ideas are being translated into a concrete technology design, and in which prototypes are made and tested” (2017: 14-15). In this way, the ETIA cannot be seen as solely an ex post product assessment. Furthermore, Forsberg et al. have argued that the ETIA has a versatile quality and a process that is “clearly described in a number of steps which can be easily followed – and crucially, adapted – by the user according to application-specific factors, contextual conditions and assessment purpose” (2017: 50). If an assessor wishes to adapt the ETIA to an ex post assessment, the whole process stands out as transparent for decision-makers, stakeholders and the public.


Given Wright’s (2011, 2015) rich descriptions of the ETIA as a process tool and Reijers et al.’s (2017) presentation of the ETIA as an intra assessment method, is it viable and justifiable to apply the ETIA as an ex post product assessment? Based on practice, it seems that the reply is yes. Dittrich et al. (2013) apply a different form of Ethical Impact Assessment (Kenneally, Bailey, & Maughan 2010) to a hypothetical botnet. Wright and Friedewald (2013) highlight the similarities between Kenneally et al.’s form of Ethical Impact Assessment and their own. Eventual uses of the ETIA as an ex post assessment need to address how the ETIA might change when used as an ex post assessment rather than as an intra or ex ante assessment.


The ETIA is a comprehensive approach, but is it too comprehensive? The Socratic approach with its original 33 questions seems more manageable than the countless specifications of the ETIA. Rodrigues et al. (2016) present the ETIA as it was applied in the FP7 PULSE project. Here, Rodrigues et al. show that the ETIA is a valuable tool that is capable of analytically identifying threats, vulnerabilities and risks to ethical, social and legal principles so that mitigation measures might be put in place.


The literature lacks a study that looks at the possible integration between ETIA and HTA, even though Flaming (2017) states that ETIA is a valuable method for an ethics review in a discussion where HTA figures.


However, since the ETIA operates on the basis of a checklist, in the same manner as the Socratic approach and the Core HTA Model, there should not be any formal problems with such an integration. The working method design in the ETIA is characterized by quality checks in the same systematic way as the HTA. In addition, the emphasis on direct stakeholder inclusion seems to move beyond the desktop approach signaled in the Socratic approach.


The Ethical Matrix

The Ethical Matrix is a structured approach for analyzing impacts of technologies according to stakeholder groups and the ethical principles of fairness, autonomy and well-being (Kaiser & Forsberg 2001). The Ethical Matrix draws on Beauchamp and Childress’ (2013) principlism for these principles. The Ethical Matrix displays how these principles are conceptualized for a set of stakeholders in a two-dimensional matrix.


In Kaiser and Forsberg’s structure of the Ethical Matrix, the number of stakeholders may vary from case to case, since the relevant stakeholders are connected to the issue at hand, and the conceptualization of the principles allows for flexibility. In a related manner, Kaiser et al. (2007) suggested experimenting with a bottom-up version where the participants also decided on the principles.


Furthermore, the Ethical Matrix can be carried out in two stages. In the first stage, the users define what the principles signify in this case for their respective groups, and in the second stage, the impacts of a novel technology are then assessed as they relate to these significations (Mepham, Kaiser, Thorstensen, Tomkins, & Millar 2006). The Ethical Matrix originates from ethics in biotechnology but has been applied to other cases as well, such as carbon capture and storage and radioactive waste management (Boucher & Gough 2012; Cotton 2009).


As Reijers et al. and Cotton point out, the Ethical Matrix could well be supplemented with structured stakeholder selection in order to avoid overlooking important stakeholders (Cotton 2014; Reijers et al. 2017). It would seem that the Ethical Matrix assumes that all members in a stakeholder group will assess the impact of a novel artifact in the same manner, and/or that the adherence to a stakeholder group is decisive for a valuation (Cotton 2014). Even though the Ethical Matrix simplifies the issues at stake by locking them into a matrix, Kaiser et al. found that “although the Ethical Matrix does not emerge as a very simple tool to use as a participatory ethical framework, it does show its potential to structure ethical concerns under varying conditions” (2007: 78). A main advantage of the Ethical Matrix is thus the assessment of how specific technological interventions affect the values from Thorstensen (2017b) . This assessment can be done through weighting or through assigning numeric values. However, “[t]here is no hard algorithm for such balancing, but the matrix structure will make the trade-offs appear much more clearly” (Forsberg et al. 2017). Nevertheless, deciding how to proceed depends on the group or a decision-maker to provide a verdict (Kermisch & Depaus 2018).


Neither ETIA nor the Ethical Matrix includes elements such as frugality, health relevance or social entrepreneurship directly. These could be introduced as dimensions under well-being and fairness in the Ethical Matrix.


In Table 1, I present an attempt at characterizing the strengths and weaknesses of the approaches discussed above in a numeric fashion. It should be obvious that Table 1 is a heuristic tool for the author and readers. It does not display an objective evaluation of these approaches and any form of summarization to reach a final score constitutes a misuse of the displayed information.


Table 1. Strengths and weaknesses of the reviewed approaches related to the indicators for selecting an assessment framework 




What seems to differentiate these socio-ethical assessment frameworks from RRI is the lack of reflection on purposes – with the exception of the Socratic approach. They differ from RiH on frugality and social entrepreneurship. With respect to the RRI keys – and with the noted exception of “open access” and “science education” – the four frameworks are rather similar except for the lack of inclusion of sustainability in the Socratic approach and the HTA Core model. The lack of sustainability issues in assessments on technologies outside the fields of biology and chemistry was also a finding in the EST-Frame project (Thorstensen et al. 2013) . This poses a challenge for RRI practitioners on how to incorporate such aspects into a product assessment. On a more general note, it may be possible to supplement all the methods listed above with novel elements, to the extent that these do not interfere substantially with the inherent systems in the approaches.


Based on the arguments above, ETIA, the Ethical Matrix, the HTA core model and the Socratic approach all have different virtues, in addition to some downsides. In terms of the different criteria, the approaches are all transparent and they all seem comprehensive with a strong emphasis on including different points of view. Furthermore, they all address the concern of other possible impacts in a structured manner raised by Stilgoe et al. with respect to a reflection on future impacts, but only the Socratic approach allows for changes in such impacts and reflections on ignorance. The Ethical Matrix is the only approach that does not have a specific focus on training or an element pointing towards distribution of responsibilities. Otherwise, all the candidates allow for assessing the substantive themes.


Conclusion 

In order for ETIA, the Ethical Matrix and the HTA core model to be fully developed into RRI tools, they need to be adapted to include uncertainties and ignorance. Methodically, they could all achieve this by modifying their output structure to highlight these elements. I would assume that fulfilling the goals of science education depends to some degree on convincing the technology developers or the natural scientists to accept the methodology and to include the results in curricula in order to connect to praxis (Mejlgaard et al. 2018). Open Science – or Open Access – presupposes discussions regarding ownership and distribution of the whole epistemic scientific endeavor and has a series of meanings in different contexts (Opening science  2014). This necessitates widening the concepts of who benefits (cui bono) and distributive justice to include the epistemic dimensions in addition to the material ones.


RRI and HTA are both future-oriented with respect to the process dimensions, as they typically concern some level of policy-relevant advice. The structure of HTA outcomes might well inform RRI in the clear division of epistemic domains, whereas RRI might inform HTA in highlighting that one cannot assume that the medical epistemic or the risk epistemic should have superior value compared to other epistemic domains (Refolo et al. 2016). HTAs make more modest claims than RRI when it comes to addressing “significant societal needs and challenges” since it is oriented towards a single item and its comparators. RRI can supply HTA with clearer reflections on the purposes of an assessment, the social meanings of both cure and care, and alternate framings of illness and disease (Cuijpers & van Lente 2015; Moors & Peine 2016). RiH contributions regarding the business model, frugality and, especially, health relevance are welcome specifications from RRI to HTA. They share a commitment to reflection on alternatives, but this has a more solid foundation in HTA due to the inherent demands for evidence of the novel technology’s superior effect. Consequently, the meeting between these two processes might be mutually beneficial.


Regarding user-friendliness or usability, the modified Socratic approach and the Ethical Matrix seem to have the simplest structure, while ETIA and the HTA core model pose a range of very specific questions with several sub-questions. However, practice has shown that the ETIA lists could be reduced to a set of specific issues to be investigated further. The Socratic approach and the Ethical Matrix are intentionally very general, but this does not in itself indicate ease of use, as documented by Kaiser et al. (2007) with respect to the Ethical Matrix.


It is fair to say that the Socratic approach covers all aspects of the health research life cycle and has great utility in comparing how products might differ in an ethically robust manner. ETIA, on the other hand, targets innovations in ICT far more specifically, and the questions it poses make little sense outside of this precise field. The HTA Core model provides a custom fit between the socio-ethical dimension and the effectiveness study. The Ethical Matrix provides a systematic and comprehensive overview of conflicts and commonalities between stakeholders or issues.


Consequently, the Ethical Matrix, the Socratic approach, the ETIA and the HTA Core Model could all serve well as product assessment methods within RRI. The choice is then left open to the assessors to determine the main concerns for the assessment. The Socratic approach and the Ethical Matrix have an appealing accessibility, whereas ETIA and the HTA Core Model are more comprehensive. ETIA has a particular emphasis on privacy and ICT, whereas the Socratic approach and the HTA Core Model have a general orientation towards health and health technologies, and the Ethical Matrix is a generalist approach. The HTA Core Model has a systematic and structured integration into HTA, whereas the Ethical Matrix, the ETIA and the Socratic approach might challenge some of preset structures in an HTA.


Selecting an assessment approach for a specific project and product thus needs to take into account the particulars of that project or product. In the context of the current project, we should keep in mind Stilgoe et al.’s statement, that “the (often implicit) evocation of the highest principles that engagement might ideally fulfil can make it difficult to acknowledge and pay serious attention to the varieties of engagement that are very much less than perfect but still somehow ‘good”’ (2013: 1572). Whereas the Socratic approach and the Ethical Matrix encompass the ethics in their own way and in a multiperspectivistic manner, the ETIA and the HTA Core Model include the different topics from the literature and the stakeholder engagement in a very structured but theoretical way. Especially the prioritized emphasis on privacy, the inclusion of elements on empowerment and a clear social dimension make the ETIA well suited for being applied as an RRI framework on assistive technologies. The ETIA also adequately covers standard HTA topics such as economic efficiency and implications for services, making it compatible with this already established approach in the health field.

Acknowledgements 

The project, ‘The Assisted Living Project: Responsible innovations for dignified lives at home for persons with mild cognitive impairment or dementia’, is financed by the Research Council of Norway under the SAMANSVAR strand (247620/O70). I am very grateful to all the persons who have dedicated time to the Assisted Living project, and I thank the reviewers for valuable improvements and suggestions.

 
Notes

1Wright (2011) abbreviated Ethical Impact Assessment as EIA, but in my view, this might be misunderstood as referring to Environmental Impact Assessment, hence the choice of ETIA in this paper.

2 For a description on how the Assisted Living project meets the process criteria listed here, see Forsberg and Thorstensen (2018).

3 Such a synthesis should, theoretically speaking, be feasible as the two parties seem to share what Rommetveit et al. (in press) call a system-based style of reasoning.

4 See Koops (2015) for further analysis on the meaning of “product” and “process” in RRI and how these terms relate to the underlying approach to RRI.

5 The full references for all the claims can be found in Thorstensen (2017b).

6 The domains and the dimensions are: Population health (Health relevance, Ethical, legal, and social issues, and Health equity), Health system (Inclusiveness, Responsiveness, and Level of care), Economic (Frugality), Organizational (Business model), and Environmental (Eco-responsibility) (Pacifico Silva et al. 2018).

7 Reijers et al. (2017) point out that ex post assessment should have a methodological solution for solving value conflicts as well as more general guidance on how to choose among different socio-technical alternatives. As I understand Reijers et al., this first demand is a move from evaluation to recommendation – or from assessment to appraisal in the HTA terminology (Sandman & Heintz 2014). However, in a transdisciplinary project such as the Assisted Living project, we would strive towards providing advice, such as laid out by Jan Schmidt in his work on interdisciplinary work, “it does not solve the problems itself: It supports a decision but does not provide the actual decision” (Schmidt 2011: 259). As for the second recommendation, Reijers et al. note that “ex post methods offer inadequate guidance on how to choose between sociotechnical alternatives or courses of action based on an ethical analysis” (2017: 22). This recommendation might be a possible criterion, but if all the methods developed to date share this flaw, it seems unlikely that any candidates would meet it.

8 I will not argue that this selection is exhaustive, but I believe that it encompasses the main approaches suitable for product assessment of assistive technologies. See Thorstensen (2017a, 2017b) for details on the reviewed literature.

9 Hofmann writes that, “moral aspects are frequently viewed as an ‘add on’ to ‘the real thing,’ that is, systematic reviews [of effect]” (2005b: 2).

10 The remaining frameworks were Real-Time Technology Assessment (Guston & Sarewitz 2002), ETICA framework for ethical issues in ICT (Stahl 2011); EFORRT project’s ethical framework for home telecare (EFORTT 2011), Value Sensitive Design (van den Hoven 2013).

11 https://meka.thl.fi/htacore/

12 The ethical questions in the HTA Core Model are:

  1. What are the symptoms and the burden of disease or health condition for the patient?
  2. What are the known and estimated benefits and harms for patients when implementing or not implementing the technology?
  3. What are the benefits and harms of the technology for relatives, other patients, organisations, commercial entities, society, etc.?
  4. Are there any other hidden or unintended consequences of the technology and its applications for patients/users, relatives, other patients, organisations, commercial entities, society, etc.?
  5. Are there any ethical obstacles for evidence generation regarding the benefits and harms of the intervention?
  6. Is the technology used for individuals that are especially vulnerable?
  7. Does the implementation or use of the technology affect the patient´s capability and possibility to exercise autonomy?
  8. Is there a need for any specific interventions or supportive actions concerning information in order to respect patient autonomy when the technology is used?
  9. Does the implementation or withdrawal of the technology challenge or change professional values, ethics or traditional roles?
  10. Does the implementation or use of the technology affect human dignity?
  11. Does the implementation or use of the technology affect the patient’s moral, religious or cultural integrity?
  12. Does the technology invade the sphere of privacy of the patient/user?
  13. How does implementation or withdrawal of the technology affect the distribution of health care resources?
  14. How are technologies with similar ethical issues treated in the health care system?
  15. Are there factors that could prevent a group or person from gaining access to the technology?
  16. Does the implementation or use of the technology affect the realisation of basic human rights?
  17. Can the use of the technology pose ethical challenges that have not been considered in the existing legislations and regulations?
  18. What are the ethical consequences of the choice of endpoints, cut-off values and comparators/controls in the assessment?
  19. Are there any ethical problems related to the data or the assumptions in the economic evaluation?
  20. What are the ethical consequences of conducting the technology assessment at this point of time?

13 The SATORI project, satoriproject.eu, and the European Committee for Standardization (2017) uses the term “Ethical Impact Assessment” denoting a general process for ethical evaluations according to the steps listed by Wright (2011, 2015).

14 The full list of themes is: informed consent, safety, social solidarity, inclusion and exclusion, isolation and substitution of human contact, discrimination and social sorting, universal service, accessibility, value sensitive design, sustainability, and equality and fairness (social justice). For Privacy and data protection Wright specifies the areas of: collection limitation (data minimisation) and retention, data quality, purpose specification, user limitation, confidentiality, security and protection of data, transparency (openness), individual participation and access to data, anonymity, privacy of personal communications: monitoring and location tracking, privacy of the person, and privacy of personal behaviour.

15 See Wright (2015) for the full list of steps.

16 (Brinch 2003; Droste, Herrmann-Frank, Scheibler, & Krones 2011; Hofmann 2008a, 2008b, 2010, 2017a, 2017b; Hofmann et al. 2014; Hofmann, Haustein, & Landeweerd 2017; Holte 2007; Lauvrak et al. 2012; Sánchez, Taylor, & Bing-Jonsson 2017; Vist 2007)

17 (European Committee for Standardization 2017; Rodrigues et al. 2016; Shelley-Egan, Wright, & Wadhwa 2016; Venier et al. 2013; Wright 2011, 2015; Wright & Friedewald 2013))

18 (Assasi, Tarride, O’Reilly, & Schwartz 2016; EUnetHTA 2016a, 2016b, 2017a, 2017b, 2017c, 2017d)

19 (Beekman et al. 2006; Boucher & Gough 2012; Cotton 2009, 2014; Forsberg 2007a, 2007b, 2007c; Forsberg & Kaiser 2000; Jensen, Forsberg, Gamborg, Millar, & Sandøe 2011; Kaiser & Forsberg 2001; Kermisch & Depaus 2018; Mepham 2000; Mepham et al. 2006)



References

Antoñanzas, F., Terkola, R., & Postma, M. (2016). The Value of Medicines: A Crucial but Vague Concept. PharmacoEconomics, 34(12), 1227-1239. Crossref

Assasi, N., Schwartz, L., Tarride, J.-E., Campbell, K., & Goeree, R. (2014). Methodological guidance documents for evaluation of ethical considerations in health technology assessment: a systematic review. Expert Review of Pharmacoeconomics & Outcomes Research, 14(2), 203-220. Crossref

Assasi, N., Tarride, J.-E., O’Reilly, D., & Schwartz, L. (2016). Steps toward improving ethical evaluation in health technology assessment: a proposed framework. BMC Medical Ethics, 17, 34. Crossref

Banta, D. (2003). The development of health technology assessment. Health Policy, 63(2), 121-132. Crossref

Beauchamp, T. L., & Childress, J. F. (2013). Principles of biomedical ethics (7th ed ed.). New York: Oxford University Press.

Beekman, V., de Bakker, E., Baranzke, H., Baune, O., Deblonde, M., Forsberg, E.-M., . . . Sandø, P. (2006). Ethical bio-technology assessment tools for agriculture and food production: final report ethical bio-TA tools (QLG6-CT-2002-02594). Retrieved from http://library.wur.nl/WebQuery/wurpubs/fulltext/2608

Boucher, P., & Gough, C. (2012). Mapping the ethical landscape of carbon capture and storage. Poiesis & Praxis, 9(3-4), 249-270. Crossref

Brey, P. A. E. (2012). Anticipating ethical issues in emerging IT. Ethics and Information Technology, 14(4), 305-317. Crossref

Brinch, L. (2003). Terapeutisk bruk av hematopoietiske stamceller fra navlestrengsblod: medisinsk metodevurdering basert på egen litteraturgranskning (Vol. SFT78 A3404). Oslo: Senter for medisinsk metodevurdering.

Cotton, M. (2009). Evaluating the 'Ethical Matrix' as a Radioactive Waste Management Deliberative Decision-Support Tool. Environmental Values, 18(2), 153-176. Crossref

Cotton, M. (2014). Ethics and technology assessment: a participatory approach (1st edition ed.). New York: Springer.

Cuijpers, Y., & van Lente, H. (2015). Early diagnostics and Alzheimer's disease: Beyond ‘cure’ and ‘care’. Technological Forecasting and Social Change, 93, 54-67. Crossref

Cuijpers, Y., van Lente, H., Boenink, M., & Moors, E. H. M. (2014). Quandaries of Responsible Innovation: The Case of Alzheimer’s Disease. In J. van den Hoven, N. Doorn, T. Swierstra, B.-J. Koops, & H. Romijn (Eds.), Responsible Innovation 1: Innovative Solutions for Global Issues (pp. 239-254): Springer.

Dittrich, D., Kenneally, E., & Bailey, M. (2013). Applying Ethical Principles to Information and Communication Technology Research: A Companion to the Menlo Report (ID 2342036). Retrieved from Rochester, NY: https://papers.ssrn.com
/abstract=2342036

Droste, S., Herrmann-Frank, A., Scheibler, F., & Krones, T. (2011). Ethical issues in autologous stem cell transplantation (ASCT) in advanced breast cancer: A systematic literature review. BMC Medical Ethics, 12, 6. Crossref

EFORTT. (2011). EFORTT: Ethical Frameworks for Telecare technologies for older people at home Project 217787 technologies for older people at home Project 217787 (Deliverable 7: Final Research Report). Retrieved from http://www.lancaster.ac.uk/efortt
/documents/Deliverable%207%20Final%20Research%20report.pdf

EUnetHTA. (2016a). HTA Core Model ® version 3.0 (Work Package 8). Retrieved from https://www.eunethta.eu/wp-content/uploads
/2018/03/HTACoreModel3.0-1.pdf

EUnetHTA. (2016b). Wearable cardioverter-defibrillator (WCD) therapy in primary and secondary prevention of sudden cardiac arrest in patients at risk (OTCA01). Retrieved from https://www.eunethta.eu/wp-content/uploads/2018/01
/Assessment_WCD_final.pdf

EUnetHTA. (2017a). Antibacterial-coated sutures versus non-antibacterial-coated sutures for the prevention of abdominal, superficial and deep, surgical site infection (SSI) (OTCA02). Retrieved from https://www.eunethta.eu/wp-content/uploads
/2018/01/OTCA02_Antibacterial-coated-sutures-for-the-
prevention-of-abdominal-SSI_0.pdf

EUnetHTA. (2017b). Midostaurin with Standard Chemotherapy in Flt3-Positive Acute Myeloid Leukaemia (PTJA01). Retrieved from https://www.eunethta.eu/wp-content/uploads/2018/01/Project-
Plan-PTJA01-Midostaurin-for-AML-FINAL_1_0.pdf

EUnetHTA. (2017c). Regorafenib (Stivarga©) indicated as monotherapy for the treatment of adult patients with hepatocellular carcinoma (HCC) who have been previously treated with sorafenib treatment (PTJA02). Retrieved from https://www.eunethta.eu/wp-content
/uploads/2018/01/Regorafenib_Final-Assessment-Report.pdf

EUnetHTA. (2017d). Repetitive transcranial magnetic stimulation for treatment-resistant major depression (OTCA05). Retrieved from https://www.eunethta.eu/wp-content/uploads/2018/01
/OTCA05_Repetitive-transcranial-magnetic-stimulation-
for-TRD.pdf

European Commission. (2012). Responsible research and innovation Europe's ability to respond to societal challenges: Directorate-General for Research and Innovation.

European Committee for Standardization. (2017). Ethics assessment for research and innovation - Part 2: Ethical impact assessment framework (CWA 17145-2). Retrieved from http://satoriproject.eu
/publications/cwa-part-2/

Flaming, D. (2017). Appropriate ethics review is required. Healthcare Management Forum, 30(1), 46-48. Crossref

Forsberg, E.-M. (2007a). A deliberative ethical matrix method: justification of moral advice on genetic engineering in food production. (Dr. Art), Faculty of Humanities, University of Oslo, Oslo. Available from Bibsys

Forsberg, E.-M. (2007b). Pluralism, The Ethical Matrix, and Coming to Conclusions. Journal of Agricultural and Environmental Ethics, 20(5), 455-468. Crossref

Forsberg, E.-M. (2007c). Value Pluralism And Coherentist Justification of Ethical Advice. Journal of Agricultural and Environmental Ethics, 20(1), 81-97. Crossref

Forsberg, E.-M., & Kaiser, M. (2000). Norske fiskerier mot 2020 : verdier og strategier. Retrieved from Oslo:

Forsberg, E.-M., Shelley-Egan, C., Thorstensen, E., Landeweerd, L., & Hofmann, B. (2017). Evaluating Ethical Frameworks for the Assessment of Human Cognitive Enhancement Applications. Cham, Heidelberg, New York, Dordrecht, London: Springer International Publishing.

Forsberg, E.-M., & Thorstensen, E. (2018). A Report from the Field: Doing RRI from Scratch in an Assisted Living Technology Research and Development Project. In F. Ferri, N. Dwyer, S. Raicevich, P. Grifoni, H. Altiok, H. T. Andersen, Y. Laouris, & C. Silvestri (Eds.), Governance and Sustainability of Responsible Research and Innovation Processes (pp. 19-26). Cham: Springer International Publishing.

Guston, D. H., & Sarewitz, D. (2002). Real-time technology assessment. Technology in Society, 24(1–2), 93-109. Crossref

Hofmann, B. (2005a). On value-judgements and ethics in health technology assessment. Poiesis & Praxis, 3(4), 277-295. Crossref

Hofmann, B. (2005b). Toward a procedure for integrating moral issues in health technology assessment. International Journal of Technology Assessment in Health Care, 21(3), 312-318.

Hofmann, B. (2008a). Etikk i vurdering av helsetiltak: utvikling av en metode for å synliggjøre etiske utfordringer ved vurdering av helsetiltak (Vol. nr. 26-2008). Oslo: Nasjonalt kunnskapssenter for helsetjenesten.

Hofmann, B. (2008b). Vaksiner mot humant papillomavirus (HPV): etiske aspekter ved innføring av profylaktiske HPV-vaksiner (Vol. nr. 22-2008). Oslo: Nasjonalt kunnskapssenter for helsetjenesten.
Hofmann, B. (2010). Etiske utfordringer med velferdsteknologi. Oslo: Nasjonalt kunnskapssenter for helsetjenesten.

Hofmann, B. (2013). Ethical Challenges with Welfare Technology: A Review of the Literature. Science and Engineering Ethics, 19(2), 389-406. Crossref

Hofmann, B. (2017a). Ethical issues with colorectal cancer screening—a systematic review. Journal of Evaluation in Clinical Practice, 23(3), 631-641. Crossref

Hofmann, B. (2017b). Toward a Method for Exposing and Elucidating Ethical Issues with Human Cognitive Enhancement Technologies. Science and Engineering Ethics, 23(2), 413-429. Crossref

Hofmann, B., Droste, S., Oortwijn, W., Cleemput, I., & Sacchini, D. (2014). Harmonization of Ethics in Health Technology Assessment: A Revision of the Socratic Approach. International Journal of Technology Assessment in Health Care; Cambridge, 30(1), 3-9. Crossref

Hofmann, B., Haustein, D., & Landeweerd, L. (2017). Smart-Glasses: Exposing and Elucidating the Ethical Issues. Science and Engineering Ethics, 23(3), 701-721. Crossref

Hofmann, B., Oortwijn, W., Lysdahl, K. B., Refolo, P., Sacchini, D., van der Wilt, G. J., & Gerhardus, A. (2015). Integrating Ethics in Health Technology Assessment: many ways to Rome. International Journal of Technology Assessment in Health Care, 31(03), 131-137.

Holte, T. O. (2007). Mannlig infertilitet: intracytoplasmatisk spermieinjeksjon (ICSI) med spermier uthentet fra bitestikkel eller testikkel (Vol. nr. 7-2007). Oslo: Nasjonalt kunnskapssenter for helsetjenesten.

INAHTA. (2016, 2016/08/05/). HTA Tools & Resources. INAHTA. Retrieved from http://www.inahta.org/hta-tools-resources/

Jensen, K. K., Forsberg, E.-M., Gamborg, C., Millar, K., & Sandøe, P. (2011). Facilitating Ethical Reflection Among Scientists Using the Ethical Matrix. Science and Engineering Ethics, 17(3), 425-445. Crossref

Kaiser, M., & Forsberg, E.-M. (2001). Assessing Fisheries – Using an Ethical Matrix in a Participatory Process. Journal of Agricultural and Environmental Ethics, 14(2), 191-200. Crossref

Kaiser, M., Millar, K., Thorstensen, E., & Tomkins, S. (2007). Developing the ethical matrix as a decision support framework: GM fish as a case study. Journal of Agricultural and Environmental Ethics, 20(1), 65-80. Crossref

Kenneally, E., Bailey, M., & Maughan, D. (2010, 2010/01/25/). A Framework for Understanding and Applying Ethical Principles in Network and Security Research. Paper presented at the International Conference on Financial Cryptography and Data Security.

Kermisch, C., & Depaus, C. (2018). The Strength of Ethical Matrixes as a Tool for Normative Analysis Related to Technological Choices: The Case of Geological Disposal for Radioactive Waste. Science and Engineering Ethics, 24(1), 29-48. Crossref

Koops, B.-J. (2015). The Concepts, Approaches, and Applications of Responsible Innovation. In B.-J. Koops, I. Oosterlaken, H. Romijn, T. Swierstra, & J. van den Hoven (Eds.), Responsible Innovation 2 (pp. 1-15). Cham: Springer International Publishing.

Lampe, K., Mäkelä, M., Garrido, M. V., Anttila, H., Autti-Rämö, I., Hicks, N. J., . . . Kristensen, F. B. (2009). The HTA Core Model: A novel method for producing and reporting health technology assessments. International Journal of Technology Assessment in Health Care, 25(Supplement S2), 9-20. Crossref

Lauvrak, V., Norderhaug, I. N., Hagen, G., Acharya, G., Forus, A., Hofmann, B., . . . Nasjonalt kunnskapssenter for, h. (2012). Tidlig ultralyd i svangerskapsomsorgen (Vol. Januar 2012). Oslo: Nasjonalt kunnskapssenter for helsetjenesten.

Lysdahl, K. B., Brereton, L., Oortwijn, W., Mozygemba, K., Refolo, P., Sacchini, D., . . . Hofmann, B. (2016). Guidance to Ethical Aspects. In K. B. Lysdahl, K. Mozygemba, J. Burns, J. B. Chilcott, J. B. Brönneke, & B. Hofmann (Eds.), Guidance for assessing effectiveness, economic aspects, ethical aspects, socio-cultural aspects and legal aspects in complex technologies (pp. 60-75).

Mejlgaard, N., Christensen, M. V., Strand, R., Buljan, I., Carrió, M., Cayetano i Giralt, M., . . . Wuketich, M. (2018). Teaching Responsible Research and Innovation: A Phronetic Perspective. Science and Engineering Ethics.Crossref

Mepham, B. (2000). A Framework for the Ethical Analysis of Novel Foods: The Ethical Matrix. Journal of Agricultural and Environmental Ethics, 12(2), 165-176. Crossref

Mepham, B., Kaiser, M., Thorstensen, E., Tomkins, S., & Millar, K. (2006). Ethical Matrix Manual. Retrieved from The Hague: http://library.wur.nl/WebQuery/wurpubs/426154

Moors, E., & Peine, A. (2016). Valuing Diagnostic Innovations: Towards Responsible Health Technology Assessment. In M. Boenink, H. van Lente, & E. Moors (Eds.), Emerging Technologies for Diagnosing Alzheimer's Disease (pp. 245-261). London: Palgrave Macmillan UK.

Mozygemba, K., Hofmann, B., Lysdahl, K. B., Pfadenhauer, L. M., Van Der Wilt, G. J., & Gerhardus, A. (2016). Guidance to Soci-Cultural Aspects. In K. B. Lysdahl, K. Mozygemba, J. Burns, J. B. Chilcott, J. B. Brönneke, & B. Hofmann (Eds.), Guidance for assessing effectiveness, economic aspects, ethical aspects, socio-cultural aspects and legal aspects in complex technologies (pp. 76-100).

Opening science: the evolving guide on how the internet is changing research, collaboration and scholarly publishing. (2014).  (S. Bartling & S. Friesike Eds.). Heidelberg: Springer Open.

ORBIT. (2017). Project Self-Assessment Tool | Orbit RRI. Orbit. Retrieved from https://www.orbit-rri.org/self-assessment-
tool-survey/

Owen, R. (2014). The UK Engineering and Physical Sciences Research Council's commitment to a framework for responsible innovation. Journal of Responsible Innovation, 1(1), 113-117. Crossref

Owen, R., Macnaghten, P., & Stilgoe, J. (2012). Responsible research and innovation: From science in society to science for society, with society. Science and Public Policy, 39(6), 751-760. Crossref

Pacifico Silva, H., Lehoux, P., Miller, F. A., & Denis, J.-L. (2018). Introducing responsible innovation in health: a policy-oriented framework. Health Research Policy and Systems, 16(1). Crossref

Pols, J., & Willems, D. (2011). Innovation and evaluation: taming and unleashing telecare technology: Taming and unleashing telecare technology. Sociology of Health & Illness, 33(3), 484-498. Crossref

Refolo, P., Sacchini, D., Brereton, L., Gerhardus, A., Hofmann, B., Lysdahl, K. B., . . . Van Der Wilt, G. J. (2016). Why is it so difficult to integrate ethics in Health Technology Assessment (HTA)? The epistemological viewpoint. European Review for Medical and Pharmacological Sciences, 20, 4202-4208.

Reijers, W., Wright, D., Brey, P., Weber, K., Rodrigues, R., O’Sullivan, D., & Gordijn, B. (2017). Methods for Practising Ethics in Research and Innovation: A Literature Review, Critical Analysis and Recommendations. Science and Engineering Ethics. Crossref

Reuzel, R. (2004). Interactive technology assessment of paediatric cochlear implantation. Poiesis & Praxis, 2(2-3), 119-137. Crossref

Ribeiro, B. E., Smith, R. D. J., & Millar, K. (2016). A Mobilising Concept? Unpacking Academic Representations of Responsible Research and Innovation. Science and Engineering Ethics. Crossref

Roberts, C., & Mort, M. (2009). Reshaping what counts as care: Older people, work and new technologies. ALTER - European Journal of Disability Research / Revue Européenne de Recherche sur le Handicap, 3(2), 138-158. Crossref

Rodrigues, R., Wright, D., & Kroener, I. (2016). Review of ethical issues affecting PULSE (Ethical Impact Assessment Report) (D. 8.2.). Retrieved from http://www.webcitation.org/73j423nue

Rommetveit, K., van Dijk, N., Gunnarsdottir, K., O’Riordan, K., Gutwirth, S., Strand, R., & Wynne, B. (in press). Working responsibly across boundaries? Some practical and theoretical lessons. In R. von Schomberg & J. Hankins (Eds.), International Handbook on Responsible Innovation: Edward Elgar.

Sánchez, V. G., Taylor, I., & Bing-Jonsson, P. C. (2017). Ethics of Smart House Welfare Technology for Older Adults. International Journal of Technology Assessment in Health Care, 1-9. Crossref

Sandman, L., & Heintz, E. (2014). Assessment vs. appraisal of ethical aspects of health technology assessment: can the distinction be upheld? GMS Health Technology Assessment, 10. Crossref

Schmidt, J. C. (2011). What is a problem? Poiesis & Praxis, 7(4), 249-274. Crossref

Schot, J., & Rip, A. (1997). The past and future of constructive technology assessment. Technological Forecasting and Social Change, 54(2), 251-268.

Shelley-Egan, C., Wright, D., & Wadhwa, K. (2016). Plan for Ethical Impact Assessment (D. 8.1.). Retrieved from http://www.webcitation.org/73j3yOm4C

Stahl, B. C. (2011). IT for a better future: how to integrate ethics, politics and innovation. In R. Von Schomberg (Ed.), Towards Responsible Research and Innovation in the Information and Communication Technologies and Security Technologies Field (pp. 17-33). Luxembourg: Publication Office of the European Union.

Stahl, B. C., & Coeckelbergh, M. (2016). Ethics of healthcare robotics: Towards responsible research and innovation. Robotics and Autonomous Systems, 86, 152-161. Crossref

Stahl, B. C., Obach, M., Yaghmaei, E., Ikonen, V., Chatfield, K., & Brem, A. (2017). The Responsible Research and Innovation (RRI) Maturity Model: Linking Theory and Practice. Sustainability, 9(6), 1036. Crossref

Stilgoe, J., Owen, R., & Macnaghten, P. (2013). Developing a framework for responsible innovation. Research Policy, 42(9), 1568-1580. Crossref

Strand, R., Spaapen, J., Bauer, M. W., Hogan, E., Revuelta, G., & Stagl, S. (2015). Indicators for promoting and monitoring Responsible Research and Innovation (EUR 26866 EN). Retrieved from Brussels: http://www.webcitation.org/73j3tEbDS

Thorstensen, E. (2017a). Literature review of responsible research and innovation on assisted living technologies for the Assisted Living Project. Retrieved from http://www.webcitation.org/73j3mwvt7

Thorstensen, E. (2017b). Responsible Help at Home: Establishing Indicators for a Product Assessment Methodology. In D. M. Bowman, A. M. Dijkstra, C. Fautz, J. Guivant, K. Konrad, C. Shelley-Egan, & S. Woll (Eds.), The Politics and Situatedness of Emerging Technologies (pp. 167-182). Berlin: IOS Press.

Thorstensen, E., Boucher, P., Forsberg, E.-M., de Bakker, E., Boegaardt, M.-J., Bryndum, N., . . . Nielsen, R. Ø. (2013). EST-Frame deliverable 1.2: The current and future context for EST analysis. Retrieved from http://www.webcitation.org/73FoGL2UD

Thygesen, H. (2009). Technology and good dementia care. A study of technology and ethics in everyday care practice. (Doctoral thesis), Centre for Technology, Innovation and Culture (TIK), University of Oslo, Oslo. Retrieved from https://brage.bibsys.no/xmlui/handle
/11250/2463491 Available from brage.bibsys.no

Topo, P. (2008). Technology Studies to Meet the Needs of People With Dementia and Their Caregivers: A Literature Review. Journal of Applied Gerontology, 28(1), 5-37. Crossref

van den Hoven, J. (2013). Value Sensitive Design and Responsible Innovation. In R. Owen, J. Bessant, & M. Heintz (Eds.), Responsible innovation (pp. 75-83). Chichester, UK: John Wiley & Sons, Ltd.

Venier, S., Mordini, E., Friedewald, M., Schütz, P., Hallinan, D., Wright, D., . . . Turnheim, B. (2013). Final Report – A Privacy and Ethical Impact Assessment Framework for Emerging Sciences and Technologies (Deliverable 4, PRESCIENT Project). Retrieved from www.prescient-project.eu

Vist, G. E. (2007). Helseeffekt av nyfødtscreening for medfødte stoffskiftesykdommer (Vol. nr. 22-2007). Oslo: Nasjonalt kunnskapssenter for helsetjenesten.

von Schomberg, R. (2011). Prospects for Technology Assessment in a framework of responsible research and innovation. Technikfolgen abschätzen lehren: Bildungspotenziale transdisziplinärer Methoden, 39-61. Crossref

von Schomberg, R. (2013). A Vision of Responsible Research and Innovation. Responsible Innovation: Managing the Responsible Emergence of Science and Innovation in Society, 51-74.

Wickson, F., & Forsberg, E.-M. (2015). Standardising Responsibility? The Significance of Interstitial Spaces. Science and Engineering Ethics, 21(5), 1159-1180. Crossref

Wouters, E. J. M., Weijers, T. C. M., & Finch, T. L. (2017). Successful Implementation of Technological Innovations in Health Care Organizations. In J. van Hoof, G. Demiris, & E. J. M. Wouters (Eds.), Handbook of Smart Homes, Health Care and Well-Being (pp. 179-189). Cham: Springer International Publishing.

Wright, D. (2011). A framework for the ethical impact assessment of information technology. Ethics and Information Technology, 13(3), 199-226. Crossref

Wright, D. (2015). Ethical Impact Assessment. In J. B. Holbrook & C. Mitcham (Eds.), Ethics, science, technology, and engineering: a global resource (pp. 163-167).

Wright, D., & Friedewald, M. (2013). Integrating privacy and ethical impact assessments. Science and Public Policy, 40(6), 755-766. Crossref