In his Bachelor Thesis, Viktor Lakic investigated the decay happening in datasets when the resources that Web-URLs point to become unavailable. This Link-Rot can cause problems for reproducibility, as datasets can shrink over time, potentially changing the outcome of experiments which use them. A paper based on the data that Viktor collected in his thesis, co-authored by Luca Rossetto and Abraham Bernstein, was recently presented at the 2023 International Conference on Multimedia Modeling in the Special Session on ‘Multimedia Datasets for Repeatable Experimentation’. The paper was awarded the ‘Best Special Session Paper Award’, honoring the best contribution across all special sessions of the conference. Congratulations!

In order to facilitate and coordinate this evaluation campaign, we implemented a Web-based software infrastructure – dubbed Alan’s Speakeasy – that provides a graphical interface for human users to connect and have conversations in chatrooms and allows students to connect their conversational agents. Speakeasy also allows students to evaluate the conversational agents they talk to, using a survey that asks users to assess the accuracy of the conversational agents.   

We are honored to have been granted new funding by the UZH Teaching Fund via the innovation program that will allow us to extend the scope of Speakeasy. This project has a two-fold goal: Firstly, we would like to extend the implementation of the current software infrastructure – Speakeasy – to incorporate further features that will make the software more usable and more elaborate for running evaluation campaigns for conversational agents within the ATAI class.  Secondly, we would like to expand the scope of our software to reach a larger audience. Specifically, we would like to make our software useful for other organizations. We would like other research groups working on various aspects of AI (inside and outside UZH) to be able to reuse our software for teaching practical aspects of AI. 

The paper, which looks into how the kind of expert giving the advice — i.e., a human or an AI advisor — influences trust, perceived responsibility, and reliance.

For more information about the paper, please check it out in our on-line library, over at the ACM Digital Library, and the video presentation prepared by Suzanne.

For your convenience you can also find the abstract right here:

While artificial intelligence (AI) is increasingly applied for decision-making processes, ethical decisions pose challenges for AI applications. Given that humans cannot always agree on the right thing to do, how would ethical decision-making by AI systems be perceived and how would responsibility be ascribed in human-AI collaboration? In this study, we investigate how the expert type (human vs. AI) and level of expert autonomy (adviser vs. decider) influence trust, perceived responsibility, and reliance. We find that participants consider humans to be more morally trustworthy but less capable than their AI equivalent. This shows in participants’ reliance on AI: AI recommendations and decisions are accepted more often than the human expert’s. However, AI team experts are perceived to be less responsible than humans, while programmers and sellers of AI systems are deemed partially responsible instead.