On June 8th, 2021, Vishnu Baburajan successfully defended his Ph.D. thesis entitled “Automated Text Analysis on Open-Ended Response Surveys: Measuring Attitudes Regarding Autonomous Vehicles.”

Congratulations, Vishnu!

He was supervised by Professors Francisco C. Pereira and João Abreu (Instituto Superior Técnico – IST, Portugal).

Examiners were Prof. Catarina Silva (University of Coimbra), Prof. Teresa Galvão (Faculty of Engineering of the University of Porto), and Prof. Filipe Moura (IST), and Prof. Paulo da Fonseca Teixeira (IST)

The defense session was chaired by Prof. Luís Picado Santos (IST, Portugal).

Find below the abstract of the thesis! For more information, feel free to contact Vishnu at vibabu@dtu.dk, vishnu.baburajan@tecnico.ulisboa.pt, or directly via his Linkedin page.

For practical reasons, surveys that aim for a large number of respondents tend to restrict themselves to closed-ended responses. Despite potentially bringing richer insights, the use of open-ended questions poses great challenges in terms of extracting useful information while significantly increasing the analysis time. Nevertheless, automatic text analysis techniques could speed up the analysis of open-ended responses. Furthermore, the use of open-ended questions in conjunction with the closed-ended questions is likely to influence the responses to the closed-ended responses.

Considering this, this thesis pursued the following four objectives, a. to analyse if the method of collecting qualitative data influences the survey responses, b. to develop an approach to extract open-ended responses from a survey and process the data, c. to compare the relative performance of the open-ended and closed-ended responses in analysing qualitative data and d. to develop a framework that measures attitudes while allowing respondents to choose their preferred type of question (closed- or open-ended).

This thesis analyses the suitability of using Topic Modelling to extract information from the open-ended responses to measure attitudes. As a case-study throughout the whole thesis, questionnaires that collect information on the attitudes related to Autonomous Vehicles (AV) were used. To do this study, alternative versions of the questionnaires, that consider open- and/or closed-ended questions, was be presented randomly to respondents. Two datasets were collected- 1. 364 responses from India on the intention-to-use Shared AVs and 2. 3002 responses from the USA on the intention-to-use AVs for commute trips. To quantify the relative benefits, we evaluated the relative performance of the alternative versions of the questionnaire to measure attitudes. In this regard, statistical models estimated using each of these independent datasets will be evaluated based on their predictive capability. Besides, the responses to the attitudinal questions will be evaluated to analyse if the mode of asking questions influences the attitudes being measured. Having estimated the models, this thesis also developed a framework that measures attitudes by allowing respondents to choose their preferred type of question.

Our results indicate that the use of open-ended questions before the set of Likert scale questions could alter the responses to the Likert scale questions. The consequence is a reduction in the number of neutral responses and an increase in positive attitude among those answering the questionnaire with open-ended questions. We also evaluated the suitability of using Topic Modelling techniques such as Latent Dirichlet Allocation and supervised Latent Dirichlet Allocation and found them effective, however, we could not find significant improvements in performance with the use of the supervised approach. When comparing the predictive capabilities of the models estimated using questions that used Likert scale responses with and without open-ended questions, the performance of the models was superior for the dataset which had open-ended questions before the Likert scale responses- for datasets from India and the USA. However, we could not find it beneficial to fully replace Likert scale questions with open-ended questions. Using the dataset collected from the USA, we proposed a modelling framework that allows researchers/analysts to allow respondents to answer the questionnaire using question types (closed- or open-ended questions) of their choice. The performance of the proposed model was superior to that of the individually estimated models, particularly for the test set.