Ontology Editing: Requirements for Visualization of Changes and their Impact

The goal of this survey is to gather requirements, preferences,  and suggestions for how to visualise information about ontology changes in real time, within an ontology editor. It is aimed at ontology engineers, curators, editors, and maintainers.

Dear Participant

In this survey, we are looking for you to answer questions about visualization of ontology changes. Specifically, what kind of feedback would you like to receive visually while you are editing an ontology? There are 4 sections in this survey: basic demographics, general visualization preferences, specific questions about mockups, and a wrap up.

This survey is deployed using LimeSurvey, which adheres to the privacy protection of the University of Zurich. There are no risks or benefits of any kind involved with answering this survey. Your individual privacy will be maintained in all published and written data and at any point you can exit the survey without finishing. The collected data will be evaluated and used for visualization choices of a Protégé plugin. You will be asked specifically, if we can quote answers from the second and third part at the end of the survey solely for an academic publication.

To proceed with the survey, you need to consent to the above terms by checking the box below.

With best regards,
Romana Pernischova

There are 25 questions in this survey.

Demographic Information

How old are you?

Your answer must be between 18 and 100
Only an integer value may be entered in this field.
Please write your answer here:

This question is optional, you can leave it empty if you do not want to share your age with us.

How long have you been working with ontologies? (in years)

Only numbers may be entered in this field.
Please write your answer here:

This question is optional, you can leave it empty if you do not want to share this information with us.

Occupation and work area

This question is optional. You can leave the fields empty, if you do not want to share this information with us.

Working with ontologies:

Please choose the appropriate response for each item:

YesUncertainNo
Do you work or have previously worked with ontologies?
Does or has your work include(d) changing ontologies?

Give a specific example of an ontology you work with. Add some details about the domain of the ontology and an example (if possible) what the ontology is used for.

Only answer this question if the following conditions are met:
Answer was 'Yes' or 'Uncertain' at question '4 [D02]' (Working with ontologies: (Do you work or have previously worked with ontologies?))

Please write your answer here:

Please enter a descritpion and be as specific as possible. If possible, provide the name of the ontology (ontologies) you work with.

How often do you change an ontology?

Only answer this question if the following conditions are met:
Answer was 'Yes' at question '4 [D02]' (Working with ontologies: (Does or has your work include(d) changing ontologies?))

Choose one of the following answers
Please choose only one of the following:

  • Every day (on average)
  • Every week (at least once a week on average)
  • Every month (at least once a month on average)
  • A few times a year (at least two times a year)
  • Less than two times a year
Make a comment on your choice here:
If necessary, elaborate in the comment section.

What tools/SDKs do you use for viewing and/or changing ontologies?

Only answer this question if the following conditions are met:
Answer was 'Yes' or 'Uncertain' at question '4 [D02]' (Working with ontologies: (Does or has your work include(d) changing ontologies?))

Please write your answer here:

Use a comma to separate multiple tools

What does the term "ontology change" mean for you? How do you define this term?

Please write your answer here:

Please be as precise as possible in your description.

Any other comments about the questions so far?

Please write your answer here:

Changing an Ontology

What are the changes you apply to an ontology regularly?
Please rank the following options by putting one you apply the most first and so on.

All your answers must be different and you must rank in order.
Please select from 1 to 8 answers.
Please number each box in order of preference from 1 to 8
Please choose at least 1 items.

  • Adding new content, classes, or properties
  • Adding details to already present content
  • Correcting mistakes (typos, wrong values, etc.)
  • Removing content, classes, or properties
  • Restructuring, changing subclass relationships
  • Correcting inconsistencies
  • Defining new restrictions and rules
  • Changing restrictions and rules

Are there any other ways you change ontologies, which are not mentioned above?

Please write your answer here:

Please also add, after which position you would insert it into the list above (0 = first, 1 = after the first item, etc.)

What textual summary or visualization of changes do you already use?
Please, add a comment about tools, or specific measures, where applicable.

Comment only when you choose an answer.
Please choose all that apply and provide a comment:

  • Summary of the applied changes.
  • Changed primitive measures of the ontology.
  • Changed composite measures of the ontology.
  • Some impact/consequences of the applied changes.
  • Other:

Primitive measures describe the ontology with e.g., number of axioms, classes, properties, etc.
Composite measures are made of a ratio of primitive measures, e.g. class property ratio.
Impact/Consequences e.g. consistency of the ontology, retraining of a prediction model, recalculation of an embedding used in a different task, etc.

What kind of textual summary and visualization about ontology changes would you find helpful?

Please choose the appropriate response for each item:

Textual Summary Visualization
very helpful helpful somewhat helpful not helpful don't care/know very helpful helpful somehow helpful not helpful don't care/know
Number and types of changes:
Change in primitive measures:
Change in composite measures:
Some impact/consequences on the applied changes:
Primitive measures describe the ontology with e.g., number of axioms, classes, properties, etc.
Composite measures are made of a ratio of primitive measures, e.g. class property ratio.
Impact/Consequences e.g. consistency of the ontology, retraining of a prediction model, recalculation of an embedding used in a different task, etc.

Please specify why you think any of the options above would not be helpful:

Only answer this question if the following conditions are met:
-------- Scenario 1 --------
Answer was 'not helpful' or 'don't care/know' at question '13 [E02Y]' (What kind of textual summary and visualization about ontology changes would you find helpful? (Some impact/consequences on the applied changes: Label Visualization))
-------- or Scenario 2 --------
Answer was 'not helpful' or 'don't care/know' at question '13 [E02Y]' (What kind of textual summary and visualization about ontology changes would you find helpful? (Number and types of changes: Label Textual Summary))
-------- or Scenario 3 --------
Answer was 'don't care/know' or 'not helpful' at question '13 [E02Y]' (What kind of textual summary and visualization about ontology changes would you find helpful? (Change in primitive measures: Label Visualization))
-------- or Scenario 4 --------
Answer was 'not helpful' or 'don't care/know' at question '13 [E02Y]' (What kind of textual summary and visualization about ontology changes would you find helpful? (Change in primitive measures: Label Textual Summary))
-------- or Scenario 5 --------
Answer was 'don't care/know' or 'not helpful' at question '13 [E02Y]' (What kind of textual summary and visualization about ontology changes would you find helpful? (Change in composite measures: Label Visualization))
-------- or Scenario 6 --------
Answer was 'not helpful' or 'don't care/know' at question '13 [E02Y]' (What kind of textual summary and visualization about ontology changes would you find helpful? (Change in composite measures: Label Textual Summary))
-------- or Scenario 7 --------
Answer was 'not helpful' or 'don't care/know' at question '13 [E02Y]' (What kind of textual summary and visualization about ontology changes would you find helpful? (Number and types of changes: Label Visualization))
-------- or Scenario 8 --------
Answer was 'not helpful' or 'don't care/know' at question '13 [E02Y]' (What kind of textual summary and visualization about ontology changes would you find helpful? (Some impact/consequences on the applied changes: Label Textual Summary))

Please write your answer here:

Any other comments about the questions so far?

Please write your answer here:

Mockups of a Prototype

In this section, we show you a few possible visualizations and textual summary of changes, ontology, and impact of changes.

How informative do you find the following texual summaries and visualization?
Rate each possibility from 1 (not informative) to 5 (very informative). The colors have no meaning.

 

Please choose the appropriate response for each item:

(not at all)
1
2
3
4		 (very)
5

A list of the most recently changed axioms.

 

A table view showing the change in value.

A graph visualizing one or multiple measures
(examples, will be addressed further in the next question).

    

    

One impact measure signaling the consequences of the changes with a detailed explentation of its meaning and calculation.

An indicator of ontology consistency.

How informative do you find the following visualizations showing one or multiple ontology measures?
Rate each possibility from 1 (not informative) to 5 (highly informative).

Please choose the appropriate response for each item:

(not at all)
1
2
3
4		 (very)
5

Line chart showing a selected measure

Bar chart showing one or more measures
(if they are comparable)

 

Area chart showing one or multiple measures
(if measures are additive)

Spider chart showing multiple measures which have similar scale

Can you think of any other textual summaries or visualizations to show ontology changes or change impact/consequences?

Please write your answer here:

Choose your preferred setup of textual summaries and/or visualizations, which would suit your needs during changing of an ontology.
(You do not have to use all elements in your ranking. The colors have no meaning.)

All your answers must be different and you must rank in order.
Please select at most 5 answers
Please number each box in order of preference from 1 to 5

  • Consistency display.

       

  • Impact measure display (with choice of measure).

      

  • Most recently changed axioms.  

  • Table with primitive/composite measures (with choice).

  • Graphical visualization of changed primitiv/composite measures (area chart was chosen randomly). 

       

Any other comments about the questions so far?

Please write your answer here:

Wrap-up

Interest in a Protégé plugin for visualization of changes and change consequences:

Please choose the appropriate response for each item:

Yes Unsure No
Would you be interested in seeing changes and its consequences visualized while editing an ontology?
Would you be more aware of changes and its consequences, if such numbers were at your disposal?
Do you find it necessary to alert the ontology engineer about impact of changes? 

Feedback/Comments about the topic in general:

Please write your answer here:

Can we quote your answers (anonymously) to the questions about visualizations and the mockups directly in our publication? *

Please choose only one of the following:

  • Yes
  • No

Would you be interested in participating in our follow up user study about this topic? *

Please choose only one of the following:

  • Yes
  • No
The study will involve solving simple ontology editing tasks and we will analyze your editing behaviour while using the implementation of a change visualization plugin.

Please provide an email address, if you would be interested in participating in the follow up study:

Only answer this question if the following conditions are met:
Answer was 'Yes' at question '24 [W06]' (Would you be interested in participating in our follow up user study about this topic?)

Please write your answer here:

We will only use this email address to contact you about the follow up study.

Thank you very much for completing this survey. Your answers will be considered and hopefully reported in a publication soon. With the provided email address we will contact you for a request of participation for a follow up user study as well as with the publication resulting from this study.

Thank you for your time and expertise!


03.10.2020 – 12:54

Submit your survey.
Thank you for completing this survey.