Day 2 Discussion
On day 2, we learned more about explainability versus interpretability and about deep learning XAI Methods.
Please have one of your team members reply in the comments to to each of these questions after discussing them with the team.
Here are the Space Weather Jupyter notebooks:
- More beginner oriented: https://github.com/ai2es/tai4es-trustathon-2022/blob/main/space/magnet_lstm_tutorial.ipynb
- More advanced user oriented: https://github.com/ai2es/tai4es-trustathon-2022/blob/main/space/magnet_cnn_tutorial.ipynb
The TAI4ES Space GitHub Readme page is here.
Discussion prompts
- Today we introduced the topics of AI explainability and interpretability. Building on what you learned in both sessions today, what do you think the goals of explainability and interpretability are (or should be) when working with your end user? In other words, why do (or should) these terms matter for end users?
- How can you use the techniques covered today (either in the lecture or in your trust-a-thon activity) to better meet the needs of your end users?
Answer 1:
We trained and evaluated two deep learning algorithms: a convolutional neural network (CNN) and a long short-term memory (LSTM).
The applied CNN model does not retain memory of previous time series patterns because it is not recurrent like an LSTM model. Does it speak for the LSTM model? Arguing about metric values requires technical expertise needed to make decisions in terms of interpretability. Considering such parameters as mean solar wind speed and standard deviation of wind speed, we note that such metrics as permuted RMSE show in general lower values for CNN comparing to those for LSTM. Does it enable us to prefer the CNN model in terms of interpretability? Speed mean is more important feature than standard deviation of speed. And these parameters have anomalous values during active space weather that shows how further studies are required to help the end users to make decisions in terms of explainability.
Answer 2:
We answer the following question to meet the needs of the end users better. If one or more solar wind instruments were to degrade on orbit how do you predict that might impact model performance? We consider permutation importance showing that mean solar wind speed is the second one for LSTM model making it to be a very important feature. Bearing this and the posed question in mind, degrading should worsen model performances. (edited)
1. Today we introduced the topics of AI explainability and interpretability. Building on what you learned in both sessions today, what do you think the goals of explainability and interpretability are (or should be) when working with your end user? In other words, why do (or should) these terms matter for end users?
• The topic of the explainability should cover the needs of the end users
• To help the end users to make decisions, the results should be explainable and interpretable in a clear way, especially when the end users have less technical expertise.
• To be specific, our end users are the precision navigators, and the needs are 1-hour and 24-hour prior predictions. However, the given model can only make predictions for 1 hour in advance. Additional modeling should be conducted for 24-hour advanced predictions.
2. How can you use the techniques covered today (either in the lecture or in your trust-a-thon activity) to better meet the needs of your end users?
• An additional column representing a 24 hour interval can be used for the 24-hour advanced predictions. The model allows decision with cadence steps t0, t1. We can extend that to t23.
• In general, the XAI could be used to provide a better explainable result to the end users, in terms of the model workflow and decision making.
• Providing the level of uncertainties of the predictions can be useful to the end users when making decisions.
This is fantastic work – great job making connections between the lectures, the coding activities, and the personas!