We’re excited to bring Transform 2022 back to life on July 19th and virtually July 20-28. Join AI and data leaders for sensible conversations and exciting networking opportunities. Register today!
The rapid technological advancement and widespread adoption of AI-based products and workflows is affecting many aspects of human and business activities in the banking, healthcare, advertising and many more industries. Although the accuracy of AI models is undoubtedly the most important factor to consider when using AI-based products, there is an urgent need to understand how AI can be designed to function responsibly.
Responsive AI is a framework that any software development organization needs to adopt to enhance customer confidence in the transparency, accountability, fairness and security of any deployed AI solutions. At the same time, the key to making AI accountable is that there should be a development pipeline that can promote the reproducibility of results and manage the descent of data and ML models.
Low-code machine learning is gaining popularity with tools such as PyCaret, H2O.ai and DataRobot, which allow data scientists to run pre-candid patterns for feature engineering, data cleansing, model development and statistical performance comparison. However, often the missing pieces of these packages are patterns around responsive AI that evaluates the ML model for fairness, transparency, persuasiveness, functionality and more.
Here, we demonstrate a quick and easy way to integrate PyCaret with the Microsoft RAI (responsive AI) framework that generates detailed reports showing error analysis, explanatory ability, causality, and adversity. The first part is a code walkthrough for developers showing how to create a RAI dashboard. The second part is a detailed evaluation of the RAI report.
First, we install the necessary libraries. This can be done on your local machine with Python 3.6+ or on SaaS platforms like Google Colab.
!pip install raiwidgets
!pip install pycaret
!pip install — upgrade pandas
!pip install — upgrade numpy
Panda and Nampi upgrades are needed right now but will be fixed soon. Also, don’t forget to restart runtime if you are installing in Google Colab.
Next, we load and clear data from GitHub and do feature engineering with PyCaret.
import pandas as pd, numpy as np
import matplotlib.pyplot as plt
csv_url = ‘https://raw.githubusercontent.com/sahutkarsh/loan-prediction-analytics-vidhya/master/train.csv'
dataset_v1 = pd.read_csv (csv_url)
dataset_v1 = dataset_v1.dropna()
from pycaret.classification import *
clf_setup = setup(data = dataset_v1, target = ‘Loan_Status’,
train_size=0.8, categorical_features=[‘Gender’, ‘Married’, ‘Education’,
imputation_type=’simple’, categorical_imputation = ‘mode’, ignore_features=[‘Loan_ID’], fix_imbalance=True, silent=True, session_id=123)
Dataset is a simulated loan application dataset with features like gender, marital status, employment, income etc. Of applicants. PyCaret has a great feature for making training and test data frames available through the get _config method after feature engineering. We use this to get the clean features that we will later feed into the RAI widget.
X_train = get_config(variable=”X_train”).reset_index().drop([‘index’], axis=1)
y_train = get_config(variable=”y_train”).reset_index().drop([‘index’], axis=1)[‘Loan_Status’]
X_test = get_config(variable=”X_test”).reset_index().drop([‘index’], axis=1)
y_test = get_config(variable=”y_test”).reset_index().drop([‘index’], axis=1)[‘Loan_Status’]
df_train = X_train.copy()
df_train[‘LABEL’] = y_train
df_test = X_test.copy()
df_test[‘LABEL’] = y_test
We now run PyCaret to create multiple models and compare them on recall as a statistical performance metric.
top5_results = compare_models(n_select=5, sort="Recall")
Our top model is a random forest classifier with a recall of 0.9, which we can plot here.
selected_model = top5_results plot_model(selected_model)
Now, we will write 10 lines of code to create a RAI dashboard using featured data frames and models generated from PyCaret.
cat_cols = [‘Gender_Male’, ‘Married_Yes’, ‘Dependents_0’, ‘Dependents_1’, ‘Dependents_2’, ‘Dependents_3+’, ‘Education_Not Graduate’, ‘Self_Employed_Yes’, ‘Credit_History_1.0’, ‘Property_Area_Rural’, ‘Property_Area_Semiurban’, ‘Property_Area_Urban’]
from raiwidgets import ResponsibleAIDashboard from responsibleai import RAIInsights rai_insights = RAIInsights(selected_model, df_train, df_test, ‘LABEL’, ‘classification’, categorical_features=cat_cols)
The above code, though very minimal, does a lot of things under the hood. It builds insights on RAI for classification and adds modules for clarity and error analysis. Then, a causal analysis is performed based on two characteristics of the treatment, including credit history and marital status. Also, a counterfactual analysis is performed for 10 scenarios. Now, let’s generate a dashboard.
The above code will start the dashboard on a port like 5000. On a local machine, you can go directly to http: // localhost: 5000 and see the dashboard. On Google Colab, you need to do a simple trick to see this dashboard.
from google.colab.output import eval_js
This will give you the URL to view the RAI dashboard. You can see some components of the RAI dashboard below. Here are some of the main results of this analysis that were automatically generated to complement the AutoML analysis performed by Pykerat.
Results: Responsive AI report
Error analysis: We find that the error rate for rural property areas is high and our model has a negative bias for this feature.
The ability to explain the world – the importance of specialty: We see that the importance of specialty remains the same in both groups – all data (blue) and property area rural (orange). We look for the orange mass, the property area has a big impact but nonetheless, credit history is the # 1 factor.
Local clarification: We see that credit history is also an important feature for personal speculation – Row # 20.
Counterfactual analysis: We see that a decision from N to Y (based on data) may be possible for the same row # 20 if the credit history and loan amount are changed.
Causal guess: We consider causal analysis to study the effect of two treatments, credit history and employment status and see if credit history has a more direct effect on approval.
A responsive AI analysis report featuring model error analysis, explanatory ability, causal inference, and counterfeitualities can add great value to the traditional statistical matrix of accuracy-memory that we typically use as a lever to evaluate a model. With advanced tools like PyCaret and RAI dashboards, creating these reports is easy. These reports can be developed using other tools – the main thing is that data scientists need to evaluate the models for the AI responsible for this pattern to ensure that their models are ethical with accuracy.
Dattaraj Rao is the main data scientist of Persistent,
Welcome to the VentureBeat community!
DataDecisionMakers is where experts, including tech people working on data, can share data-related insights and innovations.
If you would like to read about the latest ideas and latest information, best practices and the future of data and data tech, join us at DataDecisionMakers.
You might even consider contributing to your own article!
Read more from DataDecisionMakers