In this blogpost we are presenting how Jupyter Notebook can be used to easily present some data analytics results. This can be useful when some relatively easy data analysis needs to be done on a relatively small dataset. A Jupyter Notebook can be interactive, and if slight modifications are needed they can be easily made real time.
We are going to drive you through a step-by-step analysis of an insurance dataset. The dataset can be found here. If you want to try out the following tutorial yourself, you need to download the dataset (insurance.csv). It contains the following information:
In order to perform the analysis, first you need to install Python (preferably python3) alongside some necessary packages like pandas, numpy, and matplotlib. You also need to install Jupyter Notebook. In case you are a Windows user we suggest following this step-by-step instruction for installation.
Once everything is installed, it's time to perform the analysis. We will cover the following steps:
The output of the above command should look like this:
The output should look something like this:
We will call this function by replacing x with the corresponding numerical column names: age, bmi, and charges. The distribution of numerical attributes is the following:
We will again replace x with the corresponding categorical column names: sex, smoker, region and children. The distribution of categorical attributes is the following:
Correlation analysis can be done only among numerical attributes, so firstly, we transform the categorical attributes using a simple mapping. We map each categorical value of each attribute to a number and just replace the values with the corresponding number. This can be done by the following piece of code:
Then we create the correlation heatmap with the following code:
The result is the following:
As you can see in the graphs above, the charges of people who smoke indeed differ from the people who don't.
As you can see on the graph, the BMI of males does not differ from that of females in our data.
As you can see, with some very simple steps, you can easily gather meaningful information from a dataset. Using simple python commands you can do similar analyses on your own dataset.
If you would like to receive our Jupyter Notebook with some extended analysis, you can subscribe to our newsletter here.
We are going to drive you through a step-by-step analysis of an insurance dataset. The dataset can be found here. If you want to try out the following tutorial yourself, you need to download the dataset (insurance.csv). It contains the following information:
- age: Age of primary beneficiary
- sex: Insurance contractor gender, female / male
- bmi: Body mass index, providing an understanding of the body, weights that are relatively high or low relative to height, objective index of body weight (kg / m2) using the ratio of height to weight, ideally 18.5 to 24.9
- children: Number of children covered by health insurance / Number of dependents
- smoker: Smoker / Non - smoker
- region: The beneficiary's residential area in the US, northeast, southeast, southwest, northwest.
- charges: Individual medical costs billed by health insurance
In order to perform the analysis, first you need to install Python (preferably python3) alongside some necessary packages like pandas, numpy, and matplotlib. You also need to install Jupyter Notebook. In case you are a Windows user we suggest following this step-by-step instruction for installation.
Once everything is installed, it's time to perform the analysis. We will cover the following steps:
- Loading the dataset
- Data exploration (datatypes, missing data, distributions, etc)
- Correlation analysis between the attributes
- Hypothesis testing (based on previous findings)
- Conclusions
1. Loading the dataset
Before starting the process we need to make sure that the necessary libraries are imported. To import a library in python you need to run the import <library name>. We are going to need the following libraries:- Pandas: used for data analysis and data manipulation.
- Matplotlib: used for creating static, animated, and interactive visualizations.
- NumPy: offer mathematical functions.
- Seaborn: is a data visualization library based on Matplotlib.
import pandas as pd
import matplotlib.pyplot as plt
import numpy as np
import seaborn as sns
df = pd.read_csv("insurance.csv")
df.head(10)
The output of the above command should look like this:
2. Data exploration
Once the dataset is loaded, we can do some exploratory data analysis to better understand the data that we are working with.2.1 Datatypes
As a first step, we check the number of rows in the dataset and the datatype of the columns. You can check this by running the following command:
df.info()
The output should look something like this:
2.2 Distribution of numerical attibutes
For visualizing the distribution of the numerical attributes we will use the following function:
def distribution(x):
sns.histplot(df[x])
plt.title(f'{x.capitalize()} Density Distribution')
We will call this function by replacing x with the corresponding numerical column names: age, bmi, and charges. The distribution of numerical attributes is the following:
2.3 Distribution of categorical attibutes
For visualizing the distribution of the categorical attributes is a bit simpler, because we just simply count the occurrences of the values of the attributes. We will use this line of code:
sns.countplot(x = x, data = df)
We will again replace x with the corresponding categorical column names: sex, smoker, region and children. The distribution of categorical attributes is the following:
3. Correlation analysis between the attributes
Correlation is a statistical measure that indicates the extent to which two or more variables fluctuate in relation to each other. A positive correlation indicates the extent to which those variables increase or decrease in parallel; a negative correlation indicates the extent to which one variable increases as the other decreases.Correlation analysis can be done only among numerical attributes, so firstly, we transform the categorical attributes using a simple mapping. We map each categorical value of each attribute to a number and just replace the values with the corresponding number. This can be done by the following piece of code:
for x in ['sex', 'children', 'smoker', 'region']:
df[x] = df[x].astype('category')
cat_columns = df.select_dtypes(['category']).columns
df[cat_columns] = df[cat_columns].apply(lambda x: x.cat.codes)
Then we create the correlation heatmap with the following code:
correlation = df.corr()
sns.heatmap(correlation, annot=True, cmap = 'summer_r')
The result is the following:
4. Hypothesis testing
A statistical hypothesis test is a method of statistical inference used to decide whether the data at hand sufficiently support a particular hypothesis. You can read more about it here. We are going to visually investigate the next 2 hypotheses:- Do the charges of people who smoke differ significantly from the people who don't?
- Does the BMI of males differ significantly from that of females?
4.1 Do the charges of people who smoke differ significantly from the people who don't?
As you can see in the graphs above, the charges of people who smoke indeed differ from the people who don't.
4.2 Does the BMI of males differ significantly from that of females?
As you can see on the graph, the BMI of males does not differ from that of females in our data.
5. Conclusions
The conclusions gathered from the dataset during our analysis are the followings:- The charges of people who smoke indeed differ significantly from the people who don't.
- The BMI of males does not differ significantly from that of females in our data.
As you can see, with some very simple steps, you can easily gather meaningful information from a dataset. Using simple python commands you can do similar analyses on your own dataset.
If you would like to receive our Jupyter Notebook with some extended analysis, you can subscribe to our newsletter here.