In this project, I extend my CNN Advanced Architecture project to 3D images. The architecture is advanced in the sense that it is flexible and can read & build its architecture from JSON files. The 3D extension allows CNN3D classifier to tackle 3D tensors used heavily in scientific/medical imaging such as CT Scan images, which we will we working with in this repo. I have trained the LUNA16 dataset with the usual my CNN and then with CNN3D classes and noted that CNN3D outperfromed the predictions for the LUNA16 dataset.
LUNA16
LUng Nodule Analysis data includes 888 CT Scans of Patient's Lungs with Annotations around the possible cancer region. The annotations are provided by multiple radiologists and the detections are in binary +ve and -ve classess with specified region of interest, thickness, etc. in the metadata files. The information about the dataset can be found here.
Data Exploration
Out of total of 551065 annotations, only 1351 were labeled as nodules. We downsample the negative examples heavily to reach -ve to +ve ratio of 0.2. We will then upsample the +ve examples by rotating the images. Hence, we first balance classes to improve on the accuracy of binary classficiation since the naive accuracy will lead to high FNR. More important, the Recall rate is crucial in the classification of serious illness of cancer for a false negative will be fatal. Hence we will be keeping eye on the Recall rates throughout.
The images are in .raw formats with metadata given in the .mhd files. Pythons has SimpleITK library that opens the raw files easily. Kudos to this repo for providing the baseline code. Once the file is read, the image can be seen as having shape:
"""
Read mhd data
"""
filename = "subset6/1.3.6.1.4.1.14519.5.2.1.6279.6001.669435869708883155232318480131.mhd"
path = glob.glob(filename)
metadata = sitk.ReadImage(path[0])
image = sitk.GetArrayFromImage(metadata)
image.shape(130, 512, 512)
There are 130 slices of 512 sq pixel CTScan Image on the file above. The number of slices are different for each file which is an issue for CNN3D. We will need to resample into uniform slices. Each slice can be seen as:
plt.imshow(image[0],cmap='gray')
The first 12 slices are observed:
## the 3d image slices
fig = plt.figure(dpi=100)
for num,each_slice in enumerate(low_res[:12]):
y = fig.add_subplot(3,4,num+1)
y.imshow(each_slice,cmap='gray')
plt.show()
We can use matplotlib to plot all the 130 slices into 3d scatterplot:
Since the annotations are provided, we wont need the entire CTScan region. We compute the ROI,crop the images and output the final numpy array in CTSData.py. The output can be observed:
##load 3d data
X_test_3d = np.load("./test_imdata.npy") Here the unformly sampled (eps=10) 64x64 CTScan images is shown
## the 3d image slices of annotated ROI (shape 64x64x10)
X_test_candidate = X_test_3d[0,:,:]
fig = plt.figure(dpi=100)
for num,each_slice in enumerate(X_test_candidate):
y = fig.add_subplot(3,4,num+1)
out = each_slice.reshape(64,64)
y.imshow(out,cmap='gray')
plt.tight_layout()
plt.show()
Below shows 2 +ve samples of Nodules in depth frames. Note that the Nodule appears & disappreas with the depth:
#data augmentation for training
a = X_test_candidate.reshape(-1,64,64)
fig = plt.figure(dpi=100)
y = fig.add_subplot(1,4,1)
y.imshow(a[0],cmap='gray')
for i in range(3):
a = np.rot90(a,1,(1,2))
y = fig.add_subplot(1,4,i+2)
y.imshow(a[0],cmap='gray')
plt.show()
Classification Results:
CNN
archsm = [
{'type':'C','activation':'relu','num_output':16,'kernel_size': (3,3), 'stride': (1,1), 'drop_out': 8.3},
{'type':'P','pool':'max','kernel_size': (2,2), 'stride': (2,2), 'drop_out': 16.7},
{'type':'C','activation':'relu','num_output':32,'kernel_size': (3,3), 'stride': (1,1), 'drop_out': 0.0},
{'type':'C','activation':'relu','num_output':16,'kernel_size': (5,5), 'stride': (1,1), 'drop_out':16.7},
{'type':'P','pool':'max','kernel_size': (5,5), 'stride': (3,3), 'drop_out': 0.0},
{'type':'C','activation':'relu','num_output':32,'kernel_size': (3,3), 'stride': (1,1), 'drop_out': 0.0},
{'type':'FC','activation':'relu','num_output':20,'drop_out': 8.3},
{'type':'FC','activation':'relu','num_output':10,'drop_out': 0.0},
{'type':'T','activation':None}
]
For the architecture above, the usual 2D CNN achieved the test accuracy of 90.6% (error 0.094). The recall, which we are the most interested in maximizing, was 78% for the positive class.
final training error: 0.135
test error: 0.09450277949351452
Confusion Matrix:
[[1248 90]
[ 63 218]]
Classification Report:
precision recall f1-score support
0 0.95 0.93 0.94 1338
1 0.71 0.78 0.74 281
accuracy 0.91 1619
macro avg 0.83 0.85 0.84 1619
weighted avg 0.91 0.91 0.91 1619
Using larger feature sizes in the CNN kernel (see below) the 2D CNN achieved improved its test accuracy to 92.6% (error 0.074). The recall improved 85% for the positive class.
arch = [
{'type':'C','activation':'relu','num_output':64,'kernel_size': (3,3), 'stride': (1,1), 'drop_out': 8.3},
{'type':'P','pool':'max','kernel_size': (2,2), 'stride': (2,2), 'drop_out': 16.7},
{'type':'C','activation':'relu','num_output':128,'kernel_size': (3,3), 'stride': (1,1), 'drop_out': 0.0},
{'type':'C','activation':'relu','num_output':64,'kernel_size': (5,5), 'stride': (1,1), 'drop_out':16.7},
{'type':'P','pool':'max','kernel_size': (5,5), 'stride': (3,3), 'drop_out': 0.0},
{'type':'C','activation':'relu','num_output':128,'kernel_size': (3,3), 'stride': (1,1), 'drop_out': 0.0},
{'type':'FC','activation':'relu','num_output':20,'drop_out': 8.3},
{'type':'FC','activation':'relu','num_output':10,'drop_out': 0.0},
{'type':'T','activation':None}
]
final training error: 0.074
test error: 0.07782581840642372
Confusion Matrix:
[[1255 83]
[ 43 238]]
Classification Report:
precision recall f1-score support
0 0.97 0.94 0.95 1338
1 0.74 0.85 0.79 281
accuracy 0.92 1619
macro avg 0.85 0.89 0.87 1619
weighted avg 0.93 0.92 0.92 1619
CNN3D
The CNN3D class is implemented in cnn3d.py file. The main challenge working with the 3D kernel was to resample the depth of the CTScan Images to a uniform number. This is done in the CTSData.py file where the number eps is used to sample constant images in the z axis in the annotated region.
We extend the archsm architecture above to 3D Conv & Pool kernels. Note that archsm and arch3dsm have exact architecture except for convpooling kernels & strides in 3d.
arch3dsm = [
{'type':'C','activation':'relu','num_output':16,'kernel_size': (3,3,3), 'stride': (1,1,1), 'drop_out': 8.3},
{'type':'P','pool':'max','kernel_size': (2,2,2), 'stride': (2,2,2), 'drop_out': 16.7},
{'type':'C','activation':'relu','num_output':32,'kernel_size': (3,3,3), 'stride': (1,1,1), 'drop_out': 0.0},
{'type':'C','activation':'relu','num_output':16,'kernel_size': (5,5,1), 'stride': (1,1,1), 'drop_out':16.7},
{'type':'P','pool':'max','kernel_size': (5,5,2), 'stride': (3,3,3), 'drop_out': 0.0},
{'type':'C','activation':'relu','num_output':32,'kernel_size': (3,3,3), 'stride': (1,1,1), 'drop_out': 0.0},
{'type':'FC','activation':'relu','num_output':20,'drop_out': 8.3},
{'type':'FC','activation':'relu','num_output':10,'drop_out': 0.0},
{'type':'T','activation':None}
]
The arch3dsm improved the archsm test accuracy to 91.1% (error 0.089) while also improving the Recall to 85%. We see that for same CNN architecture, the 3D convpools minimized the FNR. The precison remaned the same.
final training error: 0.02666666666666667
test error: 0.08956145768993205
Confusion Matrix:
[[1234 104]
[ 41 240]]
Classification Report:
precision recall f1-score support
0 0.97 0.92 0.94 1338
1 0.70 0.85 0.77 281
accuracy 0.91 1619
macro avg 0.83 0.89 0.86 1619
weighted avg 0.92 0.91 0.91 1619
Unfortunately, we run into parameters explosion with CNN3D due to the added dimension. The 3D counterpart of Architecture arch above exhausted the memory resources of my Alienware R313 laptop. However, improvement in the Accuracy and Recall for arch3d is expected as well for the classifier can work with the data in additonal dimension to improve upon.
