![](\"https://img.freepik.com/premium-vector/breast-cancer-young-cartoon-woman-with-ribbon-butterfly-hearts-illustration_24640-70479.jpg?w=2000\")
\nBreast cancer is a type of cancer that affects the cells of the breast. It occurs when abnormal cells in the breast divide and grow in an uncontrolled way. These abnormal cells can form a tumor that can be seen on an x-ray or felt as a lump. Breast cancer is most commonly diagnosed in women, but men can also develop the disease. Treatment for breast cancer often involves surgery, chemotherapy, and radiation.
\nMammography is a type of medical imaging that uses low-energy X-rays to create detailed images of the breasts. It is typically used to detect and diagnose breast cancer, as well as to monitor the progress of treatment for breast cancer. The procedure involves the use of a specialized X-ray machine, which produces images that can be examined by a doctor to look for abnormalities in the breast tissue. Mammography is generally recommended for women over the age of 40, or for women with a family history of breast cancer, as a way to detect the disease in its early stages, when it is most treatable.
\n\n\n
","metadata":{}},{"cell_type":"markdown","source":"##
In the description of the dataset it is mentioed that site id is nothing but id of the hospitals where mammography is done and images are collected.
","metadata":{}},{"cell_type":"markdown","source":"### Total Number of Hospitals","metadata":{}},{"cell_type":"code","source":"print(\"Total number of hospitals from where images are collected: \",df['site_id'].nunique())","metadata":{"execution":{"iopub.status.busy":"2022-12-16T06:39:25.635102Z","iopub.execute_input":"2022-12-16T06:39:25.635481Z","iopub.status.idle":"2022-12-16T06:39:25.644688Z","shell.execute_reply.started":"2022-12-16T06:39:25.635449Z","shell.execute_reply":"2022-12-16T06:39:25.64376Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Number of images collected from each Hospital","metadata":{}},{"cell_type":"code","source":"temp = df.copy()\ntemp = temp['site_id'].value_counts().to_frame()\ntemp.reset_index(inplace=True)\ntemp.columns = ['site_id', 'image_counts']\ncolors = sns.color_palette('dark')\nplt.pie(temp['image_counts'].to_list(), labels=temp['site_id'].to_list(), autopct='%.0f%%', shadow=True, radius=2.5)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-12-16T06:39:25.6723Z","iopub.execute_input":"2022-12-16T06:39:25.672716Z","iopub.status.idle":"2022-12-16T06:39:25.810001Z","shell.execute_reply.started":"2022-12-16T06:39:25.672682Z","shell.execute_reply":"2022-12-16T06:39:25.808904Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"From the plot it can be seen that Hospital 1 contributed silghtly more images than Hospital 2.
","metadata":{}},{"cell_type":"markdown","source":"##Breast cancer is a type of cancer that affects the cells in the breast tissue. It is a common form of cancer, and it can affect both men and women. Symptoms of breast cancer may include a lump in the breast, changes in the shape or size of the breast, and changes in the appearance of the nipple. Treatment for breast cancer can vary, depending on the stage and type of cancer, but may include surgery, radiation therapy, chemotherapy, and targeted therapy. It is important for anyone who is experiencing symptoms of breast cancer to talk to their doctor for a proper diagnosis and treatment plan.
","metadata":{}},{"cell_type":"markdown","source":"### Total number of Patients","metadata":{}},{"cell_type":"code","source":"# Total number of patients\nprint(\"Total number of individual patients observed in the dataset: \",df['patient_id'].nunique())","metadata":{"execution":{"iopub.status.busy":"2022-12-16T06:39:25.814214Z","iopub.execute_input":"2022-12-16T06:39:25.815739Z","iopub.status.idle":"2022-12-16T06:39:25.823729Z","shell.execute_reply.started":"2022-12-16T06:39:25.815678Z","shell.execute_reply":"2022-12-16T06:39:25.822501Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"So, it's clear that some patients more than one pictures in the dataset.
","metadata":{}},{"cell_type":"markdown","source":"### Minimum / maximum / average number of entry of an individual patient ","metadata":{}},{"cell_type":"code","source":"print(\"Top 5 patients who contributed maximum number of datapoints contributed in the dataset: \\n\", \n df['patient_id'].value_counts().head())","metadata":{"execution":{"iopub.status.busy":"2022-12-16T06:39:25.82586Z","iopub.execute_input":"2022-12-16T06:39:25.826328Z","iopub.status.idle":"2022-12-16T06:39:25.838667Z","shell.execute_reply.started":"2022-12-16T06:39:25.826284Z","shell.execute_reply":"2022-12-16T06:39:25.836909Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(\"Top 5 patients who contributed minimum number of datapoints in the dataset: \\n\",\ndf['patient_id'].value_counts().tail())","metadata":{"execution":{"iopub.status.busy":"2022-12-16T06:39:25.84285Z","iopub.execute_input":"2022-12-16T06:39:25.843795Z","iopub.status.idle":"2022-12-16T06:39:25.858763Z","shell.execute_reply.started":"2022-12-16T06:39:25.843676Z","shell.execute_reply":"2022-12-16T06:39:25.857198Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(\"Average number of images contibuted by each individual patient in the dataset: \",\n df['patient_id'].value_counts().median())","metadata":{"execution":{"iopub.status.busy":"2022-12-16T06:39:25.860722Z","iopub.execute_input":"2022-12-16T06:39:25.861518Z","iopub.status.idle":"2022-12-16T06:39:25.873503Z","shell.execute_reply.started":"2022-12-16T06:39:25.861467Z","shell.execute_reply":"2022-12-16T06:39:25.872152Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"From the above data it is difficult to come to any conclusion that why these pepole are tested more times than others. Age and Density of breast can be the reason but there is no proper evidence.
","metadata":{}},{"cell_type":"markdown","source":"### How many patients contributed how many images?","metadata":{}},{"cell_type":"code","source":"temp = df['patient_id'].value_counts().to_frame()\ntemp.reset_index(inplace=True)\ntemp.columns = ['patient_id', 'Image-Frequency']\ntemp = temp.groupby('Image-Frequency')['patient_id'].count().to_dict()","metadata":{"execution":{"iopub.status.busy":"2022-12-16T06:39:26.491857Z","iopub.execute_input":"2022-12-16T06:39:26.492597Z","iopub.status.idle":"2022-12-16T06:39:26.5049Z","shell.execute_reply.started":"2022-12-16T06:39:26.492547Z","shell.execute_reply":"2022-12-16T06:39:26.503419Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fig, ax = plt.subplots(figsize=(20,10))\nax = sns.barplot(x=list(temp.keys()), y=list(temp.values()), label=\"Patients Frequency vs Images Frequency\")\nax.bar_label(ax.containers[0])\nplt.xlabel('Frequency of Images')\nplt.ylabel('Frequency of Patients')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-12-16T06:39:26.507167Z","iopub.execute_input":"2022-12-16T06:39:26.507624Z","iopub.status.idle":"2022-12-16T06:39:26.850222Z","shell.execute_reply.started":"2022-12-16T06:39:26.507577Z","shell.execute_reply":"2022-12-16T06:39:26.84892Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"From the plot it is clear that most of the patients have 4, 5 or 6 images in the dataset.
","metadata":{}},{"cell_type":"markdown","source":"### How many patients are from Hospital 1 and How many patients are from Hospital 2?","metadata":{}},{"cell_type":"code","source":"temp = df.copy()\ntemp = temp[['site_id', 'patient_id']]\ntemp = temp.groupby('site_id')['patient_id'].nunique().to_frame()\ntemp.reset_index(inplace=True)\ncolors = sns.color_palette('dark')\nplt.pie(temp['patient_id'].to_list(), labels=temp['site_id'].to_list(), autopct='%.0f%%', shadow=True, radius=2.5)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-12-16T06:39:26.85319Z","iopub.execute_input":"2022-12-16T06:39:26.853538Z","iopub.status.idle":"2022-12-16T06:39:27.011905Z","shell.execute_reply.started":"2022-12-16T06:39:26.853506Z","shell.execute_reply":"2022-12-16T06:39:27.010142Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Although first hosiptal contributed more data points than second hospital, second hospital has examined more number of patients than first hosiptal.
","metadata":{}},{"cell_type":"markdown","source":"##Mammography is the most common method used to detect breast cancer. This type of medical imaging uses low-energy X-rays to create detailed images of the breast tissue. These images can be examined by a doctor to look for abnormalities, such as lumps or changes in the breast tissue, that may indicate the presence of cancer. In addition to mammography, other imaging tests that may be used to detect breast cancer include ultrasound, magnetic resonance imaging (MRI), and positron emission mammography (PEM). These tests may be used in combination with mammography to get a more detailed picture of the breast tissue and to help confirm a diagnosis of breast cancer.
","metadata":{}},{"cell_type":"markdown","source":"### Count of unique image id","metadata":{}},{"cell_type":"code","source":"print(\"Total number of individial image_id: \",df['image_id'].nunique())","metadata":{"execution":{"iopub.status.busy":"2022-12-16T06:39:27.014641Z","iopub.execute_input":"2022-12-16T06:39:27.015297Z","iopub.status.idle":"2022-12-16T06:39:27.02988Z","shell.execute_reply.started":"2022-12-16T06:39:27.015236Z","shell.execute_reply":"2022-12-16T06:39:27.028351Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"All the image id is unique. It can be considered as primary key.
","metadata":{}},{"cell_type":"markdown","source":"##Breast laterality refers to the side of the body on which a person's breast is located. In other words, it refers to whether a person has a left breast, a right breast, or both. Breast cancer can affect either the left breast, the right breast, or both breasts. It is not uncommon for breast cancer to affect both breasts, even if only one breast has a visible lump or other symptoms. The side on which a person's breast cancer develops does not typically have any bearing on the severity or prognosis of the cancer. Treatment for breast cancer is typically the same regardless of the laterality of the cancer.
","metadata":{}},{"cell_type":"markdown","source":"### Realtionship between laterality and frequency of images","metadata":{}},{"cell_type":"code","source":"temp = df.groupby('laterality')['image_id'].count().to_frame()\ntemp.reset_index(inplace=True)\nfig, ax= plt.subplots(figsize=(10,8))\nplt.title(\"Laterality vs Image Count\")\nax = sns.barplot(data=temp, x='laterality', y='image_id')\nax.bar_label(ax.containers[0])\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-12-16T06:39:27.032468Z","iopub.execute_input":"2022-12-16T06:39:27.033454Z","iopub.status.idle":"2022-12-16T06:39:27.250664Z","shell.execute_reply.started":"2022-12-16T06:39:27.033391Z","shell.execute_reply":"2022-12-16T06:39:27.249869Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Here, it cane be found that number of left breast images and right breast images is almost same.
","metadata":{}},{"cell_type":"markdown","source":"### Does both of the laterality is captured for a patient?","metadata":{}},{"cell_type":"code","source":"left = set()\nright = set()\nboth = set()\nfor index, row in df.iterrows():\n if row['patient_id'] not in left and row['patient_id'] not in right and row['patient_id'] not in both:\n if row['laterality'] == 'L': left.add(row['patient_id'])\n else: right.add(row['patient_id'])\n else:\n if row['patient_id'] in both: continue\n elif row['patient_id'] in left and row['laterality'] == 'R': \n left.remove(row['patient_id'])\n both.add(row['patient_id'])\n elif row['patient_id'] in right and row['laterality'] == 'L':\n right.remove(row['patient_id'])\n both.add(row['patient_id'])","metadata":{"execution":{"iopub.status.busy":"2022-12-16T06:39:27.251978Z","iopub.execute_input":"2022-12-16T06:39:27.252479Z","iopub.status.idle":"2022-12-16T06:39:31.064449Z","shell.execute_reply.started":"2022-12-16T06:39:27.252447Z","shell.execute_reply":"2022-12-16T06:39:31.063156Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"temp = {'left': len(left), 'right': len(right), 'both': len(both)}\nprint(\"Patient count depends on image laterality: \", temp)","metadata":{"execution":{"iopub.status.busy":"2022-12-16T06:39:31.066251Z","iopub.execute_input":"2022-12-16T06:39:31.066637Z","iopub.status.idle":"2022-12-16T06:39:31.073412Z","shell.execute_reply.started":"2022-12-16T06:39:31.066602Z","shell.execute_reply":"2022-12-16T06:39:31.072222Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Here, we can find that for all the patients both of the breasts images are taken to examine.
","metadata":{}},{"cell_type":"markdown","source":"##There are typically four different views that are used in mammography: the top view, the side view, the oblique view, and the angled view. The top view is taken from directly above the breast, while the side view is taken from the side of the breast. The oblique view is taken at an angle, and the angled view is taken from below the breast. These different views are used to create a more complete and detailed image of the breast tissue, which can help doctors to identify abnormalities and diagnose breast cancer. Some mammography machines also have the capability to take 3D images, which can provide even more detailed information about the breast tissue.
\nIt can be noted that CC and MLO is most common view. Rest of the views are very rare. These views are supplementary views and all of the supplementary views which are present here are somehow related with MLO view.
","metadata":{}},{"cell_type":"markdown","source":"### Realationship between image frequency of different laterality per view","metadata":{}},{"cell_type":"code","source":"# Realtionship between view and laterality\nviews = list(df['view'].unique())\nfig, axes = plt.subplots(nrows=1, ncols=6, figsize=(20,10))\nfor i in range(len(views)):\n view = views[i]\n temp = df[df['view'] == view]\n temp = temp.groupby('laterality')['image_id'].count().to_frame()\n temp.reset_index(inplace=True)\n temp.columns = [view, 'image_count']\n sns.barplot(data=temp, x=view, y = 'image_count', ax=axes[i])\n plt.xlabel(view)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-12-16T06:39:31.348232Z","iopub.execute_input":"2022-12-16T06:39:31.349295Z","iopub.status.idle":"2022-12-16T06:39:32.212662Z","shell.execute_reply.started":"2022-12-16T06:39:31.349253Z","shell.execute_reply":"2022-12-16T06:39:32.211373Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"From the above barplots it is clear that view of camera and laterality of breasts have no specific relationship.
","metadata":{}},{"cell_type":"markdown","source":"##Age is a risk factor for breast cancer. This means that as a person gets older, their risk of developing breast cancer increases. The risk of breast cancer is generally low for women under the age of 40, but it increases significantly after the age of 40. By the time a woman reaches the age of 60, her risk of developing breast cancer is about 1 in 8. After the age of 70, the risk of breast cancer continues to increase, although the rate of increase slows down. However, it is important to note that breast cancer can affect people of any age, and it is not uncommon for younger women to be diagnosed with the disease.
","metadata":{}},{"cell_type":"markdown","source":"### Distribution of Age","metadata":{}},{"cell_type":"code","source":"print(\"Basic statistical description of age: \")\ndf['age'].describe()","metadata":{"execution":{"iopub.status.busy":"2022-12-16T06:39:32.214076Z","iopub.execute_input":"2022-12-16T06:39:32.214409Z","iopub.status.idle":"2022-12-16T06:39:32.230301Z","shell.execute_reply.started":"2022-12-16T06:39:32.214379Z","shell.execute_reply":"2022-12-16T06:39:32.229097Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sns.displot(data=df, x='age', kde=True)\nplt.title(\"Distribution of Age\")","metadata":{"execution":{"iopub.status.busy":"2022-12-16T06:39:32.232145Z","iopub.execute_input":"2022-12-16T06:39:32.232504Z","iopub.status.idle":"2022-12-16T06:39:32.973171Z","shell.execute_reply.started":"2022-12-16T06:39:32.232468Z","shell.execute_reply":"2022-12-16T06:39:32.97179Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Distribution and count of the age column\nm = {\n '20-30': 0,\n '30-40': 0,\n '40-50': 0,\n '50-60': 0,\n '60-70': 0,\n '70-80': 0,\n '80-90': 0\n}\nfor index, row in df.iterrows():\n if row['age']>=20 and row['age']<30: m['20-30'] += 1 \n elif row['age']>=30 and row['age']<40: m['30-40'] += 1\n elif row['age']>=40 and row['age']<50: m['40-50'] += 1\n elif row['age']>=50 and row['age']<60: m['50-60'] += 1\n elif row['age']>=60 and row['age']<70: m['60-70'] += 1\n elif row['age']>=70 and row['age']<80: m['70-80'] += 1\n else: m['80-90'] += 1\nfig, ax = plt.subplots(figsize=(12,10))\nax = sns.barplot(x=list(m.keys()), y=list(m.values()))\nax.bar_label(ax.containers[0])\nplt.title(\"Patient Count of Different Age Groups\")\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-12-16T06:39:32.975139Z","iopub.execute_input":"2022-12-16T06:39:32.975581Z","iopub.status.idle":"2022-12-16T06:39:37.839794Z","shell.execute_reply.started":"2022-12-16T06:39:32.975538Z","shell.execute_reply":"2022-12-16T06:39:37.838587Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"From the above chart and plot it can be noticed that people from the age group of 50 to 70 have more chances of occurence of breast canncer. The distribution looks like a bell curve. Distribution of age almost like a normal distribution.
","metadata":{}},{"cell_type":"markdown","source":"##Mammography is a type of medical imaging that uses low-energy X-rays to create detailed images of the breasts. It is typically used to detect and diagnose breast cancer, as well as to monitor the progress of treatment for breast cancer. The procedure involves the use of a specialized X-ray machine, which produces images that can be examined by a doctor to look for abnormalities in the breast tissue. Mammography is generally recommended for women over the age of 40, or for women with a family history of breast cancer, as a way to detect the disease in its early stages, when it is most treatable. Early detection of breast cancer is important because it can improve the chances of successful treatment.
","metadata":{}},{"cell_type":"markdown","source":"### Frequency images labelled as normal and cancer","metadata":{}},{"cell_type":"code","source":"print(\"Unique value od cancer column: \",df['cancer'].unique())","metadata":{"execution":{"iopub.status.busy":"2022-12-16T06:39:37.841221Z","iopub.execute_input":"2022-12-16T06:39:37.841541Z","iopub.status.idle":"2022-12-16T06:39:37.847568Z","shell.execute_reply.started":"2022-12-16T06:39:37.84151Z","shell.execute_reply":"2022-12-16T06:39:37.846331Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Here, 0 indicates negative to cancer and 1 indicates positive to cancer.
","metadata":{}},{"cell_type":"code","source":"temp = df['cancer'].value_counts().to_frame()\ntemp.reset_index(inplace=True)\ntemp.columns = ['cancer', 'patient_count']\nfig, ax = plt.subplots(figsize=(12, 8))\nax = sns.barplot(data=temp, x=temp.columns[0], y=temp.columns[1])\nax.bar_label(ax.containers[0])\nplt.title('Frequency of normal images and breast cancer images')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-12-16T06:39:37.849202Z","iopub.execute_input":"2022-12-16T06:39:37.849572Z","iopub.status.idle":"2022-12-16T06:39:38.073975Z","shell.execute_reply.started":"2022-12-16T06:39:37.84954Z","shell.execute_reply":"2022-12-16T06:39:38.072879Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
The dataset is highly imbalanced. There is total 1158 images which are labelled as cancer. Number of cancer patients is too less in the dataset. Either we need to collect more data from other resoureces or augmentation should be applied to increase the length of the dataset.
","metadata":{}},{"cell_type":"markdown","source":"### Frequency of cancer and normal patients","metadata":{}},{"cell_type":"code","source":"# Let's visualize how many patient in the dataset have cancer and how many don't\ntemp = df.groupby('patient_id')['cancer'].max().to_frame()\ntemp.reset_index(inplace=True)\ntemp = temp['cancer'].value_counts().to_frame()\ntemp.reset_index(inplace=True)\ntemp.columns = ['cancer', 'patient_count']\nfig,ax = plt.subplots(figsize=(12, 8))\nax = sns.barplot(data=temp, x=temp.columns[0], y=temp.columns[1])\nax.bar_label(ax.containers[0])\nplt.title('Frequency of normal patients and breast cancer patients')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-12-16T06:39:38.075422Z","iopub.execute_input":"2022-12-16T06:39:38.075871Z","iopub.status.idle":"2022-12-16T06:39:38.291189Z","shell.execute_reply.started":"2022-12-16T06:39:38.07581Z","shell.execute_reply":"2022-12-16T06:39:38.289891Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Among all the patients 486 patients were affected by cancer.
","metadata":{}},{"cell_type":"markdown","source":"### Realtionship between age and cancer","metadata":{}},{"cell_type":"code","source":"# Let's check patients of which age group is more affected by cancer\ntemp = df[df['cancer']==1]\ntemp = temp.groupby('patient_id')['age'].max().to_frame()\ntemp.reset_index(inplace=True)\ntemp = temp['age'].value_counts().to_frame()\ntemp.reset_index(inplace=True)\ntemp.columns = ['age', 'patient_count']\ntemp['age'].describe()","metadata":{"execution":{"iopub.status.busy":"2022-12-16T06:39:38.292967Z","iopub.execute_input":"2022-12-16T06:39:38.293709Z","iopub.status.idle":"2022-12-16T06:39:38.312223Z","shell.execute_reply.started":"2022-12-16T06:39:38.293665Z","shell.execute_reply":"2022-12-16T06:39:38.310689Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Here it can be found that patients of mid age and old age patients have more chances of getting cancer.
","metadata":{}},{"cell_type":"code","source":"# Now let's check identified cancer patients and relationship with age\ntemp = df.groupby('patient_id')[['age', 'cancer']].max()\nm = {\n \"30-40\": [0, 0],\n \"40-50\": [0, 0],\n \"50-60\": [0, 0],\n \"60-70\": [0, 0],\n \"70-80\": [0, 0],\n \"80-90\": [0, 0]\n}\nfor index, row in temp.iterrows():\n age = row['age']\n if age>=30 and age<40: \n m['30-40'][0] += 1\n if row['cancer']==1: m['30-40'][1] += 1\n elif age>=40 and age<50: \n m['40-50'][0] += 1\n if row['cancer']==1: m['40-50'][1] += 1\n elif age>=50 and age<60: \n m['50-60'][0] += 1\n if row['cancer']==1: m['50-60'][1] += 1\n elif age>=60 and age<70: \n m['60-70'][0] += 1\n if row['cancer']==1: m['60-70'][1] += 1\n elif age>=70 and age<80: \n m['70-80'][0] += 1\n if row['cancer']==1: m['70-80'][1] += 1\n else: \n m['80-90'][0] += 1\n if row['cancer']==1: m['80-90'][1] += 1\ntemp = {}\nfor key, val in m.items(): temp[key] = val[1] / val[0]\nfig, ax = plt.subplots(figsize=(12,10))\nax = sns.barplot(x=list(temp.keys()), y=list(temp.values()))\nplt.title(\"Percentage of cancer patient of different age groups\")\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-12-16T06:39:38.313681Z","iopub.execute_input":"2022-12-16T06:39:38.314306Z","iopub.status.idle":"2022-12-16T06:39:39.196725Z","shell.execute_reply.started":"2022-12-16T06:39:38.314268Z","shell.execute_reply":"2022-12-16T06:39:39.195802Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"It is clear that, age is an important factor which influence that a patient can have cancer or not. With increasing age chances of breast cancer increases in woman.
","metadata":{}},{"cell_type":"markdown","source":"### Is there any relatiopnship between cancer and laterality of breast?","metadata":{}},{"cell_type":"code","source":"temp = df[df['cancer'] == 1]['laterality'].to_frame()\nfig, ax = plt.subplots(figsize=(12,8))\nax = sns.countplot(x=temp['laterality'])\nax.bar_label(ax.containers[0])\nplt.title(\"Frequency of cancer patients and laterality\")\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-12-16T06:39:39.197804Z","iopub.execute_input":"2022-12-16T06:39:39.198456Z","iopub.status.idle":"2022-12-16T06:39:39.402535Z","shell.execute_reply.started":"2022-12-16T06:39:39.198417Z","shell.execute_reply":"2022-12-16T06:39:39.401739Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"From the data it can assumed that there is no specific relationship between laterality of breast and cancer.
\nBut from the research it is found that there is a relationship between breast cancer and laterality, or the side of the breast in which the cancer develops. According to research, breast cancer is more likely to occur in the left breast than the right breast. It is not clear why this is the case, but some theories suggest that it may be due to hormonal differences between the left and right breast or differences in the anatomy of the lymphatic system in the two breasts. However, more research is needed to fully understand this relationship.
","metadata":{}},{"cell_type":"markdown","source":"### Realtionship between view of image and cancer","metadata":{}},{"cell_type":"code","source":"temp = df[df['cancer']==1]['view'].to_frame()\nfig, ax = plt.subplots(figsize=(12,8))\nax = sns.countplot(x=temp['view'])\nax.bar_label(ax.containers[0])\nplt.title(\"relation between frequency of cancer patients and hospital id\")\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-12-16T06:39:39.403557Z","iopub.execute_input":"2022-12-16T06:39:39.404131Z","iopub.status.idle":"2022-12-16T06:39:39.622255Z","shell.execute_reply.started":"2022-12-16T06:39:39.404083Z","shell.execute_reply":"2022-12-16T06:39:39.620857Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Most of the images labelled as cancer are captured in 'MLO' and 'CC' view which are also most commomnm view to identify cancer. Other views are not directly related with cancer. Those views may have some relationship with some other attributes.
","metadata":{}},{"cell_type":"markdown","source":"##A biopsy is a medical procedure in which a small sample of tissue is removed from the body and examined under a microscope. In the context of breast cancer, a biopsy is typically performed to confirm a diagnosis of breast cancer, to determine the type of cancer, and to evaluate the characteristics of the cancer cells. This information can be used to help determine the best course of treatment for the patient. There are several different types of biopsy procedures that may be used to diagnose breast cancer, including fine needle aspiration, core needle biopsy, and surgical biopsy. These procedures are typically performed by a doctor, and the results of the biopsy are examined by a pathologist to make a diagnosis of breast cancer.
","metadata":{}},{"cell_type":"markdown","source":"### Images labelled with biopsy","metadata":{}},{"cell_type":"code","source":"fig, ax = plt.subplots(figsize=(12,8))\nax = sns.countplot(x=df['biopsy'])\nax.bar_label(ax.containers[0])\nplt.title(\"Images with biopsy\")\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-12-16T06:39:39.623934Z","iopub.execute_input":"2022-12-16T06:39:39.62429Z","iopub.status.idle":"2022-12-16T06:39:39.82835Z","shell.execute_reply.started":"2022-12-16T06:39:39.624257Z","shell.execute_reply":"2022-12-16T06:39:39.827103Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"There are 2969 images which are labelled as biopsy.
","metadata":{}},{"cell_type":"markdown","source":"### Patients who treated with biopsy","metadata":{}},{"cell_type":"code","source":"temp = df[['patient_id', 'biopsy']]\ntemp = temp.groupby('patient_id')['biopsy'].max().to_frame()\nfig, ax = plt.subplots(figsize=(12,8))\nax = sns.countplot(x=temp['biopsy'])\nax.bar_label(ax.containers[0])\nplt.title(\"Images with biopsy\")\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-12-16T06:39:39.829629Z","iopub.execute_input":"2022-12-16T06:39:39.830489Z","iopub.status.idle":"2022-12-16T06:39:40.03397Z","shell.execute_reply.started":"2022-12-16T06:39:39.83045Z","shell.execute_reply":"2022-12-16T06:39:40.032876Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"There are total 1171 patients who had biopsy.
","metadata":{}},{"cell_type":"markdown","source":"### Is there any relationship with biopsy and view of image?","metadata":{}},{"cell_type":"code","source":"temp = df[df['biopsy']==1]['view'].to_frame()\nfig, ax = plt.subplots(figsize=(12,8))\nax = sns.countplot(x=temp['view'])\nax.bar_label(ax.containers[0])\nplt.title(\"Relationship between biopsy and view\")\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-12-16T06:39:40.035331Z","iopub.execute_input":"2022-12-16T06:39:40.035635Z","iopub.status.idle":"2022-12-16T06:39:40.266209Z","shell.execute_reply.started":"2022-12-16T06:39:40.035606Z","shell.execute_reply":"2022-12-16T06:39:40.264983Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Here it can be found that for very less number of cases 'AT' and 'ML' is performed who has biopsy.
\nThe view of an image (such as an x-ray or mammogram) can affect the accuracy of a breast cancer diagnosis, especially when it comes to biopsy. A biopsy is a procedure in which a small sample of tissue is removed from the breast and examined under a microscope to determine if cancer cells are present. The view of the image can help the doctor locate the area of the breast where the biopsy will be performed. For example, the top-down or cranio-caudal view is commonly used to help the doctor determine the exact location of a biopsy. In addition, a side or oblique view may be taken to provide additional information about the breast tissue and help the doctor make a more accurate diagnosis. Overall, the view of the image can play a important role in the accuracy of a breast cancer diagnosis through biopsy.
","metadata":{}},{"cell_type":"code","source":"# Let's check the patient who had AT and ML view image and treated with biopsy\ntemp = df[(df['biopsy']==1) & ((df['view']=='AT') | (df['view']=='ML'))][['patient_id', 'view', 'laterality', 'cancer']]\ntemp","metadata":{"execution":{"iopub.status.busy":"2022-12-16T06:39:40.268094Z","iopub.execute_input":"2022-12-16T06:39:40.268567Z","iopub.status.idle":"2022-12-16T06:39:40.29372Z","shell.execute_reply.started":"2022-12-16T06:39:40.268521Z","shell.execute_reply":"2022-12-16T06:39:40.292146Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"It can be noticed that AT and ML both aren't performed on a single person who has biopsy. These two views has no direct relationship with biopsy.
","metadata":{}},{"cell_type":"markdown","source":"### Is there any relationship between laterality and biopsy?","metadata":{}},{"cell_type":"code","source":"temp = df[df['biopsy']==1]['laterality'].to_frame()\nfig, ax = plt.subplots(figsize=(12,8))\nax = sns.countplot(x=temp['laterality'])\nax.bar_label(ax.containers[0])\nplt.title(\"Relationship between biopsy and laterality\")\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-12-16T06:39:40.29492Z","iopub.execute_input":"2022-12-16T06:39:40.295631Z","iopub.status.idle":"2022-12-16T06:39:40.508704Z","shell.execute_reply.started":"2022-12-16T06:39:40.295596Z","shell.execute_reply":"2022-12-16T06:39:40.507738Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"There is no visible relationship between laterality of image and biopsy but it can be seen here that biopsy is performed more on left breast than right breast and the reason is, it is found that breast cancer is more likely to occur on left breast than right breast.
","metadata":{}},{"cell_type":"markdown","source":"### Relationship between age and biopsy","metadata":{}},{"cell_type":"code","source":"temp = df[df['biopsy']==1][['age', 'patient_id']]\ntemp = temp.groupby('patient_id')['age'].max().to_frame()\ntemp = temp['age'].value_counts().to_frame()\ntemp.columns = ['patient_count']\nfig, ax = plt.subplots(figsize=(18,6))\nsns.lineplot(data=temp, palette=\"tab10\", linewidth=2.5, ax=ax)\nplt.title(\"Relation between age and biopsy\")\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-12-16T06:39:40.510331Z","iopub.execute_input":"2022-12-16T06:39:40.511497Z","iopub.status.idle":"2022-12-16T06:39:40.791981Z","shell.execute_reply.started":"2022-12-16T06:39:40.511445Z","shell.execute_reply":"2022-12-16T06:39:40.79078Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"From the plot it can be seen that biopsy is more likely to perform on the patients who are more than 45.
\nThere is a relationship between age and the likelihood of undergoing a biopsy for breast cancer. As women get older, their risk of developing breast cancer increases, so they are more likely to undergo a biopsy to check for the disease. In addition, as women get older, they are more likely to have changes in their breast tissue that may be indicative of breast cancer, such as lumps or masses. These changes can be detected through mammography or other imaging tests, and may require a biopsy to confirm or rule out the presence of cancer. Therefore, age is an important factor to consider when determining the need for a biopsy for breast cancer.
","metadata":{}},{"cell_type":"markdown","source":"### Relationship between cancer and biopsy","metadata":{}},{"cell_type":"code","source":"temp = df[['cancer', 'biopsy']]\nfig, ax = plt.subplots(figsize=(12,6))\nsns.countplot(data=temp, x = 'biopsy', hue = 'cancer', ax=ax)\nax.bar_label(ax.containers[0])\nplt.title(\"Relation between age and biopsy\")\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-12-16T06:39:40.799962Z","iopub.execute_input":"2022-12-16T06:39:40.800362Z","iopub.status.idle":"2022-12-16T06:39:41.03057Z","shell.execute_reply.started":"2022-12-16T06:39:40.800326Z","shell.execute_reply":"2022-12-16T06:39:41.029773Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Biopsy is never performed on some patient who don't have cancer and what is the reason for this? A biopsy is a procedure in which a small sample of tissue is removed from the body and examined under a microscope to determine if cancer cells are present. Biopsies are typically performed on patients who have abnormal findings on imaging tests, such as mammography or ultrasound, or who have symptoms that may be indicative of cancer, such as a lump or mass in the breast. The decision to perform a biopsy is made by a doctor based on the individual patient's medical history, symptoms, and imaging test results. Biopsies are an important tool in the diagnosis of cancer, and can help doctors determine the type and extent of the disease, as well as the best course of treatment.
","metadata":{}},{"cell_type":"markdown","source":"##Invasive breast cancer is a type of breast cancer that has spread from the milk ducts or lobules where it originated, into the surrounding breast tissue. This type of cancer is called \"invasive\" because it has the ability to invade and damage nearby healthy tissue. Invasive breast cancer is typically more difficult to treat than non-invasive breast cancer, because it has already spread beyond the original site of the cancer. Treatment for invasive breast cancer may include surgery, radiation therapy, chemotherapy, and targeted therapy. It is important for anyone with symptoms of breast cancer to talk to their doctor for a proper diagnosis and treatment plan. Early detection and treatment of breast cancer can improve the chances of successful treatment.
","metadata":{}},{"cell_type":"markdown","source":"### Number of images with invasive cancer","metadata":{}},{"cell_type":"code","source":"print(\"Number of invasive cancer images:\", len(df[df.invasive==1]))","metadata":{"execution":{"iopub.status.busy":"2022-12-16T06:39:41.031592Z","iopub.execute_input":"2022-12-16T06:39:41.032504Z","iopub.status.idle":"2022-12-16T06:39:41.040615Z","shell.execute_reply.started":"2022-12-16T06:39:41.032469Z","shell.execute_reply":"2022-12-16T06:39:41.039178Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Number of patients with invasive cancer","metadata":{}},{"cell_type":"code","source":"temp = df[df['invasive']==1]['patient_id'].to_frame()\nprint(\"Number of patients with invasve cancer:\", temp['patient_id'].nunique())","metadata":{"execution":{"iopub.status.busy":"2022-12-16T06:39:41.041934Z","iopub.execute_input":"2022-12-16T06:39:41.042255Z","iopub.status.idle":"2022-12-16T06:39:41.055458Z","shell.execute_reply.started":"2022-12-16T06:39:41.042226Z","shell.execute_reply":"2022-12-16T06:39:41.054252Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Relationship between patients with invasive cancer and number of images contributed by individual","metadata":{}},{"cell_type":"code","source":"temp = df[df['invasive']==1][['patient_id', 'image_id']]\ntemp = temp.groupby('patient_id')['image_id'].count().to_frame()\ntemp.reset_index(inplace=True)\ntemp.columns = ['patient_id', 'image_count']\nfig, ax = plt.subplots(figsize=(12,6))\nsns.countplot(data=temp, x = 'image_count', ax=ax)\nax.bar_label(ax.containers[0])\nplt.title('''Relation between patients with invasive cancer and number of images contributed by individual \n which are labelled as invasive''')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-12-16T06:39:41.057079Z","iopub.execute_input":"2022-12-16T06:39:41.057429Z","iopub.status.idle":"2022-12-16T06:39:41.285637Z","shell.execute_reply.started":"2022-12-16T06:39:41.057398Z","shell.execute_reply":"2022-12-16T06:39:41.284795Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"It is found that minimum number of images are taken for a particular patient is 4 but here it is found that for a particular patient minimum number of images labelled as invasive is 2. So, now the question is how is it possible? If a patient has minimum 4 images how 2 images can be invasive? There can be multiple reason for this and concluding the exact reason is really difficult with the given data but most probable reasons can be (i)From some view of images its is not possible to decide that the cancer is invasive or not, (ii) As the timeframe isn't given it is also possible that cancer becomes invasive with time for a patient etc.
","metadata":{}},{"cell_type":"markdown","source":"### Is there any relationship between invasive cancer and laterality of breast?","metadata":{}},{"cell_type":"code","source":"temp = df[df['invasive']==1]['laterality'].to_frame()\nfig, ax = plt.subplots(figsize=(12,6))\nsns.countplot(data=temp, x = 'laterality', ax=ax)\nax.bar_label(ax.containers[0])\nplt.title(\"Relation between patients with invasive cancer and laterality of breast\")\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-12-16T06:39:41.287314Z","iopub.execute_input":"2022-12-16T06:39:41.28782Z","iopub.status.idle":"2022-12-16T06:39:41.50113Z","shell.execute_reply.started":"2022-12-16T06:39:41.28777Z","shell.execute_reply":"2022-12-16T06:39:41.499772Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"There is no visible relationship between invasive cancer and laterality of breast.
","metadata":{}},{"cell_type":"markdown","source":"### Is there any particular view which is related to invasive?","metadata":{}},{"cell_type":"code","source":"temp = df[df['invasive']==1]['view'].to_frame()\nfig, ax = plt.subplots(figsize=(12,6))\nsns.countplot(data=temp, x = 'view', ax=ax)\nax.bar_label(ax.containers[0])\nplt.title(\"Relation between images of patients with invasive cancer and view of image\")\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-12-16T06:39:41.502402Z","iopub.execute_input":"2022-12-16T06:39:41.503025Z","iopub.status.idle":"2022-12-16T06:39:41.726396Z","shell.execute_reply.started":"2022-12-16T06:39:41.50299Z","shell.execute_reply":"2022-12-16T06:39:41.725167Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"An important point to note that, in AT view image is captured who had biopsy and invasive cancer. We need to check that is it same patient who has biopsy and invasive cancer.
\nThe standard view for mammography is the top-down or cranio-caudal view, which allows the doctor to see the breast tissue from the nipple to the chest wall. This view is commonly used to detect invasive breast cancer because it allows the doctor to see any abnormalities or changes in the breast tissue that may be indicative of the disease. In addition, a side or oblique view (also known as a mediolateral oblique view) may be taken to provide additional information about the breast tissue. These views can help the doctor make a more accurate diagnosis of breast cancer. Overall, the top-down view is commonly used to detect invasive breast cancer, but other views may also be used to provide a more comprehensive picture of the breast tissue.
","metadata":{}},{"cell_type":"code","source":"print(\"Let's check for patient who has biopsy and invasive cancer and AT image view: \")\ndf[(df['invasive']==1) & (df['biopsy']==1) & (df['view']=='AT')][['patient_id', 'image_id', 'BIRADS', 'density', 'difficult_negative_case']]","metadata":{"execution":{"iopub.status.busy":"2022-12-16T06:39:41.728264Z","iopub.execute_input":"2022-12-16T06:39:41.728885Z","iopub.status.idle":"2022-12-16T06:39:41.749718Z","shell.execute_reply.started":"2022-12-16T06:39:41.728847Z","shell.execute_reply":"2022-12-16T06:39:41.748895Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Yes, the images are same and belongs to a single patient who had biopsy and cancer and who image is taken from 'AT' view. It can be assumed that as the person's BI-RADS score is 0 and breast density is B therefore may be doctors had some difficulty to understand than ther person has invasive cancer or not and maybe this is the reason 'AT' view is used to confirm the status of the patient.
","metadata":{}},{"cell_type":"markdown","source":"### Check relationship between age and invasive cancer","metadata":{}},{"cell_type":"code","source":"temp = df[df['invasive']==1]['age'].to_frame()\ntemp = temp['age'].value_counts().to_frame()\ntemp.columns = ['patient_count']\nfig, ax = plt.subplots(figsize=(18,6))\nsns.lineplot(data=temp, palette=\"tab10\", linewidth=2.5, ax=ax)\nplt.title(\"Relation between age and invasive cancer\")\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-12-16T06:39:41.751103Z","iopub.execute_input":"2022-12-16T06:39:41.751663Z","iopub.status.idle":"2022-12-16T06:39:42.024053Z","shell.execute_reply.started":"2022-12-16T06:39:41.751625Z","shell.execute_reply":"2022-12-16T06:39:42.023144Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"It can be noted that patients with invasive cancer belongs in the age range of 50 to 75(approx).
\nThere is a relationship between age and the likelihood of developing invasive breast cancer. As women get older, their risk of developing breast cancer increases. This is because the cells in the body naturally become more susceptible to genetic mutations over time, which can lead to the development of cancer. In addition, hormonal changes that occur during menopause can also increase the risk of breast cancer. Therefore, age is an important factor to consider when it comes to the likelihood of developing invasive breast cancer. Women who are older are more likely to develop the disease, and should be especially vigilant about checking their breasts for any changes or abnormalities and getting regular mammograms.
","metadata":{}},{"cell_type":"markdown","source":"### Patients who had invasive cancer and treated with biopsy","metadata":{}},{"cell_type":"code","source":"temp = df[df['invasive']==1]['biopsy'].to_frame()['biopsy'].value_counts()\ntemp","metadata":{"execution":{"iopub.status.busy":"2022-12-16T06:39:42.025341Z","iopub.execute_input":"2022-12-16T06:39:42.025867Z","iopub.status.idle":"2022-12-16T06:39:42.03653Z","shell.execute_reply.started":"2022-12-16T06:39:42.025819Z","shell.execute_reply":"2022-12-16T06:39:42.035317Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Here it can be found that all the patients who had invasive cancer are treated with biopsy beacuse using biopsy it can be confirmed that the cancer is invasive or not.
","metadata":{}},{"cell_type":"markdown","source":"##BI-RADS stands for Breast Imaging Reporting and Data System. It is a standardized system that is used to report the results of mammography and other imaging tests for breast cancer. The BI-RADS system is used to communicate the findings of the imaging test, as well as to make recommendations for follow-up care. Each mammogram or other imaging test is assigned a BI-RADS score, which ranges from 0 to 6. A score of 0 indicates that the test is incomplete and more information is needed, while a score of 6 indicates that cancer is present and treatment is needed. The BI-RADS score is used to help doctors and patients understand the results of the imaging test and make decisions about next steps in the diagnosis and treatment of breast cancer.
\nRefer to the website for more understanding
Getting conclusions from BI-RADS feature is really hard as most of the values are missing here. Score 0 define incomplete test, Score 1 and 2 define that the person is negative from cancer but score 2 define that there can be some adnormality in brest.
","metadata":{}},{"cell_type":"code","source":"fig, ax = plt.subplots(figsize=(12,6))\nsns.countplot(data=df, x = 'BIRADS', ax=ax)\nax.bar_label(ax.containers[0])\nplt.title(\"Relation between patients with invasive cancer and view of image\")\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-12-16T06:39:42.063143Z","iopub.execute_input":"2022-12-16T06:39:42.063551Z","iopub.status.idle":"2022-12-16T06:39:42.293277Z","shell.execute_reply.started":"2022-12-16T06:39:42.063505Z","shell.execute_reply":"2022-12-16T06:39:42.291909Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Is there any cancer is reported with normal BI-RADS score?","metadata":{}},{"cell_type":"code","source":"fig, ax = plt.subplots(figsize=(12,6))\nsns.countplot(data=df, x = 'BIRADS', hue='cancer', ax=ax)\nax.bar_label(ax.containers[0])\nplt.title(\"Relation between patients with invasive cancer and view of image\")\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-12-16T06:39:42.295154Z","iopub.execute_input":"2022-12-16T06:39:42.296002Z","iopub.status.idle":"2022-12-16T06:39:42.556958Z","shell.execute_reply.started":"2022-12-16T06:39:42.295958Z","shell.execute_reply":"2022-12-16T06:39:42.55598Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"It indicates that some patient with 0 BI-RADS score can have cancer. As 0 indicates incomplete test.
","metadata":{}},{"cell_type":"markdown","source":"### Is there any relationship between patient's age who has 0 BI-RADS score and cancer?","metadata":{}},{"cell_type":"code","source":"temp = df[(df['BIRADS']==0.0) & (df['cancer']==1)][['patient_id', 'age']]\ntemp = temp.groupby('patient_id')['age'].max().to_frame()\ntemp.reset_index(inplace=True)\ntemp = temp['age'].value_counts().to_frame()\ntemp.columns = ['patient_count']\nfig, ax = plt.subplots(figsize=(18,6))\nsns.lineplot(data=temp, palette=\"tab10\", linewidth=2.5, ax=ax)\nplt.title(\"Relation between age and patients who have cancer with 0 biopsy score\")\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-12-16T06:39:42.558272Z","iopub.execute_input":"2022-12-16T06:39:42.558803Z","iopub.status.idle":"2022-12-16T06:39:42.839084Z","shell.execute_reply.started":"2022-12-16T06:39:42.55877Z","shell.execute_reply":"2022-12-16T06:39:42.83822Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"It can be noticed that for all the patients who are 55 - 75(approx) years old and having BI-RADS score is 0 also have high chance of having cancer because with increasing age chances of breast cancer increases.
","metadata":{}},{"cell_type":"markdown","source":"### BI-RADS score and Laterality","metadata":{}},{"cell_type":"code","source":"fig, ax = plt.subplots(figsize=(12,6))\nsns.countplot(data=df, x = 'BIRADS', hue='laterality', ax=ax)\nax.bar_label(ax.containers[0])\nplt.title(\"Relation between BI-RADS score and Laterality of Image\")\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-12-16T06:39:42.840334Z","iopub.execute_input":"2022-12-16T06:39:42.84084Z","iopub.status.idle":"2022-12-16T06:39:43.288619Z","shell.execute_reply.started":"2022-12-16T06:39:42.840794Z","shell.execute_reply":"2022-12-16T06:39:43.287681Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"It's obvious there shouldn't be any relationship between these two feature and it is clear from this plot.
\nThe BI-RADS score is not related to the laterality of the image, or the side of the breast shown in the image. The BI-RADS score is based on the findings of the mammogram, and reflects the likelihood of breast cancer based on the appearance of the breast tissue. The laterality of the image is not a factor in determining the BI-RADS score. However, the laterality of the breast cancer, or the side of the breast in which the cancer develops, may be considered when determining the best course of treatment. For example, if the cancer is found in the left breast, the doctor may recommend a different treatment plan than if the cancer was found in the right breast.
","metadata":{}},{"cell_type":"markdown","source":"### BI-RADS Score and view","metadata":{}},{"cell_type":"code","source":"fig, ax = plt.subplots(figsize=(12,6))\nsns.countplot(data=df, x = 'BIRADS', hue='view', ax=ax)\nax.bar_label(ax.containers[0])\nplt.title(\"Relation between BI-RADS score and View of image\")\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-12-16T06:39:43.289969Z","iopub.execute_input":"2022-12-16T06:39:43.290494Z","iopub.status.idle":"2022-12-16T06:39:43.65048Z","shell.execute_reply.started":"2022-12-16T06:39:43.29046Z","shell.execute_reply":"2022-12-16T06:39:43.649269Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"It is obvios that there is no relationship between BI-RADS score and view but but ....
The BI-RADS score is not directly related to the view of the image, but the view of the image can affect the accuracy of the mammogram and therefore the BI-RADS score. The standard view for mammography is the top-down or cranio-caudal view, which allows the doctor to see the breast tissue from the nipple to the chest wall. This view is important because it allows the doctor to see any abnormalities or changes in the breast tissue that may be indicative of breast cancer. In addition, a side or oblique view (also known as a mediolateral oblique view) may be taken to provide additional information about the breast tissue. These views can help the doctor make a more accurate diagnosis and assign a more accurate BI-RADS score.Implant\n
In the context of mammography, an implant refers to a breast implant, which is a medical device that is used to augment or reconstruct the breasts. Breast implants are typically made of silicone or saline, and they are inserted under the breast tissue or under the chest muscle to increase the size and shape of the breasts. Breast implants are used for a variety of reasons, including to restore the breasts after mastectomy, to correct congenital defects or asymmetry of the breasts, and to enhance the appearance of the breasts. Mammography is a type of medical imaging that is used to detect and diagnose breast cancer, and it is typically used on women with breast implants as well as women without implants. Special techniques may be needed to properly image the breast tissue in women with implants, and it is important for the patient to inform the mammographer if they have implants.
","metadata":{}},{"cell_type":"markdown","source":"### Number of images labelled as implant","metadata":{}},{"cell_type":"code","source":"print(\"Number of images labelled as implant: \",len(df[df['implant']==1]))","metadata":{"execution":{"iopub.status.busy":"2022-12-16T06:39:43.652845Z","iopub.execute_input":"2022-12-16T06:39:43.653301Z","iopub.status.idle":"2022-12-16T06:39:43.66127Z","shell.execute_reply.started":"2022-12-16T06:39:43.653255Z","shell.execute_reply":"2022-12-16T06:39:43.659723Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Number of patients who treated with breast implantation","metadata":{}},{"cell_type":"code","source":"print(\"Number of patients with breast implant: \",df[df['implant']==1]['patient_id'].to_frame()['patient_id'].nunique())","metadata":{"execution":{"iopub.status.busy":"2022-12-16T06:39:43.662733Z","iopub.execute_input":"2022-12-16T06:39:43.663124Z","iopub.status.idle":"2022-12-16T06:39:43.675804Z","shell.execute_reply.started":"2022-12-16T06:39:43.663082Z","shell.execute_reply":"2022-12-16T06:39:43.674688Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### View used to take images with breast implant","metadata":{}},{"cell_type":"code","source":"temp = df[df['implant']==1][['view']]\nfig, ax = plt.subplots(figsize=(12,6))\nsns.countplot(data=temp, x = 'view', ax=ax)\nax.bar_label(ax.containers[0])\nplt.title(\"Breast implant images by view\")\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-12-16T06:39:43.677011Z","iopub.execute_input":"2022-12-16T06:39:43.677742Z","iopub.status.idle":"2022-12-16T06:39:43.889014Z","shell.execute_reply.started":"2022-12-16T06:39:43.677685Z","shell.execute_reply":"2022-12-16T06:39:43.887751Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Here we can see that all the images for breast implant are taken by using the view of 'MLO' and 'CC'.
","metadata":{}},{"cell_type":"markdown","source":"### Does breast implant has any realtionship with cancer?","metadata":{}},{"cell_type":"code","source":"temp = df[df['implant']==1]\nfig, ax = plt.subplots(figsize=(12,6))\nsns.countplot(data=temp, x = 'cancer', ax=ax)\nax.bar_label(ax.containers[0])\nplt.title(\"Cancer patients who had breast implant\")\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-12-16T06:39:43.890359Z","iopub.execute_input":"2022-12-16T06:39:43.890713Z","iopub.status.idle":"2022-12-16T06:39:44.095146Z","shell.execute_reply.started":"2022-12-16T06:39:43.890673Z","shell.execute_reply":"2022-12-16T06:39:44.09403Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"temp = df[df['implant']==0]\nfig, ax = plt.subplots(figsize=(12,6))\nsns.countplot(data=temp, x = 'cancer', ax=ax)\nax.bar_label(ax.containers[0])\nplt.title(\"Cancer patients who hadn't breast implant\")\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-12-16T06:39:44.096474Z","iopub.execute_input":"2022-12-16T06:39:44.09684Z","iopub.status.idle":"2022-12-16T06:39:44.301808Z","shell.execute_reply.started":"2022-12-16T06:39:44.096776Z","shell.execute_reply":"2022-12-16T06:39:44.30054Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"From the dataset it can be observed that the ratio of cancer affected patients who had breast implant is 13:1464 and the ratio of cancer affected patients who hadn't brast implant is 1145:52084. From the ratio it can be assumed that breast implant has no direct affect on breast cancer.
\nThere is no evidence to suggest that breast implants cause cancer. However, breast implants can make it more difficult to detect breast cancer through mammography, as the implants can obscure the underlying breast tissue. This can make it more challenging for healthcare providers to accurately assess the breast tissue and detect any potential abnormalities. Additionally, in rare cases, breast implants have been associated with a type of cancer called anaplastic large cell lymphoma (ALCL), which is a cancer of the immune system. This type of cancer is very rare and is typically not life-threatening, but it is important for patients with breast implants to be aware of this potential risk and discuss it with their healthcare provider.
","metadata":{}},{"cell_type":"markdown","source":"##Breast density is a term that is used to describe the relative amount of glandular tissue and fat in the breasts. The breasts are composed of a mixture of glandular tissue, which produces milk, and fat, which provides padding and support. The proportion of glandular tissue to fat in the breasts is referred to as breast density. Breast density is an important factor in the interpretation of mammography results, because dense breast tissue can make it more difficult to detect abnormalities on a mammogram. Breast tissue is considered dense if there is a higher proportion of glandular tissue to fat in the breasts. Dense breast tissue appears white on a mammogram, just like cancerous tissue, so it can mask the presence of a tumor or other abnormality. Women with dense breasts may be at higher risk of breast cancer, and they may need additional imaging tests, such as ultrasound or MRI, to help detect any abnormalities.
\nRelationship between density and breast cancer
\nBreast density is a measure of the amount of fibrous and glandular tissue in the breast relative to the amount of fat. Breast density is determined by a healthcare provider during a mammogram and is classified into four categories:\n
As lot of data is missing, concluding some decision from the above graphs is really difficult task. No observable relation isn't found from the plots.
\n\n The density of the breast tissue can affect the accuracy of a mammogram, as dense tissue can obscure abnormalities and make them difficult to detect. Therefore, the specific views used for mammography may vary depending on the density of the breast tissue. In general, views that provide more detail and better visualization of the breast tissue may be used for women with dense breasts, while views that provide a broader overview of the breast may be used for women with less dense breasts. The specific views used will depend on the individual patient and the recommendation of their healthcare provider.\n
","metadata":{}},{"cell_type":"markdown","source":"### Relation between cancer and density","metadata":{}},{"cell_type":"code","source":"temp = {\n 'A': len(df[(df['cancer']==1) & (df['density']=='A')]) / len(df[df['density']=='A']) * 100,\n 'B': len(df[(df['cancer']==1) & (df['density']=='B')]) / len(df[df['density']=='B']) * 100,\n 'C': len(df[(df['cancer']==1) & (df['density']=='C')]) / len(df[df['density']=='C']) * 100,\n 'D': len(df[(df['cancer']==1) & (df['density']=='D')]) / len(df[df['density']=='D']) * 100\n }\nfig, ax = plt.subplots(figsize=(12,6))\nsns.barplot(x = list(temp.keys()), y = list(temp.values()), ax=ax)\nplt.title(\"Relation between density and cancer\")\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-12-16T06:39:45.457954Z","iopub.execute_input":"2022-12-16T06:39:45.458294Z","iopub.status.idle":"2022-12-16T06:39:45.630769Z","shell.execute_reply.started":"2022-12-16T06:39:45.458262Z","shell.execute_reply":"2022-12-16T06:39:45.629902Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Relation between invasive and density","metadata":{}},{"cell_type":"code","source":"temp = {\n 'A': len(df[(df['invasive']==1) & (df['density']=='A')]) / len(df[df['density']=='A']) * 100,\n 'B': len(df[(df['invasive']==1) & (df['density']=='B')]) / len(df[df['density']=='B']) * 100,\n 'C': len(df[(df['invasive']==1) & (df['density']=='C')]) / len(df[df['density']=='C']) * 100,\n 'D': len(df[(df['invasive']==1) & (df['density']=='D')]) / len(df[df['density']=='D']) * 100\n }\nfig, ax = plt.subplots(figsize=(12,6))\nsns.barplot(x = list(temp.keys()), y = list(temp.values()), ax=ax)\nplt.title(\"Relation between density and invasive cancer\")\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-12-16T06:39:45.632447Z","iopub.execute_input":"2022-12-16T06:39:45.633285Z","iopub.status.idle":"2022-12-16T06:39:45.813819Z","shell.execute_reply.started":"2022-12-16T06:39:45.633237Z","shell.execute_reply":"2022-12-16T06:39:45.812413Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"As majority of the data is missing for density column therfore concluding any decision is hard but from research it is found that with increasing density chances of cancer is also increased which can also be observed by above three plots.
\nThere is evidence to suggest that women with dense breast tissue may be at an increased risk of developing invasive breast cancer. This is because dense breast tissue can obscure abnormalities and make them difficult to detect on a mammogram, which is the primary screening tool for breast cancer. Therefore, women with dense breast tissue may require additional imaging tests, such as ultrasound or MRI, in order to accurately assess their breast health. However, it is important to note that not all women with dense breasts will develop invasive breast cancer, and many women with dense breasts do not have any abnormalities on their mammogram. The risk of invasive breast cancer varies from person to person and is influenced by a variety of factors, including genetics, lifestyle, and medical history.
","metadata":{}},{"cell_type":"markdown","source":"##The machine used for mammography is called a mammography machine or a mammogram machine. It is a specialized X-ray machine that is designed specifically for imaging the breasts. Mammography machines use low-energy X-rays to create detailed images of the breast tissue, which can be examined by a doctor to look for abnormalities that may indicate the presence of breast cancer. Mammography machines typically consist of a flat X-ray table, a moveable X-ray tube, and a computerized system for displaying and storing the images. Some mammography machines also have the capability to take 3D images, which can provide even more detailed information about the breast tissue.
","metadata":{}},{"cell_type":"markdown","source":"### Number of machines used","metadata":{}},{"cell_type":"code","source":"print(\"Machines used for imaging: \",df['machine_id'].unique())","metadata":{"execution":{"iopub.status.busy":"2022-12-16T06:39:45.815554Z","iopub.execute_input":"2022-12-16T06:39:45.816747Z","iopub.status.idle":"2022-12-16T06:39:45.825196Z","shell.execute_reply.started":"2022-12-16T06:39:45.816695Z","shell.execute_reply":"2022-12-16T06:39:45.823842Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Which machine if used how many times for examination","metadata":{}},{"cell_type":"code","source":"fig, ax = plt.subplots(figsize=(18,6))\nsns.countplot(data=df, x = 'machine_id', ax=ax)\nplt.title(\"Machine and Views\")\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-12-16T06:39:45.827258Z","iopub.execute_input":"2022-12-16T06:39:45.828163Z","iopub.status.idle":"2022-12-16T06:39:46.008395Z","shell.execute_reply.started":"2022-12-16T06:39:45.828104Z","shell.execute_reply":"2022-12-16T06:39:46.006658Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Does machine id refers to the type of machine?","metadata":{}},{"cell_type":"code","source":"machine_hospital = {}\nfor mid in df['machine_id'].unique():\n machine_hospital[mid] = df[df['machine_id']==mid]['site_id'].unique()\nprint(\"Machine and the hospital it is associated with: \")\nfor key, val in machine_hospital.items():\n print(\"Machine Id: \",key,\" present in Hospital Id: \",val)","metadata":{"execution":{"iopub.status.busy":"2022-12-16T06:39:46.010385Z","iopub.execute_input":"2022-12-16T06:39:46.011332Z","iopub.status.idle":"2022-12-16T06:39:46.039195Z","shell.execute_reply.started":"2022-12-16T06:39:46.011278Z","shell.execute_reply":"2022-12-16T06:39:46.037996Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Here it can be found that a machine with an id isn't present in both of the hospital. Therfore, it can't be concluded that machine_id defines the type of the machine or it's just simple id of machines.
","metadata":{}},{"cell_type":"markdown","source":"### Which machine is used for which view?","metadata":{}},{"cell_type":"code","source":"fig, ax = plt.subplots(figsize=(18,6))\nsns.countplot(data=df, x = 'machine_id', hue='view', ax=ax)\nplt.title(\"Machine and Views\")\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-12-16T06:39:46.040445Z","iopub.execute_input":"2022-12-16T06:39:46.040767Z","iopub.status.idle":"2022-12-16T06:39:46.514062Z","shell.execute_reply.started":"2022-12-16T06:39:46.040738Z","shell.execute_reply":"2022-12-16T06:39:46.512733Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"From the plot it can be seen that ''CC' and 'MLO' can be captured from all the device but as the number of images for other views is very small it is really difficult to get the idea which machine is associated with other views except 'CC' and \"MLO'
","metadata":{}},{"cell_type":"code","source":"temp = df[(df['view']!='CC') & (df['view']!='MLO')]\nfig, ax = plt.subplots(figsize=(18,6))\nsns.countplot(data=temp, x = 'view', hue='machine_id', ax=ax)\nplt.title(\"Machine and Views\")\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-12-16T06:39:46.516421Z","iopub.execute_input":"2022-12-16T06:39:46.517184Z","iopub.status.idle":"2022-12-16T06:39:46.828111Z","shell.execute_reply.started":"2022-12-16T06:39:46.517136Z","shell.execute_reply":"2022-12-16T06:39:46.826882Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Here, it can be observed that some machine is only used for some particular views. Other machines are used for all other views.
","metadata":{}},{"cell_type":"markdown","source":"### Cancer and Machine Id","metadata":{}},{"cell_type":"code","source":"temp = df[df['cancer']==1]\nfig, ax = plt.subplots(figsize=(18,6))\nsns.countplot(data=temp, x = 'machine_id', ax=ax)\nplt.title(\"Machine and Cancer\")\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-12-16T06:39:46.83009Z","iopub.execute_input":"2022-12-16T06:39:46.830553Z","iopub.status.idle":"2022-12-16T06:39:47.066222Z","shell.execute_reply.started":"2022-12-16T06:39:46.830506Z","shell.execute_reply":"2022-12-16T06:39:47.06507Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"temp = {}\nfor machine in df['machine_id'].unique():\n temp[machine] = len(df[(df['machine_id']==machine) & (df['cancer']==1)]) / len(df['machine_id']==machine) * 100\nfig, ax = plt.subplots(figsize=(18,6))\nsns.barplot(x = list(temp.keys()), y = list(temp.values()), ax=ax)\nax.bar_label(ax.containers[0])\nplt.title(\"Machine and Ratio of Cancer\")\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-12-16T06:39:47.067483Z","iopub.execute_input":"2022-12-16T06:39:47.06786Z","iopub.status.idle":"2022-12-16T06:39:47.371582Z","shell.execute_reply.started":"2022-12-16T06:39:47.067801Z","shell.execute_reply":"2022-12-16T06:39:47.370255Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Most of the patients who are examined with 49 have cancer but we shouldn't think in this way that patients who are examined with machine 49 has higher chances of affected by cancer. It's very natural that if a machine is used more than other machines to detect cancer then the machine will successfully identify more cancer patients just beacuse it gets more chance to detect cancer.
","metadata":{}},{"cell_type":"code","source":"temp = df[(df['cancer']==1) & ((df['view']!='CC') & (df['view']!='MLO'))]\ntemp[['view', 'machine_id', 'age', 'density']]","metadata":{"execution":{"iopub.status.busy":"2022-12-16T06:39:47.37319Z","iopub.execute_input":"2022-12-16T06:39:47.373522Z","iopub.status.idle":"2022-12-16T06:39:47.396759Z","shell.execute_reply.started":"2022-12-16T06:39:47.373491Z","shell.execute_reply":"2022-12-16T06:39:47.395674Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Machine 49 is the only machine is found which can be used to capture other view of images for cancer other than 'CC' and 'MLO'.
","metadata":{}},{"cell_type":"markdown","source":"### Biopsy and Machine Id","metadata":{}},{"cell_type":"code","source":"temp = df[df['biopsy']==1]\nfig, ax = plt.subplots(figsize=(18,6))\nsns.countplot(data=temp, x = 'machine_id', ax=ax)\nplt.title(\"Machine and Biopsy\")\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-12-16T06:39:47.398413Z","iopub.execute_input":"2022-12-16T06:39:47.398764Z","iopub.status.idle":"2022-12-16T06:39:47.637291Z","shell.execute_reply.started":"2022-12-16T06:39:47.398732Z","shell.execute_reply":"2022-12-16T06:39:47.635903Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"temp = {}\nfor machine in df['machine_id'].unique():\n temp[machine] = len(df[(df['machine_id']==machine) & (df['biopsy']==1)]) / len(df['machine_id']==machine) * 100\nfig, ax = plt.subplots(figsize=(18,6))\nsns.barplot(x = list(temp.keys()), y = list(temp.values()), ax=ax)\nax.bar_label(ax.containers[0])\nplt.title(\"Machine and Views\")\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-12-16T06:39:47.639306Z","iopub.execute_input":"2022-12-16T06:39:47.639801Z","iopub.status.idle":"2022-12-16T06:39:47.955654Z","shell.execute_reply.started":"2022-12-16T06:39:47.639753Z","shell.execute_reply":"2022-12-16T06:39:47.95438Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"No relation can be found from this plot but it can be observed patients who had taken image by using machine with id 49 had biopsy that other patients who had used other machines.
","metadata":{}},{"cell_type":"code","source":"temp = df[(df['biopsy']==1) & ((df['view']!='CC') & (df['view']!='MLO'))]\ntemp[['view', 'machine_id', 'age', 'density']]","metadata":{"execution":{"iopub.status.busy":"2022-12-16T06:39:47.958174Z","iopub.execute_input":"2022-12-16T06:39:47.959348Z","iopub.status.idle":"2022-12-16T06:39:47.98797Z","shell.execute_reply.started":"2022-12-16T06:39:47.959299Z","shell.execute_reply":"2022-12-16T06:39:47.986745Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"It can be found that for Biopsy machine_id 49 is used and except 'CC' and 'MLO' in some cases 'AT' and 'ML' is also used. In this cases patient's age are more than 50 in most of the cases and density is also not 'A'.
","metadata":{}},{"cell_type":"markdown","source":"### Invasive Cancer and Machine Id","metadata":{}},{"cell_type":"code","source":"temp = df[df['invasive']==1]\nfig, ax = plt.subplots(figsize=(18,6))\nsns.countplot(data=temp, x = 'machine_id', ax=ax)\nplt.title(\"Machine and Biopsy\")\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-12-16T06:39:47.989421Z","iopub.execute_input":"2022-12-16T06:39:47.990484Z","iopub.status.idle":"2022-12-16T06:39:48.244725Z","shell.execute_reply.started":"2022-12-16T06:39:47.990448Z","shell.execute_reply":"2022-12-16T06:39:48.24354Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"temp = {}\nfor machine in df['machine_id'].unique():\n temp[machine] = len(df[(df['machine_id']==machine) & (df['invasive']==1)]) / len(df['machine_id']==machine) * 100\nfig, ax = plt.subplots(figsize=(18,6))\nsns.barplot(x = list(temp.keys()), y = list(temp.values()), ax=ax)\nax.bar_label(ax.containers[0])\nplt.title(\"Machine and Ratio of Invasive Cancer\")\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-12-16T06:39:48.24623Z","iopub.execute_input":"2022-12-16T06:39:48.246671Z","iopub.status.idle":"2022-12-16T06:39:48.560463Z","shell.execute_reply.started":"2022-12-16T06:39:48.246629Z","shell.execute_reply":"2022-12-16T06:39:48.559286Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"temp = df[(df['invasive']==1) & (df['view']!='CC') & (df['view']!='MLO')]\ntemp[['view', 'machine_id', 'age', 'density']]","metadata":{"execution":{"iopub.status.busy":"2022-12-16T06:39:48.562481Z","iopub.execute_input":"2022-12-16T06:39:48.563558Z","iopub.status.idle":"2022-12-16T06:39:48.587109Z","shell.execute_reply.started":"2022-12-16T06:39:48.563507Z","shell.execute_reply":"2022-12-16T06:39:48.585767Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"By observing all the plots it can assumed that some machine is used more for imaging than others and percentage of identifying cancer is more than any other machines.
","metadata":{}},{"cell_type":"markdown","source":"### Machine and Density","metadata":{}},{"cell_type":"code","source":"temp = df[df['density']=='A']\nfig, ax = plt.subplots(figsize=(18,6))\nsns.countplot(data=temp, x = 'machine_id', hue='view', ax=ax)\nplt.title(\"Machine and Density-A\")\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-12-16T06:39:48.590914Z","iopub.execute_input":"2022-12-16T06:39:48.591256Z","iopub.status.idle":"2022-12-16T06:39:49.113087Z","shell.execute_reply.started":"2022-12-16T06:39:48.591227Z","shell.execute_reply":"2022-12-16T06:39:49.111911Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"temp = df[df['density']=='B']\nfig, ax = plt.subplots(figsize=(18,6))\nsns.countplot(data=temp, x = 'machine_id', hue='view', ax=ax)\nplt.title(\"Machine and Density-B\")\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-12-16T06:39:49.114619Z","iopub.execute_input":"2022-12-16T06:39:49.11577Z","iopub.status.idle":"2022-12-16T06:39:49.473036Z","shell.execute_reply.started":"2022-12-16T06:39:49.115726Z","shell.execute_reply":"2022-12-16T06:39:49.472174Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"temp = df[df['density']=='C']\nfig, ax = plt.subplots(figsize=(18,6))\nsns.countplot(data=temp, x = 'machine_id', hue='view', ax=ax)\nplt.title(\"Machine and Density-C\")\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-12-16T06:39:49.47411Z","iopub.execute_input":"2022-12-16T06:39:49.475149Z","iopub.status.idle":"2022-12-16T06:39:49.857846Z","shell.execute_reply.started":"2022-12-16T06:39:49.475102Z","shell.execute_reply":"2022-12-16T06:39:49.856549Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"temp = df[df['density']=='D']\nfig, ax = plt.subplots(figsize=(18,6))\nsns.countplot(data=temp, x = 'machine_id', hue='view', ax=ax)\nplt.title(\"Machine and Density-D\")\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-12-16T06:39:49.859759Z","iopub.execute_input":"2022-12-16T06:39:49.860652Z","iopub.status.idle":"2022-12-16T06:39:50.160052Z","shell.execute_reply.started":"2022-12-16T06:39:49.860598Z","shell.execute_reply":"2022-12-16T06:39:50.158906Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"No particular observation is found using these plots. Most of the density value is missing that is one of the biggest disadvantage for using density for any analysis.
","metadata":{}},{"cell_type":"markdown","source":"A mammography machine, also known as a mammogram machine or mammograph, is a specialized x-ray machine that is used to create detailed images of the breast tissue. Mammography machines use low-dose x-rays to create images of the breast tissue, which are then used by healthcare providers to assess the health of the breasts and detect any abnormalities. Mammography machines typically consist of a x-ray tube, a digital detector, and a compression device to hold the breast in place during the exam. Some mammography machines may also include additional features, such as computer-aided detection (CAD) software, which can help to identify potential abnormalities in the breast tissue.
\nIn this dataset, it can be found that some machines is used most of the times and the reason is unknown. It is also hard to say that machine_id defines the type of machine or not. Some machine also identifies more positive cases than others but there is no evidence presen that machine_id manipulates the target value somehow or not.
","metadata":{}},{"cell_type":"markdown","source":"##A difficult negative case in the context of mammography refers to a situation where a mammogram appears to be normal, but there is a suspicion of breast cancer based on other factors, such as the patient's symptoms or family history. In these cases, it may be difficult to determine whether or not the patient has breast cancer based on the mammogram alone. In these situations, additional imaging tests, such as ultrasound or MRI, may be needed to help confirm or rule out the presence of breast cancer. It is important for patients with a difficult negative case to talk to their doctor about their symptoms and concerns, and to follow their doctor's recommendations for further testing and monitoring.
","metadata":{}},{"cell_type":"markdown","source":"### Difficult Negative Case: Images and Patients","metadata":{}},{"cell_type":"code","source":"print(\"Total number of images of difficult negative case: \",len(df[df['difficult_negative_case']==True]))","metadata":{"execution":{"iopub.status.busy":"2022-12-16T06:39:50.161399Z","iopub.execute_input":"2022-12-16T06:39:50.161764Z","iopub.status.idle":"2022-12-16T06:39:50.171403Z","shell.execute_reply.started":"2022-12-16T06:39:50.161723Z","shell.execute_reply":"2022-12-16T06:39:50.17032Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(\"Total number of patients with difficult negative case: \",df[df['difficult_negative_case']==True]['patient_id'].nunique())","metadata":{"execution":{"iopub.status.busy":"2022-12-16T06:39:50.172926Z","iopub.execute_input":"2022-12-16T06:39:50.173273Z","iopub.status.idle":"2022-12-16T06:39:50.18445Z","shell.execute_reply.started":"2022-12-16T06:39:50.173242Z","shell.execute_reply":"2022-12-16T06:39:50.183144Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### View used for difficult negative case","metadata":{}},{"cell_type":"code","source":"temp = df[df['difficult_negative_case']==True]\nfig, ax = plt.subplots(figsize=(18,6))\nsns.countplot(data=temp, x = 'view', hue='density', ax=ax)\nplt.title(\"View and Difficult Negative Case\")\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-12-16T06:39:50.1862Z","iopub.execute_input":"2022-12-16T06:39:50.186637Z","iopub.status.idle":"2022-12-16T06:39:50.490572Z","shell.execute_reply.started":"2022-12-16T06:39:50.186592Z","shell.execute_reply":"2022-12-16T06:39:50.489397Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"temp = df[(df['difficult_negative_case']==True) & (df['view']!='CC') & (df['view']!='MLO')]\nfig, ax = plt.subplots(figsize=(18,6))\nsns.countplot(data=temp, x = 'view', hue='density', ax=ax)\nplt.title(\"View and Difficult Negative Case\")\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-12-16T06:39:50.491758Z","iopub.execute_input":"2022-12-16T06:39:50.49208Z","iopub.status.idle":"2022-12-16T06:39:50.731744Z","shell.execute_reply.started":"2022-12-16T06:39:50.492051Z","shell.execute_reply":"2022-12-16T06:39:50.730955Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"In some cases, a mammogram may appear normal even though there is an underlying abnormality present in the breast tissue. This is known as a false-negative result, and it can occur for a variety of reasons, such as dense breast tissue that obscures the abnormality or a small abnormality that is difficult to detect on a mammogram. In these situations, additional imaging tests may be necessary to accurately assess the breast tissue and detect any abnormalities.Some common views that may be used for mammography in difficult negative cases include lateral oblique views, breast implant displacement views, and MRI of the breast. These additional views can help the healthcare provider to more accurately assess the breast tissue and detect any potential abnormalities. The specific views that are used will depend on the individual patient and the recommendation of their healthcare provider.
","metadata":{}},{"cell_type":"markdown","source":"### Machine used for difficult negative case","metadata":{}},{"cell_type":"code","source":"temp = df[(df['difficult_negative_case']==True)]\nfig, ax = plt.subplots(figsize=(18,8))\nsns.countplot(data=temp, x = 'machine_id', hue='density', ax=ax)\nplt.title(\"Machine and Difficult Negative Case\")\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-12-16T06:39:50.732967Z","iopub.execute_input":"2022-12-16T06:39:50.733446Z","iopub.status.idle":"2022-12-16T06:39:51.117552Z","shell.execute_reply.started":"2022-12-16T06:39:50.733415Z","shell.execute_reply":"2022-12-16T06:39:51.116115Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"From the previous observations and this plot it can't be said that is any machine is specificallyb used for difficult negative cases.
","metadata":{}},{"cell_type":"markdown","source":"### Difficult Negative Case and BI-RADS","metadata":{}},{"cell_type":"code","source":"temp = df[(df['difficult_negative_case']==True)]\nfig, ax = plt.subplots(figsize=(18,8))\nsns.countplot(data=temp, x = 'BIRADS', hue='density', ax=ax)\nplt.title(\"BI-RADS and Difficult Negative Case\")\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-12-16T06:39:51.118723Z","iopub.execute_input":"2022-12-16T06:39:51.119466Z","iopub.status.idle":"2022-12-16T06:39:51.36243Z","shell.execute_reply.started":"2022-12-16T06:39:51.119427Z","shell.execute_reply":"2022-12-16T06:39:51.361246Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"All of difficult negative case has Bi-RADS score 0.0. In difficult negative cases of breast cancer, the most common BI-RADS score is probably a 0. A BI-RADS 0 score indicates that the mammogram is inconclusive and additional imaging is necessary to assess the breast tissue. However, it is important to note that the specific BI-RADS score assigned to a mammogram will depend on the individual patient and the findings of the exam.
","metadata":{}},{"cell_type":"markdown","source":"### Difficult Negative Case and Biopsy","metadata":{}},{"cell_type":"code","source":"temp = df[(df['difficult_negative_case']==True)]\nm = {\n 'biopsy': len(temp[temp['biopsy']==1]) / len(df[df['biopsy']==1]),\n 'not_biopsy': len(temp[temp['biopsy']==0]) / len(df[df['biopsy']==0])\n}\nfig, ax = plt.subplots(figsize=(18,6))\nsns.barplot(x = list(m.keys()), y = list(m.values()), ax=ax)\nplt.title(\"Relation between biopsy and difficult negative case\")\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-12-16T06:39:51.363943Z","iopub.execute_input":"2022-12-16T06:39:51.364789Z","iopub.status.idle":"2022-12-16T06:39:51.572778Z","shell.execute_reply.started":"2022-12-16T06:39:51.364752Z","shell.execute_reply":"2022-12-16T06:39:51.571556Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Rate of difficult negative case is higher in who had biopsy because it's common that when doctors aren't sure about the status of the cancer they use biopsy to confirm the state ofd the cancer.
","metadata":{}},{"cell_type":"markdown","source":"### Difficult Negative Case and Implant","metadata":{}},{"cell_type":"code","source":"temp = df[(df['difficult_negative_case']==True)]\nm = {\n 'implant': len(temp[temp['implant']==1]) / len(df[df['implant']==1]),\n 'not_implant': len(temp[temp['implant']==0]) / len(df[df['implant']==0])\n}\nfig, ax = plt.subplots(figsize=(18,6))\nsns.barplot(x = list(m.keys()), y = list(m.values()), ax=ax)\nplt.title(\"Relation between implant and difficult negative case\")\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-12-16T06:39:51.585152Z","iopub.execute_input":"2022-12-16T06:39:51.585573Z","iopub.status.idle":"2022-12-16T06:39:51.802646Z","shell.execute_reply.started":"2022-12-16T06:39:51.585538Z","shell.execute_reply":"2022-12-16T06:39:51.801426Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Here we can find that cases with difficult negative case is higher in who had implant because implant breasts is diificult to analyse using mammography than normal breasts.
","metadata":{}},{"cell_type":"markdown","source":"##Here it is found that biopsy is highly correlated with cancer but it will not be given and although it is highly correlated we should not use it as a feature to train the model because a patient should be identified that she has cancer or not depending on biopsy is performed on her or not.
","metadata":{}},{"cell_type":"markdown","source":"#\n
All the images doesn't have same width and height therfore these images should be resized to a particular height and width.
","metadata":{}},{"cell_type":"code","source":"plt.hist(aspect_ratios)\nplt.xlabel(\"Aspect Ratio\")\nplt.ylabel(\"Frequency\")\nplt.title(\"Aspect Ratio of DICOM Images\")\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-12-16T07:34:22.039205Z","iopub.execute_input":"2022-12-16T07:34:22.039964Z","iopub.status.idle":"2022-12-16T07:34:22.46472Z","shell.execute_reply.started":"2022-12-16T07:34:22.03992Z","shell.execute_reply":"2022-12-16T07:34:22.463489Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Beside resolution aspect ratio is also different for all the images.
","metadata":{}},{"cell_type":"markdown","source":"##As we can see it is very difficult to conclude that the image has cancer or not but one thing to observe that the images has a big empty space which should be removed for proper classification.
","metadata":{}},{"cell_type":"markdown","source":"### Images with Invasive Cancer and Cancer which is not invasive","metadata":{}},{"cell_type":"code","source":"print(\"Images of invasive cancer: \")\ntemp = df[df['invasive']==1]\ntemp = temp.sample(4, replace=False)\nplot_images(temp)","metadata":{"execution":{"iopub.status.busy":"2022-12-16T07:34:33.682974Z","iopub.execute_input":"2022-12-16T07:34:33.683447Z","iopub.status.idle":"2022-12-16T07:34:40.390085Z","shell.execute_reply.started":"2022-12-16T07:34:33.683404Z","shell.execute_reply":"2022-12-16T07:34:40.388573Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(\"Images with cancer but not invasive: \")\ntemp = df[(df['cancer']==1) & (df['invasive']==0)]\ntemp = temp.sample(4, replace=False)\nplot_images(temp)","metadata":{"execution":{"iopub.status.busy":"2022-12-16T07:34:40.39164Z","iopub.execute_input":"2022-12-16T07:34:40.392058Z","iopub.status.idle":"2022-12-16T07:34:45.5842Z","shell.execute_reply.started":"2022-12-16T07:34:40.39202Z","shell.execute_reply":"2022-12-16T07:34:45.582674Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Images with implant and without implant and have cancer","metadata":{}},{"cell_type":"code","source":"print(\"Images with implant and cancer: \")\ntemp = df[(df['cancer']==1) & (df['implant']==1)]\ntemp = temp.sample(4, replace=False)\nplot_images(temp)","metadata":{"execution":{"iopub.status.busy":"2022-12-16T07:34:45.585789Z","iopub.execute_input":"2022-12-16T07:34:45.586307Z","iopub.status.idle":"2022-12-16T07:34:51.136579Z","shell.execute_reply.started":"2022-12-16T07:34:45.58626Z","shell.execute_reply":"2022-12-16T07:34:51.135284Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(\"Images without implant and cancer: \")\ntemp = df[(df['cancer']==0) & (df['implant']==0)]\ntemp = temp.sample(4, replace=False)\nplot_images(temp)","metadata":{"execution":{"iopub.status.busy":"2022-12-16T07:34:51.138537Z","iopub.execute_input":"2022-12-16T07:34:51.139052Z","iopub.status.idle":"2022-12-16T07:34:59.581508Z","shell.execute_reply.started":"2022-12-16T07:34:51.139004Z","shell.execute_reply":"2022-12-16T07:34:59.580311Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(\"Images with implant and don't have cancer: \")\ntemp = df[(df['cancer']==0) & (df['implant']==1)]\ntemp = temp.sample(4, replace=False)\nplot_images(temp)","metadata":{"execution":{"iopub.status.busy":"2022-12-16T07:34:59.583356Z","iopub.execute_input":"2022-12-16T07:34:59.584482Z","iopub.status.idle":"2022-12-16T07:35:04.480032Z","shell.execute_reply.started":"2022-12-16T07:34:59.584432Z","shell.execute_reply":"2022-12-16T07:35:04.479076Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"It can be noted here that images with implant is hard to analyze which is clear from image as images with implant have a big dark spot which makes the analysis more difficult.
","metadata":{}},{"cell_type":"markdown","source":"### Images of patients with and without biopsy","metadata":{}},{"cell_type":"code","source":"print(\"Images with biopsy: \")\ntemp = df[df['biopsy']==1]\ntemp = temp.sample(4, replace=False)\nplot_images(temp)","metadata":{"execution":{"iopub.status.busy":"2022-12-16T07:35:04.481557Z","iopub.execute_input":"2022-12-16T07:35:04.481945Z","iopub.status.idle":"2022-12-16T07:35:11.224104Z","shell.execute_reply.started":"2022-12-16T07:35:04.48191Z","shell.execute_reply":"2022-12-16T07:35:11.22274Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(\"Images without biopsy: \")\ntemp = df[df['biopsy']==0]\ntemp = temp.sample(4, replace=False)\nplot_images(temp)","metadata":{"execution":{"iopub.status.busy":"2022-12-16T07:35:11.225759Z","iopub.execute_input":"2022-12-16T07:35:11.22614Z","iopub.status.idle":"2022-12-16T07:35:20.531667Z","shell.execute_reply.started":"2022-12-16T07:35:11.226106Z","shell.execute_reply":"2022-12-16T07:35:20.530409Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"It hard to classify that someone had biopsy or not from these images for a person who doesn't belong from medical domain.
","metadata":{}},{"cell_type":"markdown","source":"### View: CC","metadata":{}},{"cell_type":"code","source":"print(\"Images with 'CC' view and have cancer: \")\ntemp = df[(df['view']=='CC') & (df['cancer']==1)]\ntemp = temp.sample(4, replace=False)\nplot_images(temp)","metadata":{"execution":{"iopub.status.busy":"2022-12-16T07:35:20.533315Z","iopub.execute_input":"2022-12-16T07:35:20.533684Z","iopub.status.idle":"2022-12-16T07:35:28.028424Z","shell.execute_reply.started":"2022-12-16T07:35:20.53365Z","shell.execute_reply":"2022-12-16T07:35:28.027541Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(\"Images with 'CC' view and without cancer: \")\ntemp = df[(df['view']=='CC') & (df['cancer']==0)]\ntemp = temp.sample(4, replace=False)\nplot_images(temp)","metadata":{"execution":{"iopub.status.busy":"2022-12-16T07:35:28.029928Z","iopub.execute_input":"2022-12-16T07:35:28.030486Z","iopub.status.idle":"2022-12-16T07:35:33.878682Z","shell.execute_reply.started":"2022-12-16T07:35:28.03045Z","shell.execute_reply":"2022-12-16T07:35:33.877635Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### View: MLO","metadata":{}},{"cell_type":"code","source":"print(\"Images with 'MLO' view and have cancer: \")\ntemp = df[(df['view']=='MLO') & (df['cancer']==1)]\ntemp = temp.sample(4, replace=False)\nplot_images(temp)","metadata":{"execution":{"iopub.status.busy":"2022-12-16T07:35:33.88011Z","iopub.execute_input":"2022-12-16T07:35:33.880446Z","iopub.status.idle":"2022-12-16T07:35:42.610712Z","shell.execute_reply.started":"2022-12-16T07:35:33.880416Z","shell.execute_reply":"2022-12-16T07:35:42.609807Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(\"Images with 'MLO' view and without cancer: \")\ntemp = df[(df['view']=='MLO') & (df['cancer']==0)]\ntemp = temp.sample(4, replace=False)\nplot_images(temp)","metadata":{"execution":{"iopub.status.busy":"2022-12-16T07:35:42.612147Z","iopub.execute_input":"2022-12-16T07:35:42.61287Z","iopub.status.idle":"2022-12-16T07:35:49.780296Z","shell.execute_reply.started":"2022-12-16T07:35:42.612812Z","shell.execute_reply":"2022-12-16T07:35:49.779122Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Irrespective of view it's very difficult to identify from a image that it has cancer or not if he/she doestn't belong from a medical background.
","metadata":{}},{"cell_type":"markdown","source":"### Let's visulaize some images from difficult negative case","metadata":{}},{"cell_type":"code","source":"print(\"Images of difficult negative case: \")\ntemp = df[df['difficult_negative_case']==True]\ntemp = temp.sample(16, replace=False)\nplot_images(temp,4,4)","metadata":{"execution":{"iopub.status.busy":"2022-12-16T07:35:49.781857Z","iopub.execute_input":"2022-12-16T07:35:49.782202Z","iopub.status.idle":"2022-12-16T07:36:10.100809Z","shell.execute_reply.started":"2022-12-16T07:35:49.782162Z","shell.execute_reply":"2022-12-16T07:36:10.099202Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"##An average image is a representation of the typical features or characteristics of a group of images. It is created by combining multiple individual images and taking the average value of each pixel across all of the images. This can be useful for a variety of purposes, such as removing noise or highlighting common features in a dataset of images.
\nContrast refers to the difference in intensity between the light and dark areas of an image. In the context of average images, contrast can refer to the difference in intensity between the average image and the individual images that were used to create it. For example, if the individual images have a wide range of intensities, the average image may have lower contrast compared to the original images. On the other hand, if the individual images have similar intensities, the average image may have higher contrast compared to the original images. The contrast of an average image can be adjusted by changing the way the individual images are combined to create the average.
","metadata":{}},{"cell_type":"markdown","source":"### Images of different views","metadata":{}},{"cell_type":"code","source":"def filenames_from_dataframe(df_sample: pd.DataFrame):\n filenames = []\n for index, row in df_sample.iterrows():\n filenames.append(os.path.join(source_path, str(row['patient_id']), str(row['image_id'])+'.dcm'))\n return filenames","metadata":{"execution":{"iopub.status.busy":"2022-12-16T07:36:10.103165Z","iopub.execute_input":"2022-12-16T07:36:10.103655Z","iopub.status.idle":"2022-12-16T07:36:10.112649Z","shell.execute_reply.started":"2022-12-16T07:36:10.103611Z","shell.execute_reply":"2022-12-16T07:36:10.111374Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def resize_image(dcm_image, new_height=30, new_width=30):\n image_data = dcm_image.pixel_array\n jpeg_image = Image.fromarray(image_data)\n jpeg_image = jpeg_image.resize((new_width, new_height))\n return jpeg_image","metadata":{"execution":{"iopub.status.busy":"2022-12-16T07:36:10.114503Z","iopub.execute_input":"2022-12-16T07:36:10.114907Z","iopub.status.idle":"2022-12-16T07:36:10.126682Z","shell.execute_reply.started":"2022-12-16T07:36:10.11487Z","shell.execute_reply":"2022-12-16T07:36:10.125184Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def average_image(df_sample: pd.DataFrame, title: str):\n # Get filenames\n dcm_filenames = filenames_from_dataframe(df_sample)\n # Load the DICOM images\n dcm_images = [pydicom.dcmread(f) for f in dcm_filenames]\n # Resize all images\n jpeg_images = [resize_image(img) for img in dcm_images]\n # Convert the image to 3D array\n image_data = np.stack([np.array(im) for im in jpeg_images])\n # Calculate the average image\n average_image = np.mean(image_data, axis=0)\n plt.imshow(average_image, cmap='gray')\n plt.axis('off')\n plt.title(title)\n return average_image","metadata":{"execution":{"iopub.status.busy":"2022-12-16T07:36:10.128645Z","iopub.execute_input":"2022-12-16T07:36:10.129521Z","iopub.status.idle":"2022-12-16T07:36:10.140655Z","shell.execute_reply.started":"2022-12-16T07:36:10.129467Z","shell.execute_reply":"2022-12-16T07:36:10.139315Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def difference_images(image1, image2, title):\n contrast_mean = image1 - image2\n plt.imshow(contrast_mean, cmap='bwr')\n plt.title(title)\n plt.axis('off')\n return contrast_mean","metadata":{"execution":{"iopub.status.busy":"2022-12-16T07:36:10.142213Z","iopub.execute_input":"2022-12-16T07:36:10.143451Z","iopub.status.idle":"2022-12-16T07:36:10.151883Z","shell.execute_reply.started":"2022-12-16T07:36:10.143401Z","shell.execute_reply":"2022-12-16T07:36:10.150756Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(\"Average image of CC view: \")\nfig, ax = plt.subplots(figsize=(20,10))\nfig.add_subplot(1, 2, 1)\n_ = average_image(df[(df['view']=='CC') & (df['laterality']=='L')].sample(200), 'Laterality: Left')\nfig.add_subplot(1, 2, 2)\n_ = average_image(df[(df['view']=='CC') & (df['laterality']=='R')].sample(200), 'Laterality: Right')","metadata":{"execution":{"iopub.status.busy":"2022-12-16T07:37:45.779815Z","iopub.execute_input":"2022-12-16T07:37:45.780322Z","iopub.status.idle":"2022-12-16T07:42:33.020527Z","shell.execute_reply.started":"2022-12-16T07:37:45.780281Z","shell.execute_reply":"2022-12-16T07:42:33.019194Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(\"Average image of MLO view: \")\nfig, ax = plt.subplots(figsize=(20,10))\nfig.add_subplot(1, 2, 1)\n_ = average_image(df[(df['view']=='MLO') & (df['laterality']=='L')].sample(200), 'Laterality: Left')\nfig.add_subplot(1, 2, 2)\n_ = average_image(df[(df['view']=='MLO') & (df['laterality']=='R')].sample(200), 'Laterality: Right')","metadata":{"execution":{"iopub.status.busy":"2022-12-16T07:42:33.022612Z","iopub.execute_input":"2022-12-16T07:42:33.022992Z","iopub.status.idle":"2022-12-16T07:47:38.555318Z","shell.execute_reply.started":"2022-12-16T07:42:33.022958Z","shell.execute_reply":"2022-12-16T07:47:38.553894Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Cancer and Non-cancer Patients","metadata":{}},{"cell_type":"code","source":"print(\"Average image of cacer patients: \")\nfig, ax = plt.subplots(figsize=(20,10))\nfig.add_subplot(1, 2, 1)\ncancer_left = average_image(df[(df['cancer']==1) & (df['laterality']=='L')].sample(200), 'Laterality: Left')\nfig.add_subplot(1, 2, 2)\ncancer_right = average_image(df[(df['cancer']==1)& (df['laterality']=='R')].sample(200), 'Laterality: Right')","metadata":{"execution":{"iopub.status.busy":"2022-12-16T07:47:38.55714Z","iopub.execute_input":"2022-12-16T07:47:38.557509Z","iopub.status.idle":"2022-12-16T07:52:11.329849Z","shell.execute_reply.started":"2022-12-16T07:47:38.557476Z","shell.execute_reply":"2022-12-16T07:52:11.328428Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(\"Average image of non-cancer patients: \")\nfig, ax = plt.subplots(figsize=(20,10))\nfig.add_subplot(1, 2, 1)\nnoncancer_left = average_image(df[(df['cancer']==0) & (df['laterality']=='L')].sample(200), 'Laterality: Left')\nfig.add_subplot(1, 2, 2)\nnoncancer_right = average_image(df[(df['cancer']==0) & (df['laterality']=='R')].sample(200), 'Laterality: Right')","metadata":{"execution":{"iopub.status.busy":"2022-12-16T07:52:11.332355Z","iopub.execute_input":"2022-12-16T07:52:11.332727Z","iopub.status.idle":"2022-12-16T07:57:06.007532Z","shell.execute_reply.started":"2022-12-16T07:52:11.332695Z","shell.execute_reply":"2022-12-16T07:57:06.006136Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(\"Difference of image of cancer and non-cancer patients: \")\nfig, ax = plt.subplots(figsize=(20,10))\nfig.add_subplot(1, 2, 1)\n_ = difference_images(cancer_left, noncancer_left, \"Laterality: Left\")\nfig.add_subplot(1, 2, 2)\n_ = difference_images(cancer_right, noncancer_right, \"Laterality: Right\")","metadata":{"execution":{"iopub.status.busy":"2022-12-16T07:57:06.00951Z","iopub.execute_input":"2022-12-16T07:57:06.009917Z","iopub.status.idle":"2022-12-16T07:57:06.841518Z","shell.execute_reply.started":"2022-12-16T07:57:06.009874Z","shell.execute_reply":"2022-12-16T07:57:06.840056Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"No obserable pattern is found from the average image and difference between average images.
","metadata":{}},{"cell_type":"markdown","source":"#