{
  "id": 112646,
  "title": "Shape of test dataframe is different from prediction dataframe",
  "url": "/competitions/rsna-intracranial-hemorrhage-detection/discussion/112646",
  "author_name": "Shubam Sachdeva",
  "post_date": "2019-10-14T14:20:19.471000",
  "votes": 0,
  "comment_count": 1,
  "views": 0,
  "content": "<p>I am trying two different approaches.\n1. First one is to create simple CNN using Conv2d and maxpooling layers and using binary_crossentropy loss\n2. Second one is to use Resnet or Inceptionv3 </p>\n\n<p><code>\nbuild_model = tf.keras.applications.inception_v3.InceptionV3(include_top=False, weights='imagenet', input_shape=(q_size, q_size, img_channel), backend=tf.keras.backend, layers=tf.keras.layers, models=tf.keras.models, utils=tf.keras.utils)\nx = tf.keras.layers.GlobalAveragePooling2D()(build_model.output)\nx = tf.keras.layers.Dropout(0.2)(x)\nx = tf.keras.layers.Dense(128, activation='relu')(x)\nx = tf.keras.layers.Dropout(0.2)(x)\nout = tf.keras.layers.Dense(6, activation='sigmoid')(x)\nmodel = tf.keras.models.Model(inputs=build_model.input, outputs=out)\n</code></p>\n\n<p><code>model.compile(optimizer = tf.keras.optimizers.Adam(lr=1e-5),\n              loss='binary_crossentropy',\n              metrics=['binary_accuracy'])</code></p>\n\n<p>First one runs smoothly with no warnings or errors but when i use resnet or inceptionv3\ni am facing below warning while <strong>predicting the test data</strong> :</p>\n\n<p><strong>/opt/conda/lib/python3.6/site-packages/ipykernellauncher.py:53: RuntimeWarning: divide by zero encountered in truedivide\n/opt/conda/lib/python3.6/site-packages/ipykernellauncher.py:53: RuntimeWarning: invalid value encountered in truedivide</strong></p>\n\n<p>The shape of test dataframe and predictions dataframe is different. I am thinking it is because of above warning. I have no idea how to fix it. Any help would be appreciated!</p>",
  "messages": [
    {
      "id": 649321,
      "postDate": "2019-10-15T07:44:06.943Z",
      "content": "<p>Some more information, about what you are doing with the pandas DataFrames and how you load data etc. would be helpful. By the look of it there doesn't seem anything wrong with the model definition (and if it trains it's probably fine). </p>\n\n<p>Assuming that you mean the first dimension of the DataFrames (i.e. the number of samples), and the numbers are not hugely different. I have at least one idea:</p>\n\n<p>Depending on how you load the data, you might be loading the test-data in batches. If the remainder of test_data_length / batch_size is not 0, you will probably have less predictions in the end. A Batchsize of 1 could help here. </p>",
      "rawMarkdown": "Some more information, about what you are doing with the pandas DataFrames and how you load data etc. would be helpful. By the look of it there doesn't seem anything wrong with the model definition (and if it trains it's probably fine). \n\nAssuming that you mean the first dimension of the DataFrames (i.e. the number of samples), and the numbers are not hugely different. I have at least one idea:\n\nDepending on how you load the data, you might be loading the test-data in batches. If the remainder of test_data_length / batch_size is not 0, you will probably have less predictions in the end. A Batchsize of 1 could help here. "
    },
    {
      "id": 648707,
      "postDate": "2019-10-14T14:20:19.473Z",
      "content": "<p>I am trying two different approaches.\n1. First one is to create simple CNN using Conv2d and maxpooling layers and using binary_crossentropy loss\n2. Second one is to use Resnet or Inceptionv3 </p>\n\n<p><code>\nbuild_model = tf.keras.applications.inception_v3.InceptionV3(include_top=False, weights='imagenet', input_shape=(q_size, q_size, img_channel), backend=tf.keras.backend, layers=tf.keras.layers, models=tf.keras.models, utils=tf.keras.utils)\nx = tf.keras.layers.GlobalAveragePooling2D()(build_model.output)\nx = tf.keras.layers.Dropout(0.2)(x)\nx = tf.keras.layers.Dense(128, activation='relu')(x)\nx = tf.keras.layers.Dropout(0.2)(x)\nout = tf.keras.layers.Dense(6, activation='sigmoid')(x)\nmodel = tf.keras.models.Model(inputs=build_model.input, outputs=out)\n</code></p>\n\n<p><code>model.compile(optimizer = tf.keras.optimizers.Adam(lr=1e-5),\n              loss='binary_crossentropy',\n              metrics=['binary_accuracy'])</code></p>\n\n<p>First one runs smoothly with no warnings or errors but when i use resnet or inceptionv3\ni am facing below warning while <strong>predicting the test data</strong> :</p>\n\n<p><strong>/opt/conda/lib/python3.6/site-packages/ipykernellauncher.py:53: RuntimeWarning: divide by zero encountered in truedivide\n/opt/conda/lib/python3.6/site-packages/ipykernellauncher.py:53: RuntimeWarning: invalid value encountered in truedivide</strong></p>\n\n<p>The shape of test dataframe and predictions dataframe is different. I am thinking it is because of above warning. I have no idea how to fix it. Any help would be appreciated!</p>",
      "rawMarkdown": "I am trying two different approaches.\n1. First one is to create simple CNN using Conv2d and maxpooling layers and using binary_crossentropy loss\n2. Second one is to use Resnet or Inceptionv3 \n\n```\nbuild_model = tf.keras.applications.inception_v3.InceptionV3(include_top=False, weights='imagenet', input_shape=(q_size, q_size, img_channel), backend=tf.keras.backend, layers=tf.keras.layers, models=tf.keras.models, utils=tf.keras.utils)\nx = tf.keras.layers.GlobalAveragePooling2D()(build_model.output)\nx = tf.keras.layers.Dropout(0.2)(x)\nx = tf.keras.layers.Dense(128, activation='relu')(x)\nx = tf.keras.layers.Dropout(0.2)(x)\nout = tf.keras.layers.Dense(6, activation='sigmoid')(x)\nmodel = tf.keras.models.Model(inputs=build_model.input, outputs=out)\n```\n\n`model.compile(optimizer = tf.keras.optimizers.Adam(lr=1e-5),\n              loss='binary_crossentropy',\n              metrics=['binary_accuracy'])`\n\n\nFirst one runs smoothly with no warnings or errors but when i use resnet or inceptionv3\ni am facing below warning while **predicting the test data** :\n\n**/opt/conda/lib/python3.6/site-packages/ipykernellauncher.py:53: RuntimeWarning: divide by zero encountered in truedivide\n/opt/conda/lib/python3.6/site-packages/ipykernellauncher.py:53: RuntimeWarning: invalid value encountered in truedivide**\n\nThe shape of test dataframe and predictions dataframe is different. I am thinking it is because of above warning. I have no idea how to fix it. Any help would be appreciated!"
    }
  ],
  "comments": [
    {
      "id": 649321,
      "author_name": "srs",
      "author_url": "",
      "post_date": "2019-10-15T07:44:06.943000",
      "content": "<p>Some more information, about what you are doing with the pandas DataFrames and how you load data etc. would be helpful. By the look of it there doesn't seem anything wrong with the model definition (and if it trains it's probably fine). </p>\n\n<p>Assuming that you mean the first dimension of the DataFrames (i.e. the number of samples), and the numbers are not hugely different. I have at least one idea:</p>\n\n<p>Depending on how you load the data, you might be loading the test-data in batches. If the remainder of test_data_length / batch_size is not 0, you will probably have less predictions in the end. A Batchsize of 1 could help here. </p>",
      "votes": 0,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "649321": "Some more information, about what you are doing with the pandas DataFrames and how you load data etc. would be helpful. By the look of it there doesn't seem anything wrong with the model definition (and if it trains it's probably fine). \n\nAssuming that you mean the first dimension of the DataFrames (i.e. the number of samples), and the numbers are not hugely different. I have at least one idea:\n\nDepending on how you load the data, you might be loading the test-data in batches. If the remainder of test_data_length / batch_size is not 0, you will probably have less predictions in the end. A Batchsize of 1 could help here. ",
    "648707": "I am trying two different approaches.\n1. First one is to create simple CNN using Conv2d and maxpooling layers and using binary_crossentropy loss\n2. Second one is to use Resnet or Inceptionv3 \n\n```\nbuild_model = tf.keras.applications.inception_v3.InceptionV3(include_top=False, weights='imagenet', input_shape=(q_size, q_size, img_channel), backend=tf.keras.backend, layers=tf.keras.layers, models=tf.keras.models, utils=tf.keras.utils)\nx = tf.keras.layers.GlobalAveragePooling2D()(build_model.output)\nx = tf.keras.layers.Dropout(0.2)(x)\nx = tf.keras.layers.Dense(128, activation='relu')(x)\nx = tf.keras.layers.Dropout(0.2)(x)\nout = tf.keras.layers.Dense(6, activation='sigmoid')(x)\nmodel = tf.keras.models.Model(inputs=build_model.input, outputs=out)\n```\n\n`model.compile(optimizer = tf.keras.optimizers.Adam(lr=1e-5),\n              loss='binary_crossentropy',\n              metrics=['binary_accuracy'])`\n\n\nFirst one runs smoothly with no warnings or errors but when i use resnet or inceptionv3\ni am facing below warning while **predicting the test data** :\n\n**/opt/conda/lib/python3.6/site-packages/ipykernellauncher.py:53: RuntimeWarning: divide by zero encountered in truedivide\n/opt/conda/lib/python3.6/site-packages/ipykernellauncher.py:53: RuntimeWarning: invalid value encountered in truedivide**\n\nThe shape of test dataframe and predictions dataframe is different. I am thinking it is because of above warning. I have no idea how to fix it. Any help would be appreciated!"
  }
}