![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F9335656%2F6744945526c9df2e39e9fdaf4ab59010%2FCapture.PNG?generation=1669910536161507&alt=media\")
Hi Kagglers! I found it a bit difficult to grasp the evaluation metric as cited in the paper under the evaluation tab. So I decided to probe further with some simplified examples. I am just sharing my understanding here to gauge the findings & help others, though most of you guys might already understood it clearly. Do let me know if I have gone wrong somewhere.
\nAssumptions/Simplifications:
\nExample Dataset, Model Confidence & Predictions:
\n
Building the probabilistic confusion matrix (pCM):
\nThe probabilistic confusion matrix is calculated from the formula:
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F9335656%2Fef0a11bbe6d46be929ff515a97c29121%2FCapture1.PNG?generation=1669911539957936&alt=media\")
where, jref is the true label & jhyp is the label predicted by model.
\nWe will begin with datasets whose True label is 0 first -- i.e jref = 0:
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F9335656%2F0eb96f3f009e9878e4499476c116cdb9%2FCapture2.PNG?generation=1669911518835995&alt=media\")
Out of this filtered dataset, we can calculate** pCM(jref=0 , jhyp=0)** as:
\nM(x1, 0) + M(x2, 0),+M(x4, 0)+M(x8, 0)+M(x9, 0)= 0.735+0.381+0.689+0.423+0.821 = 3.049
\nand pCM(jref=0 , jhyp=1) as:
\nM(x1, 1) + M(x2, 1),+M(x4, 1)+M(x8, 1)+M(x9, 1)= 0.265+0.619+0.311+0.577+0.179 = 1.951 (alternatively 5 - 3.049)
\nSimilarly, pCM(jref=1 , jhyp=0) & pCM(jref=1 , jhyp=0) can be calculated to be 2.585 & 2.415 respectively.
\nThen probabilistic confusion matrix can be populated as below:
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F9335656%2F62d21a2ddfe7f161fafcf61b0db4af74%2FCapture3.PNG?generation=1669913937698645&alt=media\")
pPrecision, pRecall & pF1 score can then be deduced.
\nPlease forgive the naive approach or erros. Once again request all your valuable inputs. Best wishes for the competition!!!
", "messages": [ { "id": 2051877, "postDate": "2022-12-01T17:06:39.450Z", "content": "Hi Kagglers! I found it a bit difficult to grasp the evaluation metric as cited in the paper under the evaluation tab. So I decided to probe further with some simplified examples. I am just sharing my understanding here to gauge the findings & help others, though most of you guys might already understood it clearly. Do let me know if I have gone wrong somewhere.
\nAssumptions/Simplifications:
\nExample Dataset, Model Confidence & Predictions:
\n
Building the probabilistic confusion matrix (pCM):
\nThe probabilistic confusion matrix is calculated from the formula:
\n
where, jref is the true label & jhyp is the label predicted by model.
\nWe will begin with datasets whose True label is 0 first -- i.e jref = 0:
\n
Out of this filtered dataset, we can calculate** pCM(jref=0 , jhyp=0)** as:
\nM(x1, 0) + M(x2, 0),+M(x4, 0)+M(x8, 0)+M(x9, 0)= 0.735+0.381+0.689+0.423+0.821 = 3.049
\nand pCM(jref=0 , jhyp=1) as:
\nM(x1, 1) + M(x2, 1),+M(x4, 1)+M(x8, 1)+M(x9, 1)= 0.265+0.619+0.311+0.577+0.179 = 1.951 (alternatively 5 - 3.049)
\nSimilarly, pCM(jref=1 , jhyp=0) & pCM(jref=1 , jhyp=0) can be calculated to be 2.585 & 2.415 respectively.
\nThen probabilistic confusion matrix can be populated as below:
\n
pPrecision, pRecall & pF1 score can then be deduced.
\nPlease forgive the naive approach or erros. Once again request all your valuable inputs. Best wishes for the competition!!!
", "rawMarkdown": "Hi Kagglers! I found it a bit difficult to grasp the evaluation metric as cited in the paper under the evaluation tab. So I decided to probe further with some simplified examples. I am just sharing my understanding here to gauge the findings & help others, though most of you guys might already understood it clearly. Do let me know if I have gone wrong somewhere.\n\n**Assumptions/Simplifications:**\n1. We are dealing with a binary classification problem (which we really are in this competition).\n2. An imaginary dataset of 10 examples with their true labels.\n3. An imaginary model with a sigmoid head which outputs the probability/confidence(p) of a dataset example belonging to class 0 or probability (1-p) belonging to class 1. This can also be represented as model confidence score for a particular dataset (say x1) as M(x1, 0) for class 0 i.e (p) and M(x1, 1) for class 1 i.e (1-p).\n\n**Example Dataset, Model Confidence & Predictions:**\n\n\n\n**Building the probabilistic confusion matrix (pCM):**\n\nThe probabilistic confusion matrix is calculated from the formula:\n\n\n\nwhere, jref is the true label & jhyp is the label predicted by model.\n\nWe will begin with datasets whose True label is 0 first -- i.e jref = 0:\n\n\n\nOut of this filtered dataset, we can calculate** pCM(jref=0 , jhyp=0)** as:\n\nM(x1, 0) + M(x2, 0),+M(x4, 0)+M(x8, 0)+M(x9, 0)= 0.735+0.381+0.689+0.423+0.821 = **3.049**\n\nand **pCM(jref=0 , jhyp=1)** as:\n\nM(x1, 1) + M(x2, 1),+M(x4, 1)+M(x8, 1)+M(x9, 1)= 0.265+0.619+0.311+0.577+0.179 = **1.951** (alternatively 5 - 3.049)\n\nSimilarly, **pCM(jref=1 , jhyp=0) & pCM(jref=1 , jhyp=0)** can be calculated to be **2.585** & **2.415** respectively.\n\nThen probabilistic confusion matrix can be populated as below:\n\n\n\npPrecision, pRecall & pF1 score can then be deduced.\n\nPlease forgive the naive approach or erros. Once again request all your valuable inputs. Best wishes for the competition!!!\n", "votes": 9 }, { "id": 2113685, "postDate": "2023-01-24T12:22:37.137Z", "content": "Though this topic has already 2 months: worthy contribution explaining:
\nprobabilistic confusion matrix (pPrecision, pRecall & pF1) competition metric.
Though this topic has already 2 months: worthy contribution explaining:
\nprobabilistic confusion matrix (pPrecision, pRecall & pF1) competition metric.