Deep Regression Model의 Reliability 분석

Model safety (reliability) 

1) Model calibration : confidence output 값이 실제 확률과 일치하도록 교정

    -> evaluation : Reliabilbity diagrams, ECE(Expected calibration error)

    -> Post-hoc calibration : temperature scaling / 

    -> Model regularization : cross-entropy / label smoothing / focal loss ...

2) Uncertainty estimation

3) Out-of-distribution detection : 학습하지 않은 class의 데이터셋

4) Adversarial example detection : 의도적으로 오분류를 야기하도록 최적화된 noise

 

 

Classifier의 모델 예측의 reliability diagram을 그리거나 confidence score를 계산할 수 있듯이 (overfitting 여부 확인 등의 목적으로)

Regressor의 모델 예측에 대한 신뢰구간을 결정할 수 있을지 궁금해서 찾아보는 중..

 

 

Methodology

1) The Naive Method

• Train the model on the training set
• Calculate the residuals of the predictions on the training set
• Select the (1 — alpha) quantile of the distribution of the residuals
• Sum and subtract each prediction from this quantile to get the limits of the confidence interval

2) The Jackknife Method

3) The Jackknife+ Method

4) The CV+ Method

 

 

 

 

 

http://dsba.korea.ac.kr/seminar/?mod=document&uid=2638 

[Paper Review] Rethinking Calibration of Deep Neural Networks: Do Not Afraid of Overconfidence

 

https://towardsdatascience.com/generating-confidence-intervals-for-regression-models-2dd60026fbce

Generating Confidence Intervals for Regression Models