42. Rahim Entezari - TU-Graz & CSH - Improving generalization in parameter and data space

# Summary

Did you ever had the experience that you where training a network, investing a lot of time in finding the right hyper parameters and testing different initializations to push that validation accuracy over certain threshold? Only to then find out when putting the model into production, that it significantly underperforms?

If you did, then you experienced one common problem with deep neural networks. The performance gap between in and out-of distribution generalization.

Today on the show PhD Rahim Entezari is giving us a wonderful tour through his PhD journey investigating ways to understand and improve generalization performance of deep neural networks.

Rahim will explain how one can improve generalization by different methods in data or in parameter space.

We will discuss how using different forms of sparsity, or the efficient creation of deep ensemble networks by permutation of network configurations can improve generalization from a parameter space perspective.

Or, from a data perspective where we discuss how data quality and data diversity effects the generalization performance of modern deep neural networks.

I hope you enjoy this interview, full of interesting concepts and ideas from deep learning theory.

# TOC

00:00:00 Introduction

00:02:18 Background Knowledge

00:06:56 Guest Introduction

00:12:35 Generalization from a Data or Parameter Perspective

00:16:21 In and out of distribution Generalization

00:20:30 Structured and Unstructured Sparsity

00:29:55 Generalization in Parameter space

00:46:56 Generalization in Data space

# Sponsors

Quantics: Supply Chain Planning for the new normal - the never normal - https://quantics.io/

Belichberg GmbH: We do digital transformations as your innovation partner - https://belichberg.com/

# References

Rahim Entezari - https://www.linkedin.com/in/rahimentezari/