Skip to main content Link Menu Expand (external link) Document Search Copy Copied
I2 Intro Neuro/AI

Unit 2: The (deep learning) Basics

Hello and welcome to the Basics section of the I2 megadoc! We will start by throwing a few videos at you that we believe give incredibly intuitive explanations of one of the foundational building blocks of modern Deep Learning.

Task: Watch and understand the following videos. We recommend taking notes and being able to answer the synthesis questions provided below. Send your I2 teacher/mentor/overlord the answers to the questions over Discord.

Video 1: But what is a neural network? | Chapter 1, Deep learning (20 min)

Synthesis Questions:

  • What is a neuron (in terms of Neural Networks) and what does its "activation" represent?
    • Bonus: Research and consider the correlation between a biological neuron and an artificial neuron. How are they similar/different?
  • What is a network layer? How is it connected to other network layers?
  • How is a picture of a digit decomposed into a network layer?
  • What does the final layer of a neural network represent?
  • What are weights? What are biases? Can you describe in English how information is passed from one layer to the next?
  • A neural network IS/IS NOT just a very highly parameterized function (Choose one)
  • What is the purpose of the sigmoid function?

Great job! Now onto video two. Remember that all questions should be answered thoroughly and with a “because” if you can. Just to clarify some language:

Artificial Neural Network (ANN): A network as described in the video, but with just one hidden layer

Deep Neural Network: An ANN but with multiple hidden layers. When we say NN in this context, this is usually what we talk about.

Fully Connected NN: Specifying that every single possible connection is made between adjacent layers (this is implicit to the networks shown in the video, but is not always the case!)

Video 2: Gradient descent, how neural networks learn | Chapter 2, Deep learning (20 min)

Synthesis Questions:

  • Why is there a need for a train/test split for a neural network? Why is it important for a NN to be able to generalize to examples it has not seen?
  • Describe the Mean Squared Error (MSE) cost function. What does a higher value mean? What does a lower value mean? (For one training example)
    • Bonus: Assume you have a binary classifier neural network that outputs the vector [0.25, 0.75] and you are using the MSE Loss function to train the Network. The data label indicates that the output for this training example should have been [0, 1]. What is the MSE Loss for this training example?
  • What is the gradient of a function? What is gradient descent?
  • What does minimizing the loss function do to the network's performance over time?
  • Do the hidden layers of a basic NN encode any useful information assuming you use the MSE Loss function? Why or why not?

Great job! These last two videos definitely enter into more theoretical/difficult content, so be prepared. This is something you SHOULD have questions about, so post at least 1 in the Discord!

Video 3: What is backpropagation really doing? | Chapter 3, Deep learning (15 min)

Synthesis Question:

  • Describe Gradient Descent, and how it works in principle with a Deep Neural Network.

The last video may require understanding of partial derivatives (a MATH 126 concept) to fully understand. If you understand the basics of derivatives, however, this should not be too big a leap in understanding. There are no synthesis questions for this last video. Just watch and absorb!

Video 4: Backpropagation calculus | Chapter 4, Deep learning (10 min)

If you would like to jump into the math further, here is a longer lecture for JC Varun did last year walking through how backpropagation is done:

Part 1: Backpropagation, Part 1 | I2 JC

Part 2: Backpropagation, Part 2 | I2 JC

Technical Project Spec:

The project for this “Basics” section will have you finish a code template through Google Colab. Please ask questions as you work through this project. Be sure to discuss with others in your group if you have one! Share your answers as you like, the goal is to learn and we’re not holding grades over your head.

In this project, you will be implementing a Deep Neural Network (DNN)!

A few general helpful tips (if applicable):

  • Be sure to appropriately make a copy of the Colab template before starting to save your progress!
  • Renaming your copy to something that contains your name is a good idea, it will make it easier for us to review your submissions.
  • Leave comments to cement your understanding. Link syntax to ideas.
  • Read up on what MNIST is.

Now, follow the instructions on this Jupyter notebook to implement some of the things we talked about. There is an “answers” link at the bottom of the notebook that you can use if stuck. You will need to download the ‘.ipynb’ found in that directory and open it either locally or in a new colab project yourself. Ask around if you are unable to get it working!

Colab Link: Unit 2 Notebook (1 hr)

When you are finished with your code, independently verify that it works and have fun with it! If you add any additional functionality be sure to talk about it with others and give them ideas.

Remember that this is all for your learning, so do your best and don’t stress!

Congratulations! You now understand the basics of Deep Neural Network structure, how they learn, and how to create one using Python!

Non-Technical Project Spec:

The non-technical project for this unit will involve some writing! Choose 3 of the prompts below and write at least 200 (meaningful!) words on each one! We will not be strictly grading you on correctness or anything like that. This is an opportunity to deeply engage with the material you have just learned about, and creatively connect it to neuroscience!

  • What can we learn from neuroscience to improve the efficiency and performance of artificial neural networks?
  • What are the ethical implications of using insights from neuroscience to design artificial neural networks?
  • How are ANNs inspired by the structure and function of neurons in the brain?
  • What are some common applications of neural networks in real-world scenarios? Feel free to do some research on these!
  • How do neural networks relate to the broader field of machine learning? What are their strengths and weaknesses compared to other algorithms?
  • Reflecting on you learning from this unit, what is one thing you found to be most interesting about DNNs?
  • What is one concept from this unit that you would like to learn more about and why?

Be sure to submit your work through google drive using the submission form! We would prefer that you upload it to your own Drive first, then use the submission form dropbox to connect that file to your submission!