Recurrent Neural Networks (RNNs) are a specialized type of neural network specifically designed to process sequential data. Unlike traditional feedforward networks, RNNs have “memory” and can retain information about previous inputs, making them effective for tasks such as text analysis, time series prediction, and speech recognition.

Key Features of RNNs

• Internal State: RNNs use a hidden state that acts as short-term memory. At each time step, this state is updated based on the current input and the previous state.

• Shared Parameters: The same set of weights and biases are used across all time steps, simplifying the model and offering computational advantages.

• Collapsed Outputs: For sequence-to-sequence tasks, the RNN can produce output not only at each time step but also after the entire sequence has been processed.

Visual Comparison

Feedforward Neural Networks (FNNs)

Recurrent Neural Networks (RNNs)

Combining Outputs for Sequence-to-Sequence Tasks

For sequence-to-sequence tasks, the outputs from RNNs can be combined, often using dedicated layers like an encoder followed by a decoder or using advanced architectures like LSTM and GRU.

• Feedforward Neural Networks: No inherent ability to handle sequence input.
• RNNs: Process inputs sequentially and update the hidden state at each time step.

Code Example: RNN

Here is the Python code:

import numpy as np

# Define the RNN parameters
input_size = 3
hidden_size = 2

# Initialize the RNN weights and biases
W_xh = np.random.randn(hidden_size, input_size)  # Input to hidden
W_hh = np.random.randn(hidden_size, hidden_size)  # Hidden to hidden
b_h = np.random.randn(hidden_size, 1)  # Hidden bias
W_hy = np.random.randn(1, hidden_size)  # Hidden to output
b_y = np.random.randn(1, 1)  # Output bias

# Define the RNN forward pass
def rnn_forward(inputs, h_prev):
    # Compute the hidden state
    h_next = np.tanh(W_xh @ inputs + W_hh @ h_prev + b_h)
    # Compute the output
    y_pred = W_hy @ h_next + b_y
    return y_pred, h_next

# Initialize the hidden state and an input sequence
h_init = np.zeros((hidden_size, 1))
X = np.random.randn(input_size, 5)

# Perform the forward pass through time
h_t = h_init
for t in range(X.shape[1]):
    y_pred_t, h_t = rnn_forward(X[:, [t]], h_t)
    print(f"Input at time step {t}: {X[:, [t]].T}, Predicted output: {y_pred_t}")

# Example: Perform the forward pass through time
# Outputs the predicted y and the last hidden state at each time step