Create PARITY-calculatingNN_Schmidhuber1.py

PARITY-calculatingNN_Schmidhuber1.py

@@ -0,0 +1,244 @@
+# Python script to compute parity of an N-bit binary number using an N-input neural network with L hidden layers, and one output neuron, in PyTorch.
+# Customizable parameters:
+N = 5  # Default number of input bits (and data bit-width)
+L = 2  # Number of hidden layers
+hidden_layer_size = 10 # Default number of neurons per hidden layer
+epochs = 10000
+learning_rate = 0.003
+min_loss_threshold = 0.01 #(stops training epochs when loss reaches less than this number)
+
+# In essence, parity is about the total count of 1s, while XOR is a convenient operation to compute the parity bit.
+
+
+########################
+
+"""
+Python script to compute parity of an N-bit binary number using an N-input neural network with L hidden layers, and one output neuron, in PyTorch.
+
+The Python script calculates even parity. The code that determines the parity is within the generate_data function:
+
+parity = sum(bits) % 2  # Even parity
+
+The sum(bits) calculates the number of 1s in the input bit sequence. The modulo operator % 2 returns 0 if the sum is even, and 1 if the sum is odd. This result is assigned to the parity variable. Because parity is set to 1 only when sum(bits) is odd, parity is '1' when number of bits set to 1 is odd, and therefore this is an even parity calculation. This explicitly calculates even parity: the parity bit is set so the total number of 1s (including the parity bit itself) in the sequence becomes even.[1]
+
+Even parity is not about having an odd number of bits set to one; it's about ensuring that the total number of 1s (data bits + parity bit) is even. This is how error detection works with parity bits: the receiver knows they should always see an even number of 1s. If they encounter an odd number, it flags a transmission error.[1] The script itself generates random data with a variable bit-width N and then adds this parity bit to compute labels. This combined sequence (with a length N + 1) is never used in the script itself. It is the job of the Neural Network to discover the parity calculation on its own during the training phase, where only the input bits without the parity bit, and the calculated parity bit, are presented to the NN.
+
+Includes modes for inference and pretraining with explicit gradient computations and backpropagation.  Displays real-time error during training in a popup window.
+
+Inspired by: [1] Aug 28, 2024 Youtube Interview of Juergen Schmidhuber Schmidhuber at youtube =DP454c1K_vQ
+See also (seven years ago) True Artificial Intelligence will change everything | Juergen Schmidhuber | TEDxLakeComo www.youtube =-Y7PLaxXUrs
+
+[Jürgen Schmidhuber, the father of generative AI shares his groundbreaking work in deep learning and artificial intelligence. In this exclusive interview, he discusses the history of AI, some of his contributions to the field, and his vision for the future of intelligent machines. Schmidhuber offers unique insights into the exponential growth of technology and the potential impact of AI on humanity and the universe.]
+
+In this interview, Schmidhuber stated that LLMs cannot compute "parity" of bits in a binary number sequence, but that "Recurrent" NNs (RNNs) can compute "parity".   I wanted to know whether a simple feed forward NNs can compute "parity".   (If so, perhaps LLMs actually can compute "parity" if specifically trained to do so)
+
+Method of the python script informed by:
+    Create a Basic Neural Network Model - Deep Learning with PyTorch 5 - YouTube =JHWqWIoac2I (2023-06-05)
+and
+    Building a Neural Network with PyTorch in 15 Minutes  youtube =mozBidd58VQ
+
+Loss, as defined, drops smothly and with some bumps visible in the graph, depending on the "random" weights preloaded into the model each run. 
+Example results:
+   Epoch [1000/1000], Loss: 0.0395
+   Test Accuracy: 1.0000
+[Once, model loss fell to 0.0642 after 2000 epochs and did not fall below .0637 after 9999 epochs.  The final sample-tested prediction Accuracy: 0.9700]
+ This shows that the model typically does not need to run the loss all the way down to loss a less than .01 because the margins at the final neuron are earlier greater than .5 between zero/one logit values.   The model script is not specifically written to optimize these margins.  The margins just emerged.
+
+done loading libraries
+Epoch [100/1000], Loss: 0.6247
+Epoch [200/1000], Loss: 0.4112
+Epoch [300/1000], Loss: 0.1990
+Epoch [400/1000], Loss: 0.0849
+Epoch [500/1000], Loss: 0.0413
+Epoch [600/1000], Loss: 0.0250
+Epoch [700/1000], Loss: 0.0172
+Epoch [800/1000], Loss: 0.0128
+Epoch [900/1000], Loss: 0.0100
+Epoch [1000/1000], Loss: 0.0081
+
+Test Accuracy: 1.0000
+Predictions tensor([1., 1., 1., 0., 0., 1., 1., 0., 1., 1., 1., 0., 0., 0., 0., 1., 1., 0.,
+        0., 1., 0., 1., 1., 1., 1., 1., 1., 1., 0., 1., 1., 0., 1., 1., 0., 0.,
+        0., 0., 1., 0., 0., 1., 1., 1., 1., 0., 1., 1., 1., 1., 0., 0., 0., 1.,
+        0., 1., 0., 0., 1., 1., 0., 1., 0., 0., 1., 0., 0., 0., 0., 0., 0., 0.,
+        0., 1., 0., 1., 0., 1., 1., 0., 1., 0., 1., 0., 0., 1., 0., 0., 0., 1.,
+        0., 1., 0., 1., 0., 0., 0., 1., 0., 0.])
+Labels tensor([1., 1., 1., 0., 0., 1., 1., 0., 1., 1., 1., 0., 0., 0., 0., 1., 1., 0.,
+        0., 1., 0., 1., 1., 1., 1., 1., 1., 1., 0., 1., 1., 0., 1., 1., 0., 0.,
+        0., 0., 1., 0., 0., 1., 1., 1., 1., 0., 1., 1., 1., 1., 0., 0., 0., 1.,
+        0., 1., 0., 0., 1., 1., 0., 1., 0., 0., 1., 0., 0., 0., 0., 0., 0., 0.,
+        0., 1., 0., 1., 0., 1., 1., 0., 1., 0., 1., 0., 0., 1., 0., 0., 0., 1.,
+        0., 1., 0., 1., 0., 0., 0., 1., 0., 0.])
+
+
+The line that calculates test_outputs is:
+			test_outputs = model(test_data)
+
+This line uses the model object (an instance of the ParityNet class) and calls its forward method. The code is equivalent to test_outputs = model.forward(test_data). In this line test_data represents a tensor with all the parity bit sequences to check.
+
+The number of neurons and activations that directly contribute to generating test_outputs depends on the network architecture defined by N, L, and hidden_layer_size.
+
+Output Layer: The final layer has one output neuron (because we are predicting a single binary value - the parity). This neuron uses a Sigmoid activation function.
+
+Last Hidden Layer: This layer has hidden_layer_size neurons, each using a ReLU activation function. The output of each neuron in this layer feeds directly into the single output neuron.
+
+Previous Hidden Layers: If L > 1, there are L-1 previous hidden layers, each also with hidden_layer_size neurons and ReLU activations. The activations of each layer feed into the next.
+
+Input Layer: The input layer consists of N nodes which represent the input bits, and are directly connected to the first hidden layer. We might consider that a linear activation function is applied to such input layer.
+
+So, to generate test_outputs, you have the following activations:
+
+hidden_layer_size * L ReLU activations in the hidden layers.
+
+1 Sigmoid activation in the output layer.
+
+N linear activations in the input layer (optional).
+
+In summary, hidden_layer_size neurons and ReLU activations in the last hidden layer and 1 neuron with a Sigmoid activation in the output layer immediately generate the test_outputs values. All the other (L-1) * hidden_layer_size neurons with ReLU activations in the preceding hidden layers indirectly contribute by feeding into the last hidden layer. Each of the input bits are treated individually via the N nodes of the input layer, that might be considered having a linear activation or no activation at all.
+
+"""
+########################
+
+print("load libraries")
+print("import torch")
+import torch
+print("import torch.nn as nn")
+import torch.nn as nn
+print("import numpy as np")
+import numpy as np
+print("import matplotlib.pyplot as plt  # For the popup error plot")
+import matplotlib.pyplot as plt  # For the popup error plot
+print("import random")
+import random
+print("done loading libraries")
+
+# Generate training data (parity bit is XOR or all data bits)
+def generate_data(num_samples, num_bits):
+    data = []
+    labels = []
+    for _ in range(num_samples):
+        bits = [random.randint(0, 1) for _ in range(num_bits)]
+        parity = sum(bits) % 2  # Even parity: The modulo operator % 2 returns 0 if the sum is even, and 1 if the sum is odd. 
+        data.append(bits)
+        labels.append(parity)
+
+    return torch.tensor(data, dtype=torch.float32), torch.tensor(labels, dtype=torch.float32).reshape(-1, 1)
+
+train_data, train_labels = generate_data(1000, N) # 1000 N-bit numbers generated for training data.
+test_data, test_labels = generate_data(100, N) # 100 N-bit numbers generated for test data.
+
+
+# Define the neural network model
+class ParityNet(nn.Module):
+    def __init__(self, input_size, hidden_size, num_hidden_layers, output_size):
+        super(ParityNet, self).__init__()
+        layers = []
+        layers.append(nn.Linear(input_size, hidden_size))
+        layers.append(nn.ReLU()) # Activation function
+        for _ in range(num_hidden_layers - 1):
+            layers.append(nn.Linear(hidden_size, hidden_size))
+            layers.append(nn.ReLU()) # Activation function
+        layers.append(nn.Linear(hidden_size, output_size))
+        layers.append(nn.Sigmoid()) # Output layer activation function for binary classification
+        self.layers = nn.Sequential(*layers)
+
+
+    def forward(self, x):
+        return self.layers(x)
+
+
+# Create the model instance
+model = ParityNet(N, hidden_layer_size, L, 1)
+
+# Loss function and optimizer
+criterion = nn.BCELoss() # Binary Cross Entropy Loss
+optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)
+
+# Pretraining loop
+losses = [] # Store losses for plotting
+plt.ion()  # Turn on interactive plotting
+fig, ax = plt.subplots() # Create plot objects outside the loop
+
+######################## TRAINING ########################
+for epoch in range(epochs):
+    # Forward pass
+    outputs = model(train_data)
+    loss = criterion(outputs, train_labels)
+
+    # Explicit gradient computation and backpropagation
+    optimizer.zero_grad()
+    loss.backward()
+    optimizer.step()
+
+
+    losses.append(loss.item())
+
+    ################ ADD:  If loss.item() < (min_loss_threshold +0.01) Then, save the failed train_data values to failed_train_data[] and append to failed_train_data.txt   Thus, before the end of training, grab the problem bit values and accumulate these difficult training cases in an external file.  In future versions, use these hardest-to-train values to somehow extra-train the model.  ################
+
+    if loss.item() < min_loss_threshold:
+        print(f"Reached minimum loss threshold of {min_loss_threshold} at epoch {epoch+1}. Stopping training.")
+        break  # Exit the training loop
+
+    if (epoch + 1) % 100 == 0:
+        print(f'Epoch [{epoch+1}/{epochs}], Loss: {loss.item():.4f}')
+
+        # Update the plot in real time
+        ax.clear()  # Clear previous plot
+        ax.plot(losses) # plot loss history
+        ax.set_title("Training Loss")
+        ax.set_xlabel("Epoch (x 100)")
+        ax.set_ylabel("Loss")
+        plt.draw()
+        plt.pause(0.01)  # Brief pause for plot to update
+
+
+plt.ioff() # Turn off interactive mode so plot persists after training is done
+plt.show() # show final plot.
+
+######################## INFERENCE-TESTING ########################
+# Inference (Testing)
+with torch.no_grad():
+    test_outputs = model(test_data)
+    predicted = (test_outputs > 0.5).float() # Convert probabilities to binary predictions (0 or 1)
+
+    accuracy = (predicted == test_labels).sum() / len(test_labels)
+    print(f'Test Accuracy: {accuracy:.4f}')
+    print("Predictions", predicted.flatten())
+
+    print("Labels     ", test_labels.flatten())
+
+    # Separate margins for predictions of 1 and 0
+    margins_ones = test_outputs[predicted == 1] - 0.5
+    margins_zeros =  0.5 - test_outputs[predicted == 0] 
+
+
+    # Calculate and print statistics for margins of 1s
+    if margins_ones.numel() > 0:  # Check if there are any predictions of 1
+        min_margin_ones = margins_ones.min().item()
+        max_margin_ones = margins_ones.max().item()
+        avg_margin_ones = margins_ones.mean().item()
+
+        print(f"Min Margin (Ones): {min_margin_ones:.2f}")
+        print(f"Max Margin (Ones): {max_margin_ones:.2f}")
+        print(f"Avg Margin (Ones): {avg_margin_ones:.2f}")
+        print("Margins (Ones):", margins_ones.flatten().numpy())
+    else:
+        print("No predictions of 1 in the test dataset.")
+
+    # Calculate and print statistics for margins of 0s
+    if margins_zeros.numel() > 0:  # Check if there are any predictions of 0
+
+        min_margin_zeros = margins_zeros.min().item()
+        max_margin_zeros = margins_zeros.max().item()
+        avg_margin_zeros = margins_zeros.mean().item()
+
+        print(f"Min Margin (Zeros): {min_margin_zeros:.2f}")
+        print(f"Max Margin (Zeros): {max_margin_zeros:.2f}")
+        print(f"Avg Margin (Zeros): {avg_margin_zeros:.2f}")
+        print("Margins (Zeros):", margins_zeros.flatten().numpy())
+
+    else:
+        print("No predictions of 0 in the test dataset.")
+########################  ADD EXPORT WORKING-MODEL WEIGHTS TO PARTIY-[hyperparameters]NN_weights.bin HERE ########################
+
+########################  ADD A USER-INPUT (BINARY SEQUENCE) MODE HERE ########################