Update README.md

README.md

@@ -40,11 +40,13 @@ Figure 1: Training loss.
 <img src="figures/tloss_closeup.png" width="900"/>
 Figure 2: Training loss closeup. We mark two hotswap places, where the training corpus #1 was switched for internal-corpus #2 and internal-corpus #2.1 respectively. The flat region between 112k steps and 119.5k steps is caused by missing data---due to an accident, we lost these logs.
 
-Additionaly, we perform  two ablations:    
+In Figure 2, we perform  two ablations:    
 
-- (a) After first hot swap, we continued training on the corpus #1 for a while.  
+- (a) After first hot swap, we continued training on the corpus #1 for a while. Result: The fact that test loss is slightly better, signifies the slight difference between distribution of corpus #1 and corpus #2. 
 - (b) On step 94,000, the training loss stopped decreasing, increased, and around step 120,000 (near hot swap #2) started decreasing again. To ablate whether this was an effect of hot-swap,
-- we resume training from step 93,000 using corpus #3. The optimizer states were reinitialized.  
+- we resume training from step 93,000 using corpus #3. The optimizer states were reinitialized. Result: Neither corpus #3, nor optimizier state reinitialization seems to mitigate the issue of local divergence at step 94,000. 
+
+- 
 <img src="figures/vloss_closeup.png" width="900"/>
 Figure 3: Test loss closeup, testing performed on split of internal-corpus #1. See Figure 2 description for ablation explanation.