Update common.py

common.py

@@ -299,7 +299,8 @@ global_div = Div(
                 style="margin-bottom: 5px",
             ),
             Li("Normailzation Form C Discussion", style="margin-bottom: 5px"),
-        ),
+        ), 
+        id="section1",
     ),
     Section(
         H2("Motivation Behind Global Deduplication"),
@@ -331,16 +332,18 @@ global_div = Div(
         P(
             "Additionally, we maintained statistics about each matching clusters as they were formed during the final stage of deduplication. Below are the details of all four stages of our deduplication pipeline. We use Dask extensively throughout all stages of the deduplication. We have included the size of results of each stage on disk to give an idea about the scale:"
         ),
+        id="section2",
     ),
     Section(
-        H3("Stage 1: MinHash Generation"),
+        H3("MinHash Generation"),
         P(
             "We use the datasketch library to generate MinHash signatures with the number of permutations to 128. Each signature is signature represented as a MinHash object for each document. Before caluclating the signature, the text is cleaned by stripping whitespace, converting to lowercase, and removing punctuation, consecutive spaces, newlines, and tabs. Next, a list of 13-grams is generated to use as features for creating a document signature. The globally-unique document IDs and signatures are then saved to disk. The documented ID is designed by an encoding scheme which converts file names and line numbers (there is one document per line) to unique document IDs. This also helped a lot in saving disk and memory for this stage."
         ),
         P(B("This step produced 20 TB of hashes.")),
+        id="section3",
     ),
     Section(
-        H3("Stage 2: Matching Pairs Generation"),
+        H3("Matching Pairs Generation"),
         P(
             "We are using a Jaccard similarity threshold of 0.8 to identify near-duplicate documents. To do this, we divide the MinHashes into 9 bands, each with 13 hashes (also known as the range). To save memory during matching, we first store each band of MinHashes separately on disk. We then process each band individually. Within each band, documents are matched based on their hashes, and the matches are saved as document pairs. A document is considered a match if it matches another document in any of the 9 bands. Since we are looking for near-duplicates, a document may match multiple documents across different bands."
         ),
@@ -349,9 +352,10 @@ global_div = Div(
         ),
         D_code(dask_algo, block="block", language="python"),
         P(B("This step produced 9.2 TB of matching pairs from all bands.")),
+        id="section4",
     ),
     Section(
-        H3("Stage 3: Finding Duplicate Pairs"),
+        H3("Finding Duplicate Pairs"),
         P(
             "Multiple bands can create the same document pairs, leading to duplicates. The simplest way to eliminate these duplicate pairs is to call distinct() before the compute(). However, we found that Dask is not very efficient when it comes to distributed distinct execution. Additionally, since we process each band separately, this approach wouldn’t remove duplicates across different bands."
         ),
@@ -366,9 +370,10 @@ global_div = Div(
             "The resulting unique pairs are then used to identify clusters of near-duplicates by finding connected components in a graph, where the vertices represent documents and the edges represent matches."
         ),
         P(B("This step produced 1.9 TB of unique pairs.")),
+        id="section5",
     ),
     Section(
-        H3("Stage 4: Finding Connected Components using MapReduce"),
+        H3("Finding Connected Components using MapReduce"),
         Img(src="images/findcc.svg", style="max-width: 100%;"),
         P(
             "The purpose of this step is to create a set of clusters of matching pairs. For example, a list of pairs (A, B), (B, C), (D, E) is merged into a list of components (A, B, C) and (D, E). Using a third-party library like NetworkX to find connected components would require all pairs to fit into the memory of a single machine, which is not feasible. Instead, we implemented a distributed connected component finder [4] using the Dask framework, which can scale across multiple machines. The algorithm works by mapping edges by both the source and destination of pairs and reducing only edges where the source is greater than the destination. It performs successive iterations of this MapReduce computation until convergence, meaning the number of new edges produced becomes zero. In the end, every document in a cluster points to the smallest document within the cluster. Later, we compile a list of duplicate documents that need deletion and gather statistics about each component."
@@ -380,6 +385,7 @@ global_div = Div(
             "Below is the distribution of duplicate documents found across different dumps of CommonCrawl. The distribution is skewed to the right because the documents are bucketed by the dump ID of the document we retain, and we prefer documents from higher dump IDs."
         ),
         plotly2fasthtml(dup_docs_count_graph()),
+        id="section6",
     ),
     Section(
         H3("Analysis of Near-Duplicate Clusters"),
@@ -424,6 +430,7 @@ global_div = Div(
                 style="list-style-type: none",
             ),
         ),
+        id="section7",
     ),
     Section(
         H2("Normalization Form C"),
@@ -443,13 +450,13 @@ global_div = Div(
                 style="list-style-type: none",
             )
         ),  # "background-color= gray" "color= blue" maybe add this later
+        id="section8",
     ),
     Section(
         H3("NFC Examples"),
         table_div_nfc_examples,
     ),
     Section(H3("Conclusion"), P("NEED TO UPDATE")),
-    id="section1"
 )