Update common.py

common.py

@@ -265,7 +265,7 @@ global_div = Div(
             H3("Matching Pairs Generation"),
             P("We are using a Jaccard similarity threshold of 0.8 to identify near-duplicate documents. To do this, we divide the MinHashes into 9 bands, each with 13 hashes (also known as the range). To save memory during matching, we first store each band of MinHashes separately on disk. We then process each band individually. Within each band, documents are matched based on their hashes, and the matches are saved as document pairs. A document is considered a match if it matches another document in any of the 9 bands. Since we are looking for near-duplicates, a document may match multiple documents across different bands."),
             P("For partitioning and matching the hashes, we utilize Dask's bag data structure to load the document ids and MinHashes. The matching process is simply a group by operation on this bag data structure. This approach allows us to group matches efficiently and distribute the operation to multiple machines. Also doing a group by produces full components (documents that share the same signature) within a band which simplifies the later stages. The algorithm can be expressed using the Dask expression below:"),
-            Pre(Code(dask_algo,)), 
+            D_code(dask_algo, block="block", language="python"),
             P(B("This step produced 9.2 TB of matching pairs from all bands.")),
         ),
     Section(
@@ -303,7 +303,7 @@ global_div = Div(
             H3("Removing PII"),
             P("We have removed two types of PII from the dataset: email address and IP address. Regular expressions are used to identify and replace these PII with a generic placeholder. Below is an example of how we removed email addresses from the dataset:"),
             P("We have used the following regular expressions to identify and replace PII:"),
-            Ul(Li("Email:"), Li(email_code, style="list-style-type: none"), Li("IP Address:"), Li(ip_address_code, style="list-style-type: none")),
+            Ul(Li("Email:"), Li(D_code(email_code, block="block", language="python"), style="list-style-type: none"), Li("IP Address:"), Li(D_code(ip_address_code, block="block", language="python"), style="list-style-type: none")),
         ),
     Section(
             H2("Normalization Form C (NFC)"),
@@ -313,7 +313,7 @@ global_div = Div(
     Section(
             H3("NFC Implementation"),
             P("We have used the ftfy library to normalize the text to NFC. The library provides a simple API for normalizing text to NFC, which can be applied to the entire dataset one row at a time. Below is the code snippet about how we normalized text to NFC:"),
-            Ul(Li(nfc_code, style="list-style-type: none")), #"background-color= gray" "color= blue" maybe add this later
+            Ul(Li(D_code(nfc_code, block="block", language="python"), style="list-style-type: none")), #"background-color= gray" "color= blue" maybe add this later
         ),
     Section(
             H3("NFC Examples"),