Upload 5 files

README.md

@@ -1,3 +1,124 @@
----
-license: mit
----
+# CLaMP 2: Multimodal Music Information Retrieval Across 101 Languages Using Large Language Models
+
+![CLaMP 2 Overview](overview.jpg)
+
+## Overview
+CLaMP 2 is a music information retrieval model compatible with 101 languages, designed to support both ABC notation (a text-based musical notation format) and MIDI (Musical Instrument Digital Interface). This repository provides a comprehensive suite of scripts for training models, extracting features, converting various musical data formats, generating multilingual summaries of music metadata using GPT-4, and performing music classification and semantic search tasks. By leveraging the multilingual capabilities of GPT-4, CLaMP 2 aims to enhance the accuracy and inclusivity of music retrieval across diverse linguistic and musical modalities.
+
+### Links
+- [CLaMP 2 Code](https://github.com/sanderwood/clamp2)
+- [CLaMP 2 Paper](https://arxiv.org/)
+- [CLaMP 2 Model Weights](https://huggingface.co/sander-wood/clamp2/blob/main/weights_clamp2_h_size_768_lr_5e-05_batch_128_scale_1_t_length_128_t_model_FacebookAI_xlm-roberta-base_t_dropout_True_m3_True.pth)
+- [M3 Model Weights](https://huggingface.co/sander-wood/clamp2/blob/main/weights_m3_p_size_64_p_length_512_t_layers_3_p_layers_12_h_size_768_lr_0.0001_batch_16_mask_0.45.pth)
+
+Note: The model weights for both CLaMP 2 and M3 should be placed under the `code/` folder to ensure proper loading. Make sure the config hyperparameters are correctly set.
+
+## Repository Structure
+The repository is organized into the following main directories:
+
+- **`code/`**: Includes scripts for training the CLaMP 2 and M3 models and extracting features from music and text data. You can modify hyperparameters and file paths in the configuration files to suit your training needs.
+
+- **`music_classification/`**: Contains scripts for performing classification tasks via linear probe using the extracted features. This directory includes utilities for training linear classification models and making predictions on new feature data.
+
+- **`process_data/`**: Provides tools to convert between various musical data formats (ABC notation, MusicXML, MIDI, and MTF) and to summarize music metadata with GPT-4. Before using CLaMP 2 or M3, you should use these scripts to convert your files into interleaved ABC notation or MTF compatible with these models.
+  
+- **`semantic_search/`**: Provides scripts for evaluating model performance, conducting semantic searches, and calculating similarity metrics based on extracted feature vectors.
+
+## Getting Started
+### Environment Setup
+To set up the environment for CLaMP 2, run the following commands:
+
+```bash
+conda env create -f environment.yml
+conda activate clamp2
+```
+
+### Data Preparation
+1. **Convert Files**: Navigate to the `process_data/` folder and convert your music files to a compatible format (interleaved ABC notation or MTF) suitable for CLaMP 2 and M3.
+   - Use the conversion scripts in this folder for tasks like converting MusicXML to ABC and MIDI to MTF.
+
+   - After collecting MusicXML (sheet music) or MIDI (performance data), perform the following operations to convert them into interleaved ABC notation or MTF respectively for model training:
+      1. **Obtain Interleaved ABC Notation**:
+         - Convert MusicXML files to ABC using `batch_xml2abc.py`.
+         - Process the ABC files into interleaved notation using `batch_interleaved_abc.py`.
+      2. **Obtain MTF**:
+         - Convert MIDI files to MTF format using `batch_midi2mtf.py`.
+      3. **Convert Interleaved ABC Back to XML (Optional)**:
+         - Use `batch_xml2abc.py` to convert interleaved ABC files back to MusicXML.
+      4. **Convert MTF Back to MIDI (Optional)**:
+         - Use `batch_mtf2midi.py` to convert MTF files back to MIDI format.
+
+2. **Generate Multilingual Metadata Summaries**: After converting the music files, the next step is to generate multilingual summaries of the music metadata. This is done using the `gpt4_summarize.py` script, which leverages the GPT-4 API to create structured summaries in both English and a randomly selected non-English language.
+
+   **Input Example**: The input to the summarization script consists of a JSON file representing the music metadata. Here’s an example of a music entry in JSON format:
+
+  ```json
+  {
+    "title": "Hard Times Come Again No More",
+    "composer": "Stephen Foster",
+    "genres": ["Children's Music", "Folk"],
+    "description": "\"Hard Times Come Again No More\" (sometimes referred to as \"Hard Times\") is an American parlor song written by Stephen Foster, reflecting themes of sorrow and hope.",
+    "lyrics": "Let us pause in life's pleasures and count its many tears,\nWhile we all sup sorrow with the poor;\nThere's a song that will linger forever in our ears;\nOh! Hard times come again no more.\n\nChorus:\n'Tis the song, the sigh of the weary,\nHard Times, hard times, come again no more.\nMany days you have lingered around my cabin door;\nOh! Hard times come again no more.\n\nWhile we seek mirth and beauty and music light and gay,\nThere are frail forms fainting at the door;\nThough their voices are silent, their pleading looks will say\nOh! Hard times come again no more.\nChorus\n\nThere's a pale weeping maiden who toils her life away,\nWith a worn heart whose better days are o'er:\nThough her voice would be merry, 'tis sighing all the day,\nOh! Hard times come again no more.\nChorus\n\n'Tis a sigh that is wafted across the troubled wave,\n'Tis a wail that is heard upon the shore\n'Tis a dirge that is murmured around the lowly grave\nOh! Hard times come again no more.\nChorus",
+    "tags": ["folk", "traditional", "bluegrass", "nostalgic", "heartfelt", "acoustic", "melancholic", "storytelling", "American roots", "resilience"],
+    "ensembles": ["Folk Ensemble"],
+    "instruments": ["Vocal", "Violin", "Tin whistle", "Guitar", "Banjo", "Tambourine"],
+    "filepaths": [
+      "abc/American_Music/Folk_Traditions/19th_Century/Stephen_Foster/Hard_Times_Come_Again_No_More.abc",
+      "mtf/American_Music/Folk_Traditions/19th_Century/Stephen_Foster/Hard_Times_Come_Again_No_More.mtf"
+    ]
+  }
+  ```
+
+   **Output Example**: The output will be a JSON file containing the structured summary in both English and a selected non-English language. Here’s an example of the expected output:
+
+```json
+{
+  "title": "Hard Times Come Again No More",
+  "composer": "Stephen Foster",
+  "genres": ["Children's Music", "Folk"],
+  "description": "\"Hard Times Come Again No More\" (sometimes referred to as \"Hard Times\") is an American parlor song written by Stephen Foster, reflecting themes of sorrow and hope.",
+  "lyrics": "Let us pause in life's pleasures and count its many tears,\nWhile we all sup sorrow with the poor;\nThere's a song that will linger forever in our ears;\nOh! Hard times come again no more.\n\nChorus:\n'Tis the song, the sigh of the weary,\nHard Times, hard times, come again no more.\nMany days you have lingered around my cabin door;\nOh! Hard times come again no more.\n\nWhile we seek mirth and beauty and music light and gay,\nThere are frail forms fainting at the door;\nThough their voices are silent, their pleading looks will say\nOh! Hard times come again no more.\nChorus\n\nThere's a pale weeping maiden who toils her life away,\nWith a worn heart whose better days are o'er:\nThough her voice would be merry, 'tis sighing all the day,\nOh! Hard times come again no more.\nChorus\n\n'Tis a sigh that is wafted across the troubled wave,\n'Tis a wail that is heard upon the shore\n'Tis a dirge that is murmured around the lowly grave\nOh! Hard times come again no more.\nChorus",
+  "tags": ["folk", "traditional", "bluegrass", "nostalgic", "heartfelt", "acoustic", "melancholic", "storytelling", "American roots", "resilience"],
+  "ensembles": ["Folk Ensemble"],
+  "instruments": ["Vocal", "Violin", "Tin whistle", "Guitar", "Banjo", "Tambourine"],
+  "summary_en": "\"Hard Times Come Again No More,\" composed by Stephen Foster, is a poignant American parlor song that explores themes of sorrow and hope. The lyrics reflect on the contrast between life's pleasures and its hardships, inviting listeners to acknowledge both joy and suffering. With a heartfelt chorus that repeats the line \"Hard times come again no more,\" the song resonates with nostalgia and resilience. It is often performed by folk ensembles and features a variety of instruments, including vocals, violin, guitar, and banjo, encapsulating the spirit of American roots music.",
+  "summary_nen": {
+    "language": "Chinese (Simplified)",
+    "summary": "《艰难时光再无来临》是斯蒂芬·福斯特创作的一首感人至深的美国小歌厅歌曲，探讨了悲伤与希望的主题。歌词展现了生活的乐趣与艰辛之间的对比，邀请听众去感受快乐与痛苦的交织。歌曲中那句反复吟唱的“艰难时光再无来临”深情地表达了怀旧与坚韧。它常常由民谣乐队演奏，伴随着人声、小提琴、吉他和班卓琴等多种乐器，生动地展现了美国根源音乐的独特魅力。"
+  },
+  "filepaths": [
+    "abc/American_Music/Folk_Traditions/19th_Century/Stephen_Foster/Hard_Times_Come_Again_No_More.abc",
+    "mtf/American_Music/Folk_Traditions/19th_Century/Stephen_Foster/Hard_Times_Come_Again_No_More.mtf"
+  ]
+}
+```
+
+### Training and Feature Extraction
+2. **Training Models**: If you want to train CLaMP 2 or M3 models, check the scripts in the `code/` folder.
+   - Modify the `config.py` files to set your training hyperparameters and paths.
+
+3. **Extracting Features**: After training, or if you have pre-trained models, you can extract features from your data using the respective scripts in the `code/` folder.
+
+### Classification and Retrieval
+4. **Classification**: If you need to classify the extracted features, navigate to the `music_classification/` directory.
+   - Here, you'll find scripts to train linear classification models and perform inference on new data.
+
+5. **Semantic Search**: To perform semantic searches using the extracted features, refer to the scripts in the `semantic_search/` folder.
+
+## Benchmarks
+Benchmark datasets related to the experiments conducted with CLaMP 2 and M3, including data used for classification and semantic search tasks, are available in the `benchmarks.zip` file. Note that the `benchmarks.z01` file is required for proper extraction of the contents from `benchmarks.zip`.
+
+## Citation
+
+If you use CLaMP 2 or M3 in your research, please cite the following paper:
+
+```bibtex
+@inproceedings{clamp2,
+  title={CLaMP 2: Multimodal Music Information Retrieval Across 101 Languages Using Large Language Models},
+  author={Author Name and Coauthor Name},
+  booktitle={Proceedings of the Conference on Music Information Retrieval},
+  year={2024},
+  publisher={Publisher Name},
+  address={Conference Location},
+  url={https://placeholder.url}
+}

logs_clamp2_h_size_768_lr_5e-05_batch_128_scale_1_t_length_128_t_model_FacebookAI_xlm-roberta-base_t_dropout_True_m3_True.txt

@@ -0,0 +1,500 @@
+Epoch 1
+train_loss: 4.135209383230448
+eval_loss: 1.9609466393788655
+time: Sun Sep 15 16:53:25 2024
+
+Epoch 2
+train_loss: 2.9014642586344244
+eval_loss: 1.518966309229533
+time: Sun Sep 15 18:43:11 2024
+
+Epoch 3
+train_loss: 2.5554833216573805
+eval_loss: 1.2929850498835245
+time: Sun Sep 15 20:35:14 2024
+
+Epoch 4
+train_loss: 2.3372960954320985
+eval_loss: 1.1918509085973104
+time: Sun Sep 15 22:23:17 2024
+
+Epoch 5
+train_loss: 2.183457492896627
+eval_loss: 1.0725035190582275
+time: Mon Sep 16 00:11:33 2024
+
+Epoch 6
+train_loss: 2.0555087922796216
+eval_loss: 0.9798878033955892
+time: Mon Sep 16 01:59:41 2024
+
+Epoch 7
+train_loss: 1.9556777615308922
+eval_loss: 0.9242686867713928
+time: Mon Sep 16 03:47:45 2024
+
+Epoch 8
+train_loss: 1.8689270445336208
+eval_loss: 0.8476833502451578
+time: Mon Sep 16 05:35:55 2024
+
+Epoch 9
+train_loss: 1.7932923043361795
+eval_loss: 0.8043208519617716
+time: Mon Sep 16 07:24:40 2024
+
+Epoch 10
+train_loss: 1.726651672090251
+eval_loss: 0.7419429302215577
+time: Mon Sep 16 09:12:24 2024
+
+Epoch 11
+train_loss: 1.6657210920084013
+eval_loss: 0.7209376732508341
+time: Mon Sep 16 11:01:02 2024
+
+Epoch 12
+train_loss: 1.6131336856822078
+eval_loss: 0.6950737277666728
+time: Mon Sep 16 12:50:07 2024
+
+Epoch 13
+train_loss: 1.5601606700647368
+eval_loss: 0.652581254641215
+time: Mon Sep 16 14:40:19 2024
+
+Epoch 14
+train_loss: 1.5198849670231784
+eval_loss: 0.5868058403333029
+time: Mon Sep 16 16:30:09 2024
+
+Epoch 15
+train_loss: 1.4755114693644882
+eval_loss: 0.5867449243863424
+time: Mon Sep 16 18:20:11 2024
+
+Epoch 16
+train_loss: 1.4336211351749464
+eval_loss: 0.5479268968105316
+time: Mon Sep 16 20:10:42 2024
+
+Epoch 17
+train_loss: 1.4039166571722088
+eval_loss: 0.5280438164869944
+time: Mon Sep 16 22:01:01 2024
+
+Epoch 18
+train_loss: 1.365380759842085
+eval_loss: 0.5008598109086354
+time: Mon Sep 16 23:51:36 2024
+
+Epoch 19
+train_loss: 1.332988672848894
+eval_loss: 0.46479900081952413
+time: Tue Sep 17 01:41:49 2024
+
+Epoch 20
+train_loss: 1.3014001791981087
+eval_loss: 0.45230263272921245
+time: Tue Sep 17 03:33:12 2024
+
+Epoch 21
+train_loss: 1.2688752540577755
+eval_loss: 0.4297992348670959
+time: Tue Sep 17 05:23:40 2024
+
+Epoch 22
+train_loss: 1.2425967695381415
+eval_loss: 0.4219102164109548
+time: Tue Sep 17 07:14:26 2024
+
+Epoch 23
+train_loss: 1.216824040410488
+eval_loss: 0.40282649795214337
+time: Tue Sep 17 09:05:53 2024
+
+Epoch 24
+train_loss: 1.1875996505747286
+eval_loss: 0.36659018794695536
+time: Tue Sep 17 10:56:46 2024
+
+Epoch 25
+train_loss: 1.1670776548255222
+eval_loss: 0.36906688412030536
+time: Tue Sep 17 12:47:39 2024
+
+Epoch 26
+train_loss: 1.1426405137974536
+eval_loss: 0.3478178918361664
+time: Tue Sep 17 14:38:34 2024
+
+Epoch 27
+train_loss: 1.1208335824466733
+eval_loss: 0.33407697081565857
+time: Tue Sep 17 16:28:45 2024
+
+Epoch 28
+train_loss: 1.0998876758880667
+eval_loss: 0.33792892495791116
+time: Tue Sep 17 18:19:59 2024
+
+Epoch 29
+train_loss: 1.0769698478377083
+eval_loss: 0.3026650925477346
+time: Tue Sep 17 20:11:16 2024
+
+Epoch 30
+train_loss: 1.0587592209657248
+eval_loss: 0.2914476583401362
+time: Tue Sep 17 22:03:30 2024
+
+Epoch 31
+train_loss: 1.0384011404245468
+eval_loss: 0.27578969597816466
+time: Tue Sep 17 23:55:15 2024
+
+Epoch 32
+train_loss: 1.0233595809527622
+eval_loss: 0.2651842157046
+time: Wed Sep 18 01:46:45 2024
+
+Epoch 33
+train_loss: 1.001824217418977
+eval_loss: 0.2630385269721349
+time: Wed Sep 18 03:39:08 2024
+
+Epoch 34
+train_loss: 0.9853754720520442
+eval_loss: 0.25253995358943937
+time: Wed Sep 18 05:30:33 2024
+
+Epoch 35
+train_loss: 0.9676362536067821
+eval_loss: 0.24096360007921855
+time: Wed Sep 18 07:22:09 2024
+
+Epoch 36
+train_loss: 0.9507065269691086
+eval_loss: 0.2413844664891561
+time: Wed Sep 18 09:12:59 2024
+
+Epoch 37
+train_loss: 0.9362979678186832
+eval_loss: 0.23412639300028484
+time: Wed Sep 18 11:04:09 2024
+
+Epoch 38
+train_loss: 0.9174621180856977
+eval_loss: 0.21386308073997498
+time: Wed Sep 18 12:54:52 2024
+
+Epoch 39
+train_loss: 0.9090870427650668
+eval_loss: 0.19962686796983084
+time: Wed Sep 18 14:45:52 2024
+
+Epoch 40
+train_loss: 0.8918763521271409
+eval_loss: 0.20026112000147503
+time: Wed Sep 18 16:36:37 2024
+
+Epoch 41
+train_loss: 0.8786202421428222
+eval_loss: 0.18366556564966838
+time: Wed Sep 18 18:27:31 2024
+
+Epoch 42
+train_loss: 0.8670675420604148
+eval_loss: 0.17908457616964976
+time: Wed Sep 18 20:18:16 2024
+
+Epoch 43
+train_loss: 0.8505593872931582
+eval_loss: 0.17053016225496928
+time: Wed Sep 18 22:10:39 2024
+
+Epoch 44
+train_loss: 0.8421949260766888
+eval_loss: 0.17344878117243448
+time: Thu Sep 19 00:02:24 2024
+
+Epoch 45
+train_loss: 0.8267569324702205
+eval_loss: 0.1591893643140793
+time: Thu Sep 19 01:53:48 2024
+
+Epoch 46
+train_loss: 0.8144617894466949
+eval_loss: 0.15313500861326854
+time: Thu Sep 19 03:44:58 2024
+
+Epoch 47
+train_loss: 0.8041844731303666
+eval_loss: 0.14998503575722377
+time: Thu Sep 19 05:36:50 2024
+
+Epoch 48
+train_loss: 0.7938160687423412
+eval_loss: 0.1401842971642812
+time: Thu Sep 19 07:28:21 2024
+
+Epoch 49
+train_loss: 0.7808867423096515
+eval_loss: 0.1368137091398239
+time: Thu Sep 19 09:20:09 2024
+
+Epoch 50
+train_loss: 0.7702171771933628
+eval_loss: 0.13333487262328467
+time: Thu Sep 19 11:12:37 2024
+
+Epoch 51
+train_loss: 0.7604444062967384
+eval_loss: 0.13119754443566004
+time: Thu Sep 19 13:04:26 2024
+
+Epoch 52
+train_loss: 0.7496546459894258
+eval_loss: 0.1236343190073967
+time: Thu Sep 19 14:55:53 2024
+
+Epoch 53
+train_loss: 0.7406523988345118
+eval_loss: 0.12237562835216523
+time: Thu Sep 19 16:47:51 2024
+
+Epoch 54
+train_loss: 0.7331518270251398
+eval_loss: 0.11441469887892405
+time: Thu Sep 19 18:38:48 2024
+
+Epoch 55
+train_loss: 0.7238280263746373
+eval_loss: 0.10651812156041464
+time: Thu Sep 19 20:29:18 2024
+
+Epoch 56
+train_loss: 0.7141688125488486
+eval_loss: 0.10959143290917078
+time: Thu Sep 19 22:19:28 2024
+
+Epoch 57
+train_loss: 0.7053173944645842
+eval_loss: 0.10957898745934168
+time: Fri Sep 20 00:10:06 2024
+
+Epoch 58
+train_loss: 0.6992166797548109
+eval_loss: 0.09759224901596705
+time: Fri Sep 20 02:01:02 2024
+
+Epoch 59
+train_loss: 0.6855367768623795
+eval_loss: 0.10631066560745239
+time: Fri Sep 20 03:51:25 2024
+
+Epoch 60
+train_loss: 0.6812366953699432
+eval_loss: 0.08681503732999166
+time: Fri Sep 20 05:41:32 2024
+
+Epoch 61
+train_loss: 0.6744320154854127
+eval_loss: 0.08995070978999138
+time: Fri Sep 20 07:32:33 2024
+
+Epoch 62
+train_loss: 0.6627048003782218
+eval_loss: 0.08492780551314354
+time: Fri Sep 20 09:22:52 2024
+
+Epoch 63
+train_loss: 0.6554694614403961
+eval_loss: 0.09110054125388463
+time: Fri Sep 20 11:15:14 2024
+
+Epoch 64
+train_loss: 0.6519363358224428
+eval_loss: 0.08603844990332922
+time: Fri Sep 20 13:05:45 2024
+
+Epoch 65
+train_loss: 0.6432196787488694
+eval_loss: 0.07920929342508316
+time: Fri Sep 20 14:56:27 2024
+
+Epoch 66
+train_loss: 0.6355774498505016
+eval_loss: 0.08108622878789902
+time: Fri Sep 20 16:47:00 2024
+
+Epoch 67
+train_loss: 0.628098195042665
+eval_loss: 0.0835166151324908
+time: Fri Sep 20 18:37:19 2024
+
+Epoch 68
+train_loss: 0.6229319736150211
+eval_loss: 0.08126899500687917
+time: Fri Sep 20 20:27:49 2024
+
+Epoch 69
+train_loss: 0.6162204064685376
+eval_loss: 0.07405624414483707
+time: Fri Sep 20 22:18:28 2024
+
+Epoch 70
+train_loss: 0.6093617768645045
+eval_loss: 0.07916868552565574
+time: Sat Sep 21 00:10:02 2024
+
+Epoch 71
+train_loss: 0.603765148576412
+eval_loss: 0.07368899683157602
+time: Sat Sep 21 02:00:29 2024
+
+Epoch 72
+train_loss: 0.5988557130088281
+eval_loss: 0.06763924509286881
+time: Sat Sep 21 03:51:46 2024
+
+Epoch 73
+train_loss: 0.590835969827209
+eval_loss: 0.07139033873875936
+time: Sat Sep 21 05:43:51 2024
+
+Epoch 74
+train_loss: 0.5864904869113879
+eval_loss: 0.06859012718002001
+time: Sat Sep 21 07:34:23 2024
+
+Epoch 75
+train_loss: 0.5819329118342274
+eval_loss: 0.07611284777522087
+time: Sat Sep 21 09:25:24 2024
+
+Epoch 76
+train_loss: 0.5750655913014898
+eval_loss: 0.06813529431819916
+time: Sat Sep 21 11:16:26 2024
+
+Epoch 77
+train_loss: 0.5703848759963817
+eval_loss: 0.07192744488517443
+time: Sat Sep 21 13:07:32 2024
+
+Epoch 78
+train_loss: 0.5666614368024667
+eval_loss: 0.06931692684690158
+time: Sat Sep 21 14:59:16 2024
+
+Epoch 79
+train_loss: 0.5610024514409998
+eval_loss: 0.06487631574273109
+time: Sat Sep 21 16:50:56 2024
+
+Epoch 80
+train_loss: 0.5552226794301296
+eval_loss: 0.06034566586216291
+time: Sat Sep 21 18:43:49 2024
+
+Epoch 81
+train_loss: 0.5512203840912394
+eval_loss: 0.05962909683585167
+time: Sat Sep 21 20:36:01 2024
+
+Epoch 82
+train_loss: 0.5477618443893468
+eval_loss: 0.05546447386344274
+time: Sat Sep 21 22:28:13 2024
+
+Epoch 83
+train_loss: 0.5428704522615506
+eval_loss: 0.05013169844945272
+time: Sun Sep 22 00:21:20 2024
+
+Epoch 84
+train_loss: 0.5396500316264258
+eval_loss: 0.062498694161574046
+time: Sun Sep 22 02:13:07 2024
+
+Epoch 85
+train_loss: 0.5349479554715307
+eval_loss: 0.06073434228698413
+time: Sun Sep 22 04:05:17 2024
+
+Epoch 86
+train_loss: 0.5292192482811466
+eval_loss: 0.05734321524699529
+time: Sun Sep 22 05:57:05 2024
+
+Epoch 87
+train_loss: 0.5249555090607058
+eval_loss: 0.05274935985604922
+time: Sun Sep 22 07:48:52 2024
+
+Epoch 88
+train_loss: 0.523276918144503
+eval_loss: 0.05601314604282379
+time: Sun Sep 22 09:41:05 2024
+
+Epoch 89
+train_loss: 0.5179934711230115
+eval_loss: 0.057493301729361214
+time: Sun Sep 22 11:33:47 2024
+
+Epoch 90
+train_loss: 0.5129834874146376
+eval_loss: 0.05289425750573476
+time: Sun Sep 22 13:25:54 2024
+
+Epoch 91
+train_loss: 0.5104886514866054
+eval_loss: 0.0586332509915034
+time: Sun Sep 22 15:18:13 2024
+
+Epoch 92
+train_loss: 0.5067275374282622
+eval_loss: 0.0489634457975626
+time: Sun Sep 22 17:10:39 2024
+
+Epoch 93
+train_loss: 0.5038576471461468
+eval_loss: 0.05257208868861198
+time: Sun Sep 22 19:04:46 2024
+
+Epoch 94
+train_loss: 0.5013840998762528
+eval_loss: 0.05249967947602272
+time: Sun Sep 22 20:57:55 2024
+
+Epoch 95
+train_loss: 0.4949465335763684
+eval_loss: 0.048154672731955846
+time: Sun Sep 22 22:50:30 2024
+
+Epoch 96
+train_loss: 0.4925781255166608
+eval_loss: 0.052830965568621956
+time: Mon Sep 23 00:43:13 2024
+
+Epoch 97
+train_loss: 0.4875780233282
+eval_loss: 0.04684837857882182
+time: Mon Sep 23 02:35:38 2024
+
+Epoch 98
+train_loss: 0.4858591078021573
+eval_loss: 0.04507673804958661
+time: Mon Sep 23 04:28:25 2024
+
+Epoch 99
+train_loss: 0.4804891498405977
+eval_loss: 0.048148307204246524
+time: Mon Sep 23 06:21:11 2024
+
+Epoch 100
+train_loss: 0.4782898508661265
+eval_loss: 0.044317328557372096
+time: Mon Sep 23 08:13:38 2024
+

logs_m3_p_size_64_p_length_512_t_layers_3_p_layers_12_h_size_768_lr_0.0001_batch_16_mask_0.45.txt

@@ -0,0 +1,500 @@
+Epoch 1
+train_loss: 0.3055062843047765
+eval_loss: 0.03727418192925584
+time: Wed Aug  7 08:54:27 2024
+
+Epoch 2
+train_loss: 0.1718286834194018
+eval_loss: 0.020085958587123625
+time: Wed Aug  7 14:38:40 2024
+
+Epoch 3
+train_loss: 0.1476379437283353
+eval_loss: 0.013794219702922342
+time: Wed Aug  7 20:24:50 2024
+
+Epoch 4
+train_loss: 0.13554848474242498
+eval_loss: 0.011902455668817844
+time: Thu Aug  8 02:13:14 2024
+
+Epoch 5
+train_loss: 0.12781724531702496
+eval_loss: 0.008929020740909163
+time: Thu Aug  8 08:00:32 2024
+
+Epoch 6
+train_loss: 0.12264176163121285
+eval_loss: 0.008453744098166877
+time: Thu Aug  8 13:47:12 2024
+
+Epoch 7
+train_loss: 0.11872020949762974
+eval_loss: 0.007165850172573819
+time: Thu Aug  8 19:34:09 2024
+
+Epoch 8
+train_loss: 0.1153576639058103
+eval_loss: 0.006601243383027142
+time: Fri Aug  9 01:22:19 2024
+
+Epoch 9
+train_loss: 0.11312788720856465
+eval_loss: 0.005973297544609645
+time: Fri Aug  9 07:11:25 2024
+
+Epoch 10
+train_loss: 0.11096722687304313
+eval_loss: 0.005796642405204946
+time: Fri Aug  9 12:59:13 2024
+
+Epoch 11
+train_loss: 0.10913465011501206
+eval_loss: 0.005249736892483845
+time: Fri Aug  9 18:46:14 2024
+
+Epoch 12
+train_loss: 0.10780615577682347
+eval_loss: 0.005115668955435858
+time: Sat Aug 10 00:34:28 2024
+
+Epoch 13
+train_loss: 0.10650418949283817
+eval_loss: 0.00475350690255028
+time: Sat Aug 10 06:22:51 2024
+
+Epoch 14
+train_loss: 0.10524643352798381
+eval_loss: 0.004583307054632575
+time: Sat Aug 10 12:11:18 2024
+
+Epoch 15
+train_loss: 0.1041887047117438
+eval_loss: 0.004289783886142609
+time: Sat Aug 10 17:59:48 2024
+
+Epoch 16
+train_loss: 0.10343191375801945
+eval_loss: 0.004421581262192111
+time: Sat Aug 10 23:47:37 2024
+
+Epoch 17
+train_loss: 0.10256196519161385
+eval_loss: 0.004104017818401634
+time: Sun Aug 11 05:35:49 2024
+
+Epoch 18
+train_loss: 0.10170993055767087
+eval_loss: 0.0039769458375234585
+time: Sun Aug 11 11:23:39 2024
+
+Epoch 19
+train_loss: 0.1011880517951369
+eval_loss: 0.0039005329324529833
+time: Sun Aug 11 17:11:19 2024
+
+Epoch 20
+train_loss: 0.10030771156829077
+eval_loss: 0.0036845325137673237
+time: Sun Aug 11 22:59:49 2024
+
+Epoch 21
+train_loss: 0.09972109616302548
+eval_loss: 0.0038503893043940205
+time: Mon Aug 12 04:48:03 2024
+
+Epoch 22
+train_loss: 0.09932596696744844
+eval_loss: 0.00370702411211194
+time: Mon Aug 12 10:36:32 2024
+
+Epoch 23
+train_loss: 0.09888291950362459
+eval_loss: 0.0034573812171313834
+time: Mon Aug 12 16:24:55 2024
+
+Epoch 24
+train_loss: 0.09852503939581284
+eval_loss: 0.003370235667697582
+time: Mon Aug 12 22:12:14 2024
+
+Epoch 25
+train_loss: 0.09825884147004627
+eval_loss: 0.00346387299475209
+time: Tue Aug 13 04:00:42 2024
+
+Epoch 26
+train_loss: 0.09756856258879791
+eval_loss: 0.0033276399650575615
+time: Tue Aug 13 09:49:22 2024
+
+Epoch 27
+train_loss: 0.09730380131801182
+eval_loss: 0.003326884365762399
+time: Tue Aug 13 15:36:05 2024
+
+Epoch 28
+train_loss: 0.09687296288584166
+eval_loss: 0.0034621171255395573
+time: Tue Aug 13 21:23:24 2024
+
+Epoch 29
+train_loss: 0.09668537175198876
+eval_loss: 0.003284947640647648
+time: Wed Aug 14 03:10:21 2024
+
+Epoch 30
+train_loss: 0.09628572566572022
+eval_loss: 0.003119471057549999
+time: Wed Aug 14 08:56:45 2024
+
+Epoch 31
+train_loss: 0.09617123452549026
+eval_loss: 0.003124797866062776
+time: Wed Aug 14 14:43:12 2024
+
+Epoch 32
+train_loss: 0.09578377932399237
+eval_loss: 0.0030736677601092537
+time: Wed Aug 14 20:31:01 2024
+
+Epoch 33
+train_loss: 0.09558304869954821
+eval_loss: 0.003178201471396451
+time: Thu Aug 15 02:19:14 2024
+
+Epoch 34
+train_loss: 0.0952804450174092
+eval_loss: 0.0030847328114775225
+time: Thu Aug 15 08:06:29 2024
+
+Epoch 35
+train_loss: 0.09513826066486042
+eval_loss: 0.00303873973446682
+time: Thu Aug 15 13:52:17 2024
+
+Epoch 36
+train_loss: 0.09466769916316405
+eval_loss: 0.0030122215467611258
+time: Thu Aug 15 19:38:29 2024
+
+Epoch 37
+train_loss: 0.09465687754501316
+eval_loss: 0.00289094522015785
+time: Fri Aug 16 01:25:14 2024
+
+Epoch 38
+train_loss: 0.09435585222324992
+eval_loss: 0.0030173959307773393
+time: Fri Aug 16 07:11:56 2024
+
+Epoch 39
+train_loss: 0.09413478592045794
+eval_loss: 0.002968058454507435
+time: Fri Aug 16 12:59:16 2024
+
+Epoch 40
+train_loss: 0.09393180562734375
+eval_loss: 0.0030673167865746948
+time: Fri Aug 16 18:45:23 2024
+
+Epoch 41
+train_loss: 0.09365266143982799
+eval_loss: 0.00287582161187937
+time: Sat Aug 17 00:31:47 2024
+
+Epoch 42
+train_loss: 0.09359205519747489
+eval_loss: 0.0027280030162997134
+time: Sat Aug 17 06:18:32 2024
+
+Epoch 43
+train_loss: 0.09349238520961266
+eval_loss: 0.0029261269570300787
+time: Sat Aug 17 12:05:26 2024
+
+Epoch 44
+train_loss: 0.09324607778116949
+eval_loss: 0.002691730654519444
+time: Sat Aug 17 17:52:20 2024
+
+Epoch 45
+train_loss: 0.09310021795996155
+eval_loss: 0.0028863806760858132
+time: Sat Aug 17 23:38:57 2024
+
+Epoch 46
+train_loss: 0.09307358593283441
+eval_loss: 0.002793597210717352
+time: Sun Aug 18 05:25:42 2024
+
+Epoch 47
+train_loss: 0.09299390690766882
+eval_loss: 0.0027052024821456098
+time: Sun Aug 18 11:12:32 2024
+
+Epoch 48
+train_loss: 0.09253486422624911
+eval_loss: 0.0027312307396534247
+time: Sun Aug 18 16:59:16 2024
+
+Epoch 49
+train_loss: 0.09243107154309635
+eval_loss: 0.002648197562936772
+time: Sun Aug 18 22:46:41 2024
+
+Epoch 50
+train_loss: 0.09237845186490301
+eval_loss: 0.0026844193827840284
+time: Mon Aug 19 04:35:10 2024
+
+Epoch 51
+train_loss: 0.09231985249015236
+eval_loss: 0.002708845011956738
+time: Mon Aug 19 10:24:17 2024
+
+Epoch 52
+train_loss: 0.0922615721153286
+eval_loss: 0.0035362059711223225
+time: Mon Aug 19 16:11:39 2024
+
+Epoch 53
+train_loss: 0.09200190843071623
+eval_loss: 0.0025848455890180064
+time: Mon Aug 19 21:58:31 2024
+
+Epoch 54
+train_loss: 0.09200848002425245
+eval_loss: 0.0026311414897881983
+time: Tue Aug 20 03:45:36 2024
+
+Epoch 55
+train_loss: 0.09154813869071807
+eval_loss: 0.0025586662145983823
+time: Tue Aug 20 09:34:48 2024
+
+Epoch 56
+train_loss: 0.09162745474034129
+eval_loss: 0.0026280648907143545
+time: Tue Aug 20 15:23:23 2024
+
+Epoch 57
+train_loss: 0.09156280245772795
+eval_loss: 0.002539119078534093
+time: Tue Aug 20 21:11:25 2024
+
+Epoch 58
+train_loss: 0.09142590950099329
+eval_loss: 0.0026369429265152866
+time: Wed Aug 21 02:59:19 2024
+
+Epoch 59
+train_loss: 0.09139848643851392
+eval_loss: 0.0024354966580356916
+time: Wed Aug 21 08:46:23 2024
+
+Epoch 60
+train_loss: 0.09131192888740647
+eval_loss: 0.0024594995301248277
+time: Wed Aug 21 14:33:28 2024
+
+Epoch 61
+train_loss: 0.09122042933562911
+eval_loss: 0.002616936316367883
+time: Wed Aug 21 20:20:57 2024
+
+Epoch 62
+train_loss: 0.09109125168796305
+eval_loss: 0.0025555431279884297
+time: Thu Aug 22 02:08:45 2024
+
+Epoch 63
+train_loss: 0.09106527324403817
+eval_loss: 0.0025145284593781213
+time: Thu Aug 22 07:56:26 2024
+
+Epoch 64
+train_loss: 0.09095406525682191
+eval_loss: 0.0025151555842959678
+time: Thu Aug 22 13:45:57 2024
+
+Epoch 65
+train_loss: 0.09102793501718281
+eval_loss: 0.0024135450126194563
+time: Thu Aug 22 19:54:28 2024
+
+Epoch 66
+train_loss: 0.0908411063853937
+eval_loss: 0.002460922076728368
+time: Fri Aug 23 01:59:41 2024
+
+Epoch 67
+train_loss: 0.09070221083785855
+eval_loss: 0.002453409551882543
+time: Fri Aug 23 07:52:30 2024
+
+Epoch 68
+train_loss: 0.0906545008953897
+eval_loss: 0.0024080786435031784
+time: Fri Aug 23 13:41:28 2024
+
+Epoch 69
+train_loss: 0.0907353380525871
+eval_loss: 0.0024573436347799147
+time: Fri Aug 23 19:27:14 2024
+
+Epoch 70
+train_loss: 0.09040538104085095
+eval_loss: 0.0023765437401249566
+time: Sat Aug 24 01:14:45 2024
+
+Epoch 71
+train_loss: 0.09036114065518137
+eval_loss: 0.0023877528348234226
+time: Sat Aug 24 07:02:04 2024
+
+Epoch 72
+train_loss: 0.09037455027205546
+eval_loss: 0.002315233082103814
+time: Sat Aug 24 12:49:24 2024
+
+Epoch 73
+train_loss: 0.09026183628343257
+eval_loss: 0.0024284060419643228
+time: Sat Aug 24 18:35:36 2024
+
+Epoch 74
+train_loss: 0.09019025581511034
+eval_loss: 0.002393116130206718
+time: Sun Aug 25 00:21:29 2024
+
+Epoch 75
+train_loss: 0.089901714783446
+eval_loss: 0.002298152916632467
+time: Sun Aug 25 06:08:01 2024
+
+Epoch 76
+train_loss: 0.09018262871273484
+eval_loss: 0.002273971366672482
+time: Sun Aug 25 11:54:02 2024
+
+Epoch 77
+train_loss: 0.08998425874228
+eval_loss: 0.002317420323379338
+time: Sun Aug 25 17:44:05 2024
+
+Epoch 78
+train_loss: 0.08983653943919646
+eval_loss: 0.0024391192159878743
+time: Sun Aug 25 23:31:34 2024
+
+Epoch 79
+train_loss: 0.08981405456901183
+eval_loss: 0.002319374949895317
+time: Mon Aug 26 05:24:56 2024
+
+Epoch 80
+train_loss: 0.08974534569690559
+eval_loss: 0.0023008979344151066
+time: Mon Aug 26 11:28:33 2024
+
+Epoch 81
+train_loss: 0.08972110153310983
+eval_loss: 0.002406696710865237
+time: Mon Aug 26 17:33:30 2024
+
+Epoch 82
+train_loss: 0.0895689915361898
+eval_loss: 0.002241936448434926
+time: Mon Aug 26 23:39:15 2024
+
+Epoch 83
+train_loss: 0.08950625452328584
+eval_loss: 0.002408353965493697
+time: Tue Aug 27 05:37:59 2024
+
+Epoch 84
+train_loss: 0.08959725393084628
+eval_loss: 0.0023435966142665455
+time: Tue Aug 27 11:34:29 2024
+
+Epoch 85
+train_loss: 0.08970333726515986
+eval_loss: 0.0023965956810233086
+time: Tue Aug 27 17:27:31 2024
+
+Epoch 86
+train_loss: 0.08948115523227308
+eval_loss: 0.002325803569256709
+time: Tue Aug 27 23:19:43 2024
+
+Epoch 87
+train_loss: 0.08933937688654775
+eval_loss: 0.0023552257988114647
+time: Wed Aug 28 05:11:34 2024
+
+Epoch 88
+train_loss: 0.08938353908107184
+eval_loss: 0.0024397599904794043
+time: Wed Aug 28 11:01:23 2024
+
+Epoch 89
+train_loss: 0.08921640703096091
+eval_loss: 0.002223708766084243
+time: Wed Aug 28 16:50:21 2024
+
+Epoch 90
+train_loss: 0.08929300930090782
+eval_loss: 0.0022849828316260303
+time: Wed Aug 28 22:38:53 2024
+
+Epoch 91
+train_loss: 0.08910525214309825
+eval_loss: 0.0022257193633186227
+time: Thu Aug 29 04:35:16 2024
+
+Epoch 92
+train_loss: 0.08905495976636461
+eval_loss: 0.0022299331251850137
+time: Thu Aug 29 10:29:46 2024
+
+Epoch 93
+train_loss: 0.08890526102100955
+eval_loss: 0.0022962711695463786
+time: Thu Aug 29 16:23:49 2024
+
+Epoch 94
+train_loss: 0.08908289874104246
+eval_loss: 0.002243622880820028
+time: Thu Aug 29 22:15:42 2024
+
+Epoch 95
+train_loss: 0.08908785978677156
+eval_loss: 0.0022457318524397784
+time: Fri Aug 30 04:06:57 2024
+
+Epoch 96
+train_loss: 0.08888098475318565
+eval_loss: 0.002224675611787346
+time: Fri Aug 30 09:58:43 2024
+
+Epoch 97
+train_loss: 0.08888529259134526
+eval_loss: 0.0021844924980664493
+time: Fri Aug 30 15:50:16 2024
+
+Epoch 98
+train_loss: 0.08885388837534758
+eval_loss: 0.0022109088076294288
+time: Fri Aug 30 21:41:59 2024
+
+Epoch 99
+train_loss: 0.08873902663868657
+eval_loss: 0.0022606451996653202
+time: Sat Aug 31 03:34:46 2024
+
+Epoch 100
+train_loss: 0.08877080098666765
+eval_loss: 0.002279470525367602
+time: Sat Aug 31 09:28:38 2024
+

weights_clamp2_h_size_768_lr_5e-05_batch_128_scale_1_t_length_128_t_model_FacebookAI_xlm-roberta-base_t_dropout_True_m3_True.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d3f3ae78a878a1a585d7869b311ebeb6c869a823cd9f8d5f222d11857fd0bba8
+size 4450182209

weights_m3_p_size_64_p_length_512_t_layers_3_p_layers_12_h_size_768_lr_0.0001_batch_16_mask_0.45.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8c063f49e60d5811892bcf71acfa2a920ea8eae39491d031392180b0d1754dd7
+size 1360692759