Hugging Face's logo

archivartaunik
/
BE_XTTS_V2_Model_all

TensorBoard
Model card Files Metrics Community
BE_XTTS_V2_Model_all / trainer_0_log.txt
archivartaunik's picture
archivartaunik
Initial commit: Upload trained model
941dc4e
raw
history blame contribute delete
102 kB
> Training Environment:
| > Backend: Torch
| > Mixed precision: False
| > Precision: float32
| > Current device: 0
| > Num. of GPUs: 1
| > Num. of CPUs: 2
| > Num. of Torch Threads: 1
| > Torch seed: 54321
| > Torch CUDNN: True
| > Torch CUDNN deterministic: False
| > Torch CUDNN benchmark: False
| > Torch TF32 MatMul: False
> Start Tensorboard: tensorboard --logdir=checkpoints/GPT_XTTS_FT-November-16-2024_05+12AM-4b65e59
> Model has 537022379 parameters
 > EPOCH: 0/2
--> checkpoints/GPT_XTTS_FT-November-16-2024_05+12AM-4b65e59
 > TRAINING (2024-11-16 05:12:17) 
 --> TIME: 2024-11-16 05:12:23 -- STEP: 0/6312 -- GLOBAL_STEP: 0
| > loss_text_ce: 0.14104962348937988 (0.14104962348937988)
| > loss_mel_ce: 5.240347862243652 (5.240347862243652)
| > loss: 1.3453493118286133 (1.3453493118286133)
| > current_lr: 5e-06
| > step_time: 1.3408 (1.3408386707305908)
| > loader_time: 4.9701 (4.970067977905273)
 --> TIME: 2024-11-16 05:12:56 -- STEP: 50/6312 -- GLOBAL_STEP: 50
| > loss_text_ce: 0.1256498247385025 (0.13665068656206134)
| > loss_mel_ce: 4.53818941116333 (5.098864202499389)
| > loss: 1.1659598350524902 (1.3088787221908569)
| > current_lr: 5e-06
| > step_time: 0.3228 (0.38985726833343504)
| > loader_time: 0.0163 (0.025168466567993163)
 --> TIME: 2024-11-16 05:13:28 -- STEP: 100/6312 -- GLOBAL_STEP: 100
| > loss_text_ce: 0.11902274191379547 (0.12999648697674276)
| > loss_mel_ce: 4.957067012786865 (4.968110985755917)
| > loss: 1.2690224647521973 (1.2745268666744236)
| > current_lr: 5e-06
| > step_time: 0.2707 (0.3827947974205017)
| > loader_time: 0.0127 (0.02370228528976441)
 --> TIME: 2024-11-16 05:14:00 -- STEP: 150/6312 -- GLOBAL_STEP: 150
| > loss_text_ce: 0.11355013400316238 (0.12433155233661333)
| > loss_mel_ce: 4.567184925079346 (4.899189014434813)
| > loss: 1.170183777809143 (1.255880143642426)
| > current_lr: 5e-06
| > step_time: 0.3435 (0.3810132249196371)
| > loader_time: 0.0146 (0.023247008323669435)
 --> TIME: 2024-11-16 05:14:34 -- STEP: 200/6312 -- GLOBAL_STEP: 200
| > loss_text_ce: 0.10290912538766861 (0.11949403874576092)
| > loss_mel_ce: 4.486223220825195 (4.855822575092312)
| > loss: 1.1472830772399902 (1.2438291537761692)
| > current_lr: 5e-06
| > step_time: 0.386 (0.3886987578868867)
| > loader_time: 0.0135 (0.02197468876838685)
 --> TIME: 2024-11-16 05:15:08 -- STEP: 250/6312 -- GLOBAL_STEP: 250
| > loss_text_ce: 0.09945546835660934 (0.11500931948423386)
| > loss_mel_ce: 4.745148658752441 (4.819211471557615)
| > loss: 1.2111510038375854 (1.2335551972389225)
| > current_lr: 5e-06
| > step_time: 0.3539 (0.39221693420410164)
| > loader_time: 0.0328 (0.021144410133361824)
 --> TIME: 2024-11-16 05:15:42 -- STEP: 300/6312 -- GLOBAL_STEP: 300
| > loss_text_ce: 0.08811246603727341 (0.11150965929031371)
| > loss_mel_ce: 4.677680492401123 (4.792185862859089)
| > loss: 1.1914482116699219 (1.2259238803386687)
| > current_lr: 5e-06
| > step_time: 0.4595 (0.39503660122553513)
| > loader_time: 0.0157 (0.0206674067179362)
 --> TIME: 2024-11-16 05:16:16 -- STEP: 350/6312 -- GLOBAL_STEP: 350
| > loss_text_ce: 0.07782182842493057 (0.10851263429437365)
| > loss_mel_ce: 4.4658026695251465 (4.758750927788871)
| > loss: 1.1359061002731323 (1.216815891265869)
| > current_lr: 5e-06
| > step_time: 0.3491 (0.39657673018319267)
| > loader_time: 0.0113 (0.020254495484488343)
 --> TIME: 2024-11-16 05:16:50 -- STEP: 400/6312 -- GLOBAL_STEP: 400
| > loss_text_ce: 0.07956518232822418 (0.10595279274508357)
| > loss_mel_ce: 4.402898788452148 (4.737360988855363)
| > loss: 1.1206159591674805 (1.2108284452557567)
| > current_lr: 5e-06
| > step_time: 0.4288 (0.39860816538333893)
| > loader_time: 0.0131 (0.019769405126571663)
 --> TIME: 2024-11-16 05:17:25 -- STEP: 450/6312 -- GLOBAL_STEP: 450
| > loss_text_ce: 0.09433493763208389 (0.10389996631277933)
| > loss_mel_ce: 4.67876672744751 (4.717605740229288)
| > loss: 1.1932754516601562 (1.205376425849067)
| > current_lr: 5e-06
| > step_time: 0.3502 (0.40110296779208715)
| > loader_time: 0.0113 (0.019343772994147416)
 --> TIME: 2024-11-16 05:18:00 -- STEP: 500/6312 -- GLOBAL_STEP: 500
| > loss_text_ce: 0.08759494870901108 (0.1020422927737236)
| > loss_mel_ce: 4.380007266998291 (4.703697700500488)
| > loss: 1.1169005632400513 (1.201434997677803)
| > current_lr: 5e-06
| > step_time: 0.2818 (0.4035099425315857)
| > loader_time: 0.0123 (0.01882465314865113)
 --> TIME: 2024-11-16 05:18:35 -- STEP: 550/6312 -- GLOBAL_STEP: 550
| > loss_text_ce: 0.08977686613798141 (0.10058542449365962)
| > loss_mel_ce: 4.399405002593994 (4.694155686985362)
| > loss: 1.1222954988479614 (1.1986852777004242)
| > current_lr: 5e-06
| > step_time: 0.457 (0.40537407398223885)
| > loader_time: 0.0093 (0.018361483920704236)
 --> TIME: 2024-11-16 05:19:09 -- STEP: 600/6312 -- GLOBAL_STEP: 600
| > loss_text_ce: 0.08253240585327148 (0.09914301505933198)
| > loss_mel_ce: 4.624935626983643 (4.68365015745163)
| > loss: 1.1768670082092285 (1.1956982931494715)
| > current_lr: 5e-06
| > step_time: 0.3427 (0.4061408785978953)
| > loader_time: 0.0117 (0.018098548650741595)
 --> TIME: 2024-11-16 05:19:43 -- STEP: 650/6312 -- GLOBAL_STEP: 650
| > loss_text_ce: 0.07322724908590317 (0.09782060979650561)
| > loss_mel_ce: 4.512180805206299 (4.666836794706488)
| > loss: 1.1463520526885986 (1.1911643508764413)
| > current_lr: 5e-06
| > step_time: 0.4501 (0.40616705527672403)
| > loader_time: 0.0089 (0.017903365355271574)
 --> TIME: 2024-11-16 05:20:16 -- STEP: 700/6312 -- GLOBAL_STEP: 700
| > loss_text_ce: 0.07802781462669373 (0.09661007312791682)
| > loss_mel_ce: 4.662949085235596 (4.652626109123228)
| > loss: 1.1852442026138306 (1.1873090450252812)
| > current_lr: 5e-06
| > step_time: 0.3794 (0.4054310897418431)
| > loader_time: 0.0319 (0.017786686761038638)
 --> TIME: 2024-11-16 05:20:51 -- STEP: 750/6312 -- GLOBAL_STEP: 750
| > loss_text_ce: 0.08538275212049484 (0.09547996746500327)
| > loss_mel_ce: 4.033980369567871 (4.639854468345641)
| > loss: 1.0298408269882202 (1.1838336089452113)
| > current_lr: 5e-06
| > step_time: 0.3082 (0.4063705066045125)
| > loader_time: 0.0118 (0.01750638167063395)
 --> TIME: 2024-11-16 05:21:25 -- STEP: 800/6312 -- GLOBAL_STEP: 800
| > loss_text_ce: 0.08083344250917435 (0.09454715159721667)
| > loss_mel_ce: 4.453289031982422 (4.6307297816872595)
| > loss: 1.133530616760254 (1.181319233328104)
| > current_lr: 5e-06
| > step_time: 0.2832 (0.4064507481455803)
| > loader_time: 0.0279 (0.01731537014245987)
 --> TIME: 2024-11-16 05:21:59 -- STEP: 850/6312 -- GLOBAL_STEP: 850
| > loss_text_ce: 0.08859393000602722 (0.0936372162226368)
| > loss_mel_ce: 4.646214962005615 (4.619028070393733)
| > loss: 1.1837022304534912 (1.1781663215861604)
| > current_lr: 5e-06
| > step_time: 0.4332 (0.4069419201682596)
| > loader_time: 0.0095 (0.017175954930922555)
 --> TIME: 2024-11-16 05:22:33 -- STEP: 900/6312 -- GLOBAL_STEP: 900
| > loss_text_ce: 0.06786824017763138 (0.09262992711944708)
| > loss_mel_ce: 4.446366786956787 (4.6075592319170635)
| > loss: 1.1285587549209595 (1.1750472897953463)
| > current_lr: 5e-06
| > step_time: 0.3775 (0.4069652877913581)
| > loader_time: 0.0115 (0.017038922309875477)
 --> TIME: 2024-11-16 05:23:08 -- STEP: 950/6312 -- GLOBAL_STEP: 950
| > loss_text_ce: 0.06802818179130554 (0.09190385385171358)
| > loss_mel_ce: 3.9986894130706787 (4.599870970625626)
| > loss: 1.0166794061660767 (1.172943706386968)
| > current_lr: 5e-06
| > step_time: 0.3617 (0.40769850530122453)
| > loader_time: 0.009 (0.01692695592579088)
 --> TIME: 2024-11-16 05:23:42 -- STEP: 1000/6312 -- GLOBAL_STEP: 1000
| > loss_text_ce: 0.08169705420732498 (0.09112126449868078)
| > loss_mel_ce: 4.460300445556641 (4.591032954216003)
| > loss: 1.135499358177185 (1.1705385549068454)
| > current_lr: 5e-06
| > step_time: 0.2813 (0.40798303055763246)
| > loader_time: 0.0106 (0.016841878652572626)
 --> TIME: 2024-11-16 05:24:17 -- STEP: 1050/6312 -- GLOBAL_STEP: 1050
| > loss_text_ce: 0.08561321347951889 (0.09050357801928402)
| > loss_mel_ce: 4.477149963378906 (4.583879175640288)
| > loss: 1.140690803527832 (1.1685956884565811)
| > current_lr: 5e-06
| > step_time: 0.3515 (0.4086405563354492)
| > loader_time: 0.0132 (0.01667730422247023)
 --> TIME: 2024-11-16 05:24:51 -- STEP: 1100/6312 -- GLOBAL_STEP: 1100
| > loss_text_ce: 0.07064620405435562 (0.0898570882021026)
| > loss_mel_ce: 4.681677341461182 (4.5762473687258645)
| > loss: 1.188080906867981 (1.1665261146155277)
| > current_lr: 5e-06
| > step_time: 0.3981 (0.4091884717074306)
| > loader_time: 0.0344 (0.016597047935832634)
 --> TIME: 2024-11-16 05:25:26 -- STEP: 1150/6312 -- GLOBAL_STEP: 1150
| > loss_text_ce: 0.07308947294950485 (0.08921846408558928)
| > loss_mel_ce: 4.367588520050049 (4.569452874142191)
| > loss: 1.110169529914856 (1.1646678344063144)
| > current_lr: 5e-06
| > step_time: 0.4461 (0.4095196064658785)
| > loader_time: 0.0103 (0.01650891490604567)
 --> TIME: 2024-11-16 05:25:59 -- STEP: 1200/6312 -- GLOBAL_STEP: 1200
| > loss_text_ce: 0.07887333631515503 (0.08870254772404831)
| > loss_mel_ce: 4.400242805480957 (4.563704760471981)
| > loss: 1.1197789907455444 (1.163101827005546)
| > current_lr: 5e-06
| > step_time: 0.4395 (0.40895789404710114)
| > loader_time: 0.0117 (0.016428915063540137)
 --> TIME: 2024-11-16 05:26:33 -- STEP: 1250/6312 -- GLOBAL_STEP: 1250
| > loss_text_ce: 0.06801445037126541 (0.08811039642691615)
| > loss_mel_ce: 4.463882923126221 (4.556243999099734)
| > loss: 1.13297438621521 (1.1610885987281814)
| > current_lr: 5e-06
| > step_time: 0.4375 (0.40895634708404516)
| > loader_time: 0.0112 (0.016322176551818855)
 --> TIME: 2024-11-16 05:27:07 -- STEP: 1300/6312 -- GLOBAL_STEP: 1300
| > loss_text_ce: 0.07231567054986954 (0.08757366121961516)
| > loss_mel_ce: 4.3624749183654785 (4.548244301355802)
| > loss: 1.1086976528167725 (1.1589544906066038)
| > current_lr: 5e-06
| > step_time: 0.3508 (0.408849203953376)
| > loader_time: 0.0091 (0.016269375360929067)
 --> TIME: 2024-11-16 05:27:40 -- STEP: 1350/6312 -- GLOBAL_STEP: 1350
| > loss_text_ce: 0.07841095328330994 (0.08708857991077276)
| > loss_mel_ce: 4.323296070098877 (4.54062715265486)
| > loss: 1.1004267930984497 (1.1569289331966002)
| > current_lr: 5e-06
| > step_time: 0.4541 (0.4087571082291778)
| > loader_time: 0.0098 (0.01623839802212188)
 --> TIME: 2024-11-16 05:28:14 -- STEP: 1400/6312 -- GLOBAL_STEP: 1400
| > loss_text_ce: 0.07222231477499008 (0.08658362559442007)
| > loss_mel_ce: 4.286852836608887 (4.533556108134139)
| > loss: 1.0897687673568726 (1.1550349332605117)
| > current_lr: 5e-06
| > step_time: 0.4121 (0.4087279193741933)
| > loader_time: 0.0092 (0.016113448653902362)
 --> TIME: 2024-11-16 05:28:48 -- STEP: 1450/6312 -- GLOBAL_STEP: 1450
| > loss_text_ce: 0.07711783051490784 (0.08609157691227973)
| > loss_mel_ce: 3.96337890625 (4.527316927252147)
| > loss: 1.0101242065429688 (1.153352126253066)
| > current_lr: 5e-06
| > step_time: 0.3586 (0.4088464894788017)
| > loader_time: 0.0087 (0.01606827028866475)
 --> TIME: 2024-11-16 05:29:22 -- STEP: 1500/6312 -- GLOBAL_STEP: 1500
| > loss_text_ce: 0.0689086988568306 (0.08565062620987487)
| > loss_mel_ce: 4.366149425506592 (4.521239313602451)
| > loss: 1.1087645292282104 (1.1517224850654637)
| > current_lr: 5e-06
| > step_time: 0.3531 (0.4087925087610879)
| > loader_time: 0.0158 (0.015977238178253198)
 --> TIME: 2024-11-16 05:29:57 -- STEP: 1550/6312 -- GLOBAL_STEP: 1550
| > loss_text_ce: 0.059200942516326904 (0.0852852678947871)
| > loss_mel_ce: 4.0393853187561035 (4.516698020965825)
| > loss: 1.024646520614624 (1.1504958223527504)
| > current_lr: 5e-06
| > step_time: 0.4623 (0.40934916603949745)
| > loader_time: 0.0091 (0.015931478161965662)
 --> TIME: 2024-11-16 05:30:31 -- STEP: 1600/6312 -- GLOBAL_STEP: 1600
| > loss_text_ce: 0.0819307416677475 (0.08484024197328828)
| > loss_mel_ce: 4.175338268280029 (4.510828148573637)
| > loss: 1.064317226409912 (1.1489170976355687)
| > current_lr: 5e-06
| > step_time: 0.235 (0.40955067127943023)
| > loader_time: 0.0095 (0.015910512059926985)
 --> TIME: 2024-11-16 05:31:06 -- STEP: 1650/6312 -- GLOBAL_STEP: 1650
| > loss_text_ce: 0.06460896879434586 (0.0844438162558909)
| > loss_mel_ce: 4.225306510925293 (4.504031245347223)
| > loss: 1.0724788904190063 (1.1471187655131037)
| > current_lr: 5e-06
| > step_time: 0.3173 (0.4097674937681717)
| > loader_time: 0.0101 (0.01587408846074884)
 --> TIME: 2024-11-16 05:31:40 -- STEP: 1700/6312 -- GLOBAL_STEP: 1700
| > loss_text_ce: 0.07932015508413315 (0.08408904919072102)
| > loss_mel_ce: 4.527863025665283 (4.498533144277682)
| > loss: 1.151795744895935 (1.1456555485725417)
| > current_lr: 5e-06
| > step_time: 0.2878 (0.4098787098772384)
| > loader_time: 0.0122 (0.0158283355656792)
 --> TIME: 2024-11-16 05:32:14 -- STEP: 1750/6312 -- GLOBAL_STEP: 1750
| > loss_text_ce: 0.07434050738811493 (0.0837666027418204)
| > loss_mel_ce: 4.158716201782227 (4.493645459583827)
| > loss: 1.058264136314392 (1.14435301589966)
| > current_lr: 5e-06
| > step_time: 0.3185 (0.41005595888410284)
| > loader_time: 0.0245 (0.015767244475228426)
 --> TIME: 2024-11-16 05:32:48 -- STEP: 1800/6312 -- GLOBAL_STEP: 1800
| > loss_text_ce: 0.06926805526018143 (0.08341063009368042)
| > loss_mel_ce: 4.127894878387451 (4.486679943270156)
| > loss: 1.0492907762527466 (1.1425226436720972)
| > current_lr: 5e-06
| > step_time: 0.3967 (0.4098496133751338)
| > loader_time: 0.0118 (0.015738437705569767)
 --> TIME: 2024-11-16 05:33:22 -- STEP: 1850/6312 -- GLOBAL_STEP: 1850
| > loss_text_ce: 0.07369551807641983 (0.08302218583185925)
| > loss_mel_ce: 4.164562225341797 (4.4820840541736535)
| > loss: 1.059564471244812 (1.1412765602807757)
| > current_lr: 5e-06
| > step_time: 0.422 (0.4100692248988795)
| > loader_time: 0.0089 (0.015716763960348552)
 --> TIME: 2024-11-16 05:33:56 -- STEP: 1900/6312 -- GLOBAL_STEP: 1900
| > loss_text_ce: 0.07696900516748428 (0.08272866983162723)
| > loss_mel_ce: 4.4198689460754395 (4.477659515456153)
| > loss: 1.1242095232009888 (1.1400970465885982)
| > current_lr: 5e-06
| > step_time: 0.3598 (0.4099533319473265)
| > loader_time: 0.0092 (0.01567801525718283)
 --> TIME: 2024-11-16 05:34:30 -- STEP: 1950/6312 -- GLOBAL_STEP: 1950
| > loss_text_ce: 0.07173514366149902 (0.08237915507875951)
| > loss_mel_ce: 4.137192726135254 (4.474048470961746)
| > loss: 1.052232027053833 (1.1391069069886834)
| > current_lr: 5e-06
| > step_time: 0.4671 (0.410018314214853)
| > loader_time: 0.0093 (0.015620202162326867)
 --> TIME: 2024-11-16 05:35:04 -- STEP: 2000/6312 -- GLOBAL_STEP: 2000
| > loss_text_ce: 0.0743638277053833 (0.08209441693685943)
| > loss_mel_ce: 4.170366287231445 (4.469159315228466)
| > loss: 1.0611824989318848 (1.137813433438541)
| > current_lr: 5e-06
| > step_time: 0.4513 (0.4101108170747756)
| > loader_time: 0.0092 (0.015560936093330341)
 --> TIME: 2024-11-16 05:35:38 -- STEP: 2050/6312 -- GLOBAL_STEP: 2050
| > loss_text_ce: 0.07275821268558502 (0.0818381885439157)
| > loss_mel_ce: 4.54878044128418 (4.465182435686999)
| > loss: 1.1553846597671509 (1.1367551563135023)
| > current_lr: 5e-06
| > step_time: 0.4566 (0.40992009674630503)
| > loader_time: 0.0091 (0.01559479806481334)
 --> TIME: 2024-11-16 05:36:13 -- STEP: 2100/6312 -- GLOBAL_STEP: 2100
| > loss_text_ce: 0.08702623099088669 (0.08153459277358788)
| > loss_mel_ce: 4.495286464691162 (4.461959453196755)
| > loss: 1.145578145980835 (1.1358735118309675)
| > current_lr: 5e-06
| > step_time: 0.3738 (0.4102420771689641)
| > loader_time: 0.0091 (0.0155538245609828)
 --> TIME: 2024-11-16 05:36:47 -- STEP: 2150/6312 -- GLOBAL_STEP: 2150
| > loss_text_ce: 0.09227096289396286 (0.0812623859682055)
| > loss_mel_ce: 4.202178478240967 (4.457476372607921)
| > loss: 1.0736123323440552 (1.1346846898212006)
| > current_lr: 5e-06
| > step_time: 0.4643 (0.41030180609503447)
| > loader_time: 0.0295 (0.015532794109610584)
 --> TIME: 2024-11-16 05:37:21 -- STEP: 2200/6312 -- GLOBAL_STEP: 2200
| > loss_text_ce: 0.06780247390270233 (0.08097483231432057)
| > loss_mel_ce: 4.271034240722656 (4.454012486934666)
| > loss: 1.0847091674804688 (1.1337468300353413)
| > current_lr: 5e-06
| > step_time: 0.2913 (0.41031556183641604)
| > loader_time: 0.0092 (0.01549857562238516)
 --> TIME: 2024-11-16 05:37:54 -- STEP: 2250/6312 -- GLOBAL_STEP: 2250
| > loss_text_ce: 0.07375873625278473 (0.08073310671084452)
| > loss_mel_ce: 4.200867652893066 (4.450289535946321)
| > loss: 1.06865656375885 (1.1327556608517984)
| > current_lr: 5e-06
| > step_time: 0.44 (0.41008684084150515)
| > loader_time: 0.013 (0.015436785591973161)
 --> TIME: 2024-11-16 05:38:29 -- STEP: 2300/6312 -- GLOBAL_STEP: 2300
| > loss_text_ce: 0.08081822842359543 (0.08048732298385831)
| > loss_mel_ce: 4.507962226867676 (4.447564959733386)
| > loss: 1.1471951007843018 (1.1320130707906655)
| > current_lr: 5e-06
| > step_time: 0.354 (0.410421716544939)
| > loader_time: 0.029 (0.01538707028264581)
 --> TIME: 2024-11-16 05:39:04 -- STEP: 2350/6312 -- GLOBAL_STEP: 2350
| > loss_text_ce: 0.06630444526672363 (0.0802260493170073)
| > loss_mel_ce: 4.426083564758301 (4.44343660618397)
| > loss: 1.1230969429016113 (1.1309156640539793)
| > current_lr: 5e-06
| > step_time: 0.4744 (0.41050941213648356)
| > loader_time: 0.0185 (0.015371735552523962)
 --> TIME: 2024-11-16 05:39:38 -- STEP: 2400/6312 -- GLOBAL_STEP: 2400
| > loss_text_ce: 0.062453869730234146 (0.07998104143422087)
| > loss_mel_ce: 3.9379587173461914 (4.439550826648876)
| > loss: 1.0001031160354614 (1.1298829669505368)
| > current_lr: 5e-06
| > step_time: 0.3993 (0.4105106142163274)
| > loader_time: 0.0095 (0.015317946573098465)
 --> TIME: 2024-11-16 05:40:11 -- STEP: 2450/6312 -- GLOBAL_STEP: 2450
| > loss_text_ce: 0.08195612579584122 (0.07974749845205512)
| > loss_mel_ce: 4.202617168426514 (4.435498746073978)
| > loss: 1.0711432695388794 (1.1288115608935467)
| > current_lr: 5e-06
| > step_time: 0.3509 (0.4104179735572967)
| > loader_time: 0.0094 (0.015275796773482307)
 --> TIME: 2024-11-16 05:40:45 -- STEP: 2500/6312 -- GLOBAL_STEP: 2500
| > loss_text_ce: 0.07751452922821045 (0.07954578723758451)
| > loss_mel_ce: 4.377040386199951 (4.431676327705386)
| > loss: 1.1136387586593628 (1.1278055286169062)
| > current_lr: 5e-06
| > step_time: 0.3004 (0.4102775672912594)
| > loader_time: 0.0096 (0.015241906833648648)
 --> TIME: 2024-11-16 05:41:18 -- STEP: 2550/6312 -- GLOBAL_STEP: 2550
| > loss_text_ce: 0.0628400668501854 (0.07933548642578064)
| > loss_mel_ce: 4.155436038970947 (4.428821353070878)
| > loss: 1.0545690059661865 (1.1270392096743878)
| > current_lr: 5e-06
| > step_time: 0.2849 (0.4100887838999427)
| > loader_time: 0.0092 (0.015195185810911855)
 --> TIME: 2024-11-16 05:41:53 -- STEP: 2600/6312 -- GLOBAL_STEP: 2600
| > loss_text_ce: 0.05574564263224602 (0.07911101086065163)
| > loss_mel_ce: 4.069448471069336 (4.4254817464718466)
| > loss: 1.0312985181808472 (1.126148188962387)
| > current_lr: 5e-06
| > step_time: 0.272 (0.4103006030046019)
| > loader_time: 0.0263 (0.01514078406187201)
 --> TIME: 2024-11-16 05:42:27 -- STEP: 2650/6312 -- GLOBAL_STEP: 2650
| > loss_text_ce: 0.06676211208105087 (0.07889940332551036)
| > loss_mel_ce: 4.553724765777588 (4.422222133402551)
| > loss: 1.1551216840744019 (1.1252803836903489)
| > current_lr: 5e-06
| > step_time: 0.4238 (0.4103159884686737)
| > loader_time: 0.0091 (0.015103501463836062)
 --> TIME: 2024-11-16 05:43:01 -- STEP: 2700/6312 -- GLOBAL_STEP: 2700
| > loss_text_ce: 0.0795527845621109 (0.07868529184135009)
| > loss_mel_ce: 4.389614105224609 (4.418640684551657)
| > loss: 1.1172916889190674 (1.1243314935101414)
| > current_lr: 5e-06
| > step_time: 0.2478 (0.41029349256444825)
| > loader_time: 0.0271 (0.015098693988941302)
 --> TIME: 2024-11-16 05:43:35 -- STEP: 2750/6312 -- GLOBAL_STEP: 2750
| > loss_text_ce: 0.07262878865003586 (0.07849209816076527)
| > loss_mel_ce: 4.145724296569824 (4.415807526328339)
| > loss: 1.0545883178710938 (1.123574905525555)
| > current_lr: 5e-06
| > step_time: 0.4633 (0.4104896060770205)
| > loader_time: 0.0251 (0.015073683565313134)
 --> TIME: 2024-11-16 05:44:10 -- STEP: 2800/6312 -- GLOBAL_STEP: 2800
| > loss_text_ce: 0.06295352429151535 (0.07828177218325426)
| > loss_mel_ce: 4.119234561920166 (4.412699763008518)
| > loss: 1.0455470085144043 (1.1227453831476835)
| > current_lr: 5e-06
| > step_time: 0.2906 (0.41048514877046827)
| > loader_time: 0.0251 (0.015038793342454062)
 --> TIME: 2024-11-16 05:44:44 -- STEP: 2850/6312 -- GLOBAL_STEP: 2850
| > loss_text_ce: 0.06549959629774094 (0.07808753700240655)
| > loss_mel_ce: 4.092444896697998 (4.409075240837893)
| > loss: 1.0394861698150635 (1.121790693914682)
| > current_lr: 5e-06
| > step_time: 0.4662 (0.41053425772148233)
| > loader_time: 0.0089 (0.01500671947211547)
 --> TIME: 2024-11-16 05:45:17 -- STEP: 2900/6312 -- GLOBAL_STEP: 2900
| > loss_text_ce: 0.0673990473151207 (0.07787926167121209)
| > loss_mel_ce: 4.285707473754883 (4.40628382641693)
| > loss: 1.0882766246795654 (1.1210407715008193)
| > current_lr: 5e-06
| > step_time: 0.243 (0.41026544011872357)
| > loader_time: 0.0108 (0.014998454636540877)
 --> TIME: 2024-11-16 05:45:50 -- STEP: 2950/6312 -- GLOBAL_STEP: 2950
| > loss_text_ce: 0.07621020078659058 (0.07772157309419014)
| > loss_mel_ce: 4.382558345794678 (4.4032394812470725)
| > loss: 1.1146920919418335 (1.1202402630094772)
| > current_lr: 5e-06
| > step_time: 0.4795 (0.4101526127023209)
| > loader_time: 0.0089 (0.014979729167485613)
 --> TIME: 2024-11-16 05:46:23 -- STEP: 3000/6312 -- GLOBAL_STEP: 3000
| > loss_text_ce: 0.071445532143116 (0.077551932901144)
| > loss_mel_ce: 4.200898170471191 (4.400510898749026)
| > loss: 1.0680859088897705 (1.11951570737362)
| > current_lr: 5e-06
| > step_time: 0.2775 (0.4099723690350847)
| > loader_time: 0.0093 (0.01497580401102699)
 --> TIME: 2024-11-16 05:46:57 -- STEP: 3050/6312 -- GLOBAL_STEP: 3050
| > loss_text_ce: 0.05569316819310188 (0.07738125754795111)
| > loss_mel_ce: 4.55543851852417 (4.3980534308855574)
| > loss: 1.152782917022705 (1.1188586715401203)
| > current_lr: 5e-06
| > step_time: 0.4242 (0.40985388873053347)
| > loader_time: 0.01 (0.014954099889661418)
 --> TIME: 2024-11-16 05:47:30 -- STEP: 3100/6312 -- GLOBAL_STEP: 3100
| > loss_text_ce: 0.0658261701464653 (0.07719683897591399)
| > loss_mel_ce: 4.23691987991333 (4.394840703548915)
| > loss: 1.0756864547729492 (1.1180093851973945)
| > current_lr: 5e-06
| > step_time: 0.2835 (0.4096331007249891)
| > loader_time: 0.0095 (0.014918303258957374)
 --> TIME: 2024-11-16 05:48:03 -- STEP: 3150/6312 -- GLOBAL_STEP: 3150
| > loss_text_ce: 0.05693594738841057 (0.0770294156930749)
| > loss_mel_ce: 4.581088066101074 (4.392640948446965)
| > loss: 1.1595059633255005 (1.117417590674901)
| > current_lr: 5e-06
| > step_time: 0.3488 (0.4095654312769569)
| > loader_time: 0.0098 (0.014877082960946192)
 --> TIME: 2024-11-16 05:48:38 -- STEP: 3200/6312 -- GLOBAL_STEP: 3200
| > loss_text_ce: 0.055283624678850174 (0.07686245409888218)
| > loss_mel_ce: 4.443179130554199 (4.389762326851479)
| > loss: 1.1246156692504883 (1.1166561948880576)
| > current_lr: 5e-06
| > step_time: 0.385 (0.40969572730362386)
| > loader_time: 0.0115 (0.0148609425127506)
 --> TIME: 2024-11-16 05:49:11 -- STEP: 3250/6312 -- GLOBAL_STEP: 3250
| > loss_text_ce: 0.06971205770969391 (0.07671435921238025)
| > loss_mel_ce: 4.499574184417725 (4.387151097077582)
| > loss: 1.1423215866088867 (1.115966363686784)
| > current_lr: 5e-06
| > step_time: 0.421 (0.4096534439967226)
| > loader_time: 0.0095 (0.014817639644329339)
 --> TIME: 2024-11-16 05:49:45 -- STEP: 3300/6312 -- GLOBAL_STEP: 3300
| > loss_text_ce: 0.05925505980849266 (0.07657632985914299)
| > loss_mel_ce: 4.109288692474365 (4.384651157061251)
| > loss: 1.0421359539031982 (1.1153068712624656)
| > current_lr: 5e-06
| > step_time: 0.3811 (0.409571605451179)
| > loader_time: 0.0112 (0.014786646149375197)
 --> TIME: 2024-11-16 05:50:19 -- STEP: 3350/6312 -- GLOBAL_STEP: 3350
| > loss_text_ce: 0.07157226651906967 (0.07642515530670761)
| > loss_mel_ce: 4.134022235870361 (4.381695856051648)
| > loss: 1.0513986349105835 (1.1145302522894185)
| > current_lr: 5e-06
| > step_time: 0.5425 (0.40964991754560304)
| > loader_time: 0.011 (0.014754214856161975)
 --> TIME: 2024-11-16 05:50:53 -- STEP: 3400/6312 -- GLOBAL_STEP: 3400
| > loss_text_ce: 0.07942454516887665 (0.07628176516360224)
| > loss_mel_ce: 3.882560968399048 (4.378922877381818)
| > loss: 0.9904963970184326 (1.1138011601216673)
| > current_lr: 5e-06
| > step_time: 0.2869 (0.4096354986639581)
| > loader_time: 0.009 (0.014744288079878837)
 --> TIME: 2024-11-16 05:51:27 -- STEP: 3450/6312 -- GLOBAL_STEP: 3450
| > loss_text_ce: 0.056921374052762985 (0.07613630145032342)
| > loss_mel_ce: 4.273232460021973 (4.375891508848759)
| > loss: 1.0825384855270386 (1.1130069521371893)
| > current_lr: 5e-06
| > step_time: 0.468 (0.409553972603618)
| > loader_time: 0.0094 (0.014744156339894145)
 --> TIME: 2024-11-16 05:52:00 -- STEP: 3500/6312 -- GLOBAL_STEP: 3500
| > loss_text_ce: 0.06732886284589767 (0.07599910227954393)
| > loss_mel_ce: 4.382206439971924 (4.37335870974403)
| > loss: 1.1123838424682617 (1.112339452556204)
| > current_lr: 5e-06
| > step_time: 0.2939 (0.40940657513482204)
| > loader_time: 0.0151 (0.014739869390215168)
 --> TIME: 2024-11-16 05:52:34 -- STEP: 3550/6312 -- GLOBAL_STEP: 3550
| > loss_text_ce: 0.07324003428220749 (0.07584365832658725)
| > loss_mel_ce: 4.132204055786133 (4.369998701861194)
| > loss: 1.0513609647750854 (1.111460589606999)
| > current_lr: 5e-06
| > step_time: 0.5533 (0.40937352462553617)
| > loader_time: 0.0243 (0.014734801977453074)
 --> TIME: 2024-11-16 05:53:08 -- STEP: 3600/6312 -- GLOBAL_STEP: 3600
| > loss_text_ce: 0.06939627230167389 (0.07570108027404386)
| > loss_mel_ce: 4.327859878540039 (4.367811241547251)
| > loss: 1.0993140935897827 (1.1108780800468414)
| > current_lr: 5e-06
| > step_time: 0.3494 (0.40936351968182433)
| > loader_time: 0.0098 (0.014737505250506907)
 --> TIME: 2024-11-16 05:53:41 -- STEP: 3650/6312 -- GLOBAL_STEP: 3650
| > loss_text_ce: 0.06719934195280075 (0.07554710717642149)
| > loss_mel_ce: 4.1143670082092285 (4.364833859744117)
| > loss: 1.04539155960083 (1.110095241314745)
| > current_lr: 5e-06
| > step_time: 0.4064 (0.4091842932243867)
| > loader_time: 0.0247 (0.014706908186821065)
 --> TIME: 2024-11-16 05:54:14 -- STEP: 3700/6312 -- GLOBAL_STEP: 3700
| > loss_text_ce: 0.06182626634836197 (0.07541211481231294)
| > loss_mel_ce: 4.09785270690918 (4.3626119869464)
| > loss: 1.0399197340011597 (1.109506025072691)
| > current_lr: 5e-06
| > step_time: 0.3813 (0.40911137361784217)
| > loader_time: 0.0093 (0.014724237338916653)
 --> TIME: 2024-11-16 05:54:49 -- STEP: 3750/6312 -- GLOBAL_STEP: 3750
| > loss_text_ce: 0.07254482805728912 (0.07527881705462937)
| > loss_mel_ce: 4.584766387939453 (4.36126676953632)
| > loss: 1.16432785987854 (1.1091363963286085)
| > current_lr: 5e-06
| > step_time: 0.4366 (0.40925710175832086)
| > loader_time: 0.0093 (0.014699342219034808)
 --> TIME: 2024-11-16 05:55:23 -- STEP: 3800/6312 -- GLOBAL_STEP: 3800
| > loss_text_ce: 0.0627090334892273 (0.07513165061117007)
| > loss_mel_ce: 4.076931476593018 (4.358279784729589)
| > loss: 1.0349100828170776 (1.1083528585496705)
| > current_lr: 5e-06
| > step_time: 0.3768 (0.40930955441374506)
| > loader_time: 0.0096 (0.014697030406249175)
 --> TIME: 2024-11-16 05:55:57 -- STEP: 3850/6312 -- GLOBAL_STEP: 3850
| > loss_text_ce: 0.07055863738059998 (0.07499221722897775)
| > loss_mel_ce: 4.172000408172607 (4.3562919089701095)
| > loss: 1.06063973903656 (1.1078210312360295)
| > current_lr: 5e-06
| > step_time: 0.5109 (0.40928321522551675)
| > loader_time: 0.0095 (0.014694777463937712)
 --> TIME: 2024-11-16 05:56:31 -- STEP: 3900/6312 -- GLOBAL_STEP: 3900
| > loss_text_ce: 0.05641128867864609 (0.07485600629200542)
| > loss_mel_ce: 4.285912036895752 (4.35403639457163)
| > loss: 1.085580825805664 (1.1072230999439192)
| > current_lr: 5e-06
| > step_time: 0.4081 (0.40925035959635003)
| > loader_time: 0.0094 (0.014678690678034052)
 --> TIME: 2024-11-16 05:57:04 -- STEP: 3950/6312 -- GLOBAL_STEP: 3950
| > loss_text_ce: 0.052118584513664246 (0.07473501495262494)
| > loss_mel_ce: 4.031824111938477 (4.3521925745734515)
| > loss: 1.020985722541809 (1.1067318970342228)
| > current_lr: 5e-06
| > step_time: 0.4334 (0.4091322811947591)
| > loader_time: 0.0249 (0.014660903290857221)
 --> TIME: 2024-11-16 05:57:38 -- STEP: 4000/6312 -- GLOBAL_STEP: 4000
| > loss_text_ce: 0.06849060952663422 (0.07462227894086396)
| > loss_mel_ce: 4.521576881408691 (4.3505150068402125)
| > loss: 1.1475168466567993 (1.1062843210846187)
| > current_lr: 5e-06
| > step_time: 0.3576 (0.40909703320264795)
| > loader_time: 0.0238 (0.014657238006591775)
 --> TIME: 2024-11-16 05:58:12 -- STEP: 4050/6312 -- GLOBAL_STEP: 4050
| > loss_text_ce: 0.06884074211120605 (0.07447727979333317)
| > loss_mel_ce: 4.213171482086182 (4.347906116144141)
| > loss: 1.0705029964447021 (1.1055958486486366)
| > current_lr: 5e-06
| > step_time: 0.3157 (0.40919531951715893)
| > loader_time: 0.0287 (0.01464328795303531)
 --> TIME: 2024-11-16 05:58:47 -- STEP: 4100/6312 -- GLOBAL_STEP: 4100
| > loss_text_ce: 0.06674935668706894 (0.07435059579952465)
| > loss_mel_ce: 4.2534284591674805 (4.345590059815367)
| > loss: 1.0800445079803467 (1.1049851636770298)
| > current_lr: 5e-06
| > step_time: 0.3375 (0.40933356279280103)
| > loader_time: 0.0229 (0.014632555740635548)
 --> TIME: 2024-11-16 05:59:20 -- STEP: 4150/6312 -- GLOBAL_STEP: 4150
| > loss_text_ce: 0.059395864605903625 (0.07423514790563701)
| > loss_mel_ce: 4.043345928192139 (4.343909515013161)
| > loss: 1.025685429573059 (1.1045361654298853)
| > current_lr: 5e-06
| > step_time: 0.4503 (0.40926081904445766)
| > loader_time: 0.029 (0.014619567135730399)
 --> TIME: 2024-11-16 05:59:53 -- STEP: 4200/6312 -- GLOBAL_STEP: 4200
| > loss_text_ce: 0.06306908279657364 (0.07411855739851796)
| > loss_mel_ce: 4.346102714538574 (4.341247700452788)
| > loss: 1.1022928953170776 (1.1038415641869819)
| > current_lr: 5e-06
| > step_time: 0.3538 (0.40916310843967235)
| > loader_time: 0.0094 (0.014604035842986312)
 --> TIME: 2024-11-16 06:00:27 -- STEP: 4250/6312 -- GLOBAL_STEP: 4250
| > loss_text_ce: 0.06433973461389542 (0.07399766470579541)
| > loss_mel_ce: 4.278738498687744 (4.338944509281816)
| > loss: 1.085769534111023 (1.1032355432650627)
| > current_lr: 5e-06
| > step_time: 0.454 (0.4090877949209771)
| > loader_time: 0.009 (0.014572377485387443)
 --> TIME: 2024-11-16 06:01:00 -- STEP: 4300/6312 -- GLOBAL_STEP: 4300
| > loss_text_ce: 0.05814593657851219 (0.07389260702181694)
| > loss_mel_ce: 4.205826759338379 (4.337266996405832)
| > loss: 1.0659931898117065 (1.1027899005801185)
| > current_lr: 5e-06
| > step_time: 0.2972 (0.40903856404992)
| > loader_time: 0.0191 (0.014551900597505768)
 --> TIME: 2024-11-16 06:01:34 -- STEP: 4350/6312 -- GLOBAL_STEP: 4350
| > loss_text_ce: 0.0694170743227005 (0.07378613795956658)
| > loss_mel_ce: 4.324411392211914 (4.335603374283875)
| > loss: 1.0984570980072021 (1.102347377757917)
| > current_lr: 5e-06
| > step_time: 0.5362 (0.40902461797341494)
| > loader_time: 0.0094 (0.014546569407671324)
 --> TIME: 2024-11-16 06:02:07 -- STEP: 4400/6312 -- GLOBAL_STEP: 4400
| > loss_text_ce: 0.06521827727556229 (0.07368954618715427)
| > loss_mel_ce: 4.387492656707764 (4.333973180814211)
| > loss: 1.11317777633667 (1.1019156815247109)
| > current_lr: 5e-06
| > step_time: 0.296 (0.4088628694144159)
| > loader_time: 0.009 (0.014533966671336763)
 --> TIME: 2024-11-16 06:02:41 -- STEP: 4450/6312 -- GLOBAL_STEP: 4450
| > loss_text_ce: 0.06381751596927643 (0.07355284643139733)
| > loss_mel_ce: 4.342920780181885 (4.331906938713576)
| > loss: 1.1016845703125 (1.1013649460974715)
| > current_lr: 5e-06
| > step_time: 0.3512 (0.4087805781203706)
| > loader_time: 0.0107 (0.014527574871363245)
 --> TIME: 2024-11-16 06:03:14 -- STEP: 4500/6312 -- GLOBAL_STEP: 4500
| > loss_text_ce: 0.05934203788638115 (0.07345436779989137)
| > loss_mel_ce: 3.7845301628112793 (4.329669145584097)
| > loss: 0.9609680771827698 (1.100780878225964)
| > current_lr: 5e-06
| > step_time: 0.3461 (0.4087756458388432)
| > loader_time: 0.0102 (0.014510815514458531)
 --> TIME: 2024-11-16 06:03:49 -- STEP: 4550/6312 -- GLOBAL_STEP: 4550
| > loss_text_ce: 0.053755804896354675 (0.07334323911958353)
| > loss_mel_ce: 4.014041423797607 (4.328020704237961)
| > loss: 1.0169492959976196 (1.100340985701636)
| > current_lr: 5e-06
| > step_time: 0.3577 (0.40882894369272077)
| > loader_time: 0.0217 (0.014505649713369498)
 --> TIME: 2024-11-16 06:04:22 -- STEP: 4600/6312 -- GLOBAL_STEP: 4600
| > loss_text_ce: 0.06297792494297028 (0.07326353592879108)
| > loss_mel_ce: 4.330183982849121 (4.326637758275727)
| > loss: 1.0982904434204102 (1.099975323379041)
| > current_lr: 5e-06
| > step_time: 0.3966 (0.4086604889579441)
| > loader_time: 0.009 (0.014507787797761981)
 --> TIME: 2024-11-16 06:04:56 -- STEP: 4650/6312 -- GLOBAL_STEP: 4650
| > loss_text_ce: 0.07134847342967987 (0.07316276236887909)
| > loss_mel_ce: 4.117081642150879 (4.324634909475996)
| > loss: 1.0471075773239136 (1.099449417860278)
| > current_lr: 5e-06
| > step_time: 0.4517 (0.4087583918725292)
| > loader_time: 0.0096 (0.014492903268465413)
 --> TIME: 2024-11-16 06:05:30 -- STEP: 4700/6312 -- GLOBAL_STEP: 4700
| > loss_text_ce: 0.05721760913729668 (0.0730593386363794)
| > loss_mel_ce: 4.198358535766602 (4.322908045991933)
| > loss: 1.0638940334320068 (1.098991846074451)
| > current_lr: 5e-06
| > step_time: 0.4086 (0.4087860292576728)
| > loader_time: 0.0093 (0.014496775941645827)
 --> TIME: 2024-11-16 06:06:04 -- STEP: 4750/6312 -- GLOBAL_STEP: 4750
| > loss_text_ce: 0.05140359327197075 (0.07295399827000354)
| > loss_mel_ce: 3.9544427394866943 (4.321182998105094)
| > loss: 1.0014616250991821 (1.098534248954373)
| > current_lr: 5e-06
| > step_time: 0.5185 (0.4088686547530322)
| > loader_time: 0.0093 (0.014500873666060581)
 --> TIME: 2024-11-16 06:06:39 -- STEP: 4800/6312 -- GLOBAL_STEP: 4800
| > loss_text_ce: 0.05211125686764717 (0.07286836877775697)
| > loss_mel_ce: 4.2441725730896 (4.319516996492938)
| > loss: 1.074070930480957 (1.0980963411182179)
| > current_lr: 5e-06
| > step_time: 0.3902 (0.40893842180569934)
| > loader_time: 0.0277 (0.014510238319635374)
 --> TIME: 2024-11-16 06:07:12 -- STEP: 4850/6312 -- GLOBAL_STEP: 4850
| > loss_text_ce: 0.06591622531414032 (0.07275869951112997)
| > loss_mel_ce: 4.558130264282227 (4.3173986637961)
| > loss: 1.1560115814208984 (1.0975393405648861)
| > current_lr: 5e-06
| > step_time: 0.4899 (0.4087732684243583)
| > loader_time: 0.01 (0.014501859527273259)
 --> TIME: 2024-11-16 06:07:46 -- STEP: 4900/6312 -- GLOBAL_STEP: 4900
| > loss_text_ce: 0.05448588728904724 (0.07265365372735037)
| > loss_mel_ce: 4.140173435211182 (4.315388635421288)
| > loss: 1.0486648082733154 (1.0970105719931287)
| > current_lr: 5e-06
| > step_time: 0.3778 (0.40882356424720867)
| > loader_time: 0.011 (0.01450040909708761)
 --> TIME: 2024-11-16 06:08:19 -- STEP: 4950/6312 -- GLOBAL_STEP: 4950
| > loss_text_ce: 0.052130259573459625 (0.07255298606524581)
| > loss_mel_ce: 4.181326866149902 (4.312852641018955)
| > loss: 1.0583642721176147 (1.0963514064658788)
| > current_lr: 5e-06
| > step_time: 0.3605 (0.408706381393201)
| > loader_time: 0.0086 (0.014485458653382563)
 --> TIME: 2024-11-16 06:08:53 -- STEP: 5000/6312 -- GLOBAL_STEP: 5000
| > loss_text_ce: 0.05143498256802559 (0.07245011596903222)
| > loss_mel_ce: 4.0207366943359375 (4.311106110191345)
| > loss: 1.0180429220199585 (1.095889056217672)
| > current_lr: 5e-06
| > step_time: 0.2857 (0.40869808869361834)
| > loader_time: 0.0094 (0.0144737154483795)
 --> TIME: 2024-11-16 06:09:27 -- STEP: 5050/6312 -- GLOBAL_STEP: 5050
| > loss_text_ce: 0.05508346110582352 (0.07235008097816233)
| > loss_mel_ce: 4.245126247406006 (4.3092398699203365)
| > loss: 1.0750523805618286 (1.0953974873594734)
| > current_lr: 5e-06
| > step_time: 0.4302 (0.408695649109264)
| > loader_time: 0.0087 (0.014457910037276752)
 --> TIME: 2024-11-16 06:10:02 -- STEP: 5100/6312 -- GLOBAL_STEP: 5100
| > loss_text_ce: 0.06178094074130058 (0.07226310529180036)
| > loss_mel_ce: 4.227468490600586 (4.306985260458557)
| > loss: 1.072312355041504 (1.094812091051365)
| > current_lr: 5e-06
| > step_time: 0.4351 (0.4088309333371177)
| > loader_time: 0.0104 (0.014442010533575902)
 --> TIME: 2024-11-16 06:10:37 -- STEP: 5150/6312 -- GLOBAL_STEP: 5150
| > loss_text_ce: 0.07374465465545654 (0.07216262754812422)
| > loss_mel_ce: 4.003246784210205 (4.305393940129329)
| > loss: 1.0192478895187378 (1.0943891415318256)
| > current_lr: 5e-06
| > step_time: 0.4343 (0.4090230508452476)
| > loader_time: 0.0082 (0.014438633687287842)
 --> TIME: 2024-11-16 06:11:11 -- STEP: 5200/6312 -- GLOBAL_STEP: 5200
| > loss_text_ce: 0.06916864961385727 (0.07208807754187076)
| > loss_mel_ce: 4.122498989105225 (4.303462592134112)
| > loss: 1.0479168891906738 (1.0938876670484374)
| > current_lr: 5e-06
| > step_time: 0.2897 (0.4090257065571267)
| > loader_time: 0.0086 (0.014435988527077878)
 --> TIME: 2024-11-16 06:11:44 -- STEP: 5250/6312 -- GLOBAL_STEP: 5250
| > loss_text_ce: 0.06781736761331558 (0.07199530282261846)
| > loss_mel_ce: 3.949334144592285 (4.301925623439612)
| > loss: 1.004287838935852 (1.0934802311602116)
| > current_lr: 5e-06
| > step_time: 0.3562 (0.4089368973232447)
| > loader_time: 0.0085 (0.014413196200416185)
 --> TIME: 2024-11-16 06:12:18 -- STEP: 5300/6312 -- GLOBAL_STEP: 5300
| > loss_text_ce: 0.05325547978281975 (0.07191165995344817)
| > loss_mel_ce: 4.165010452270508 (4.299759465793402)
| > loss: 1.054566502571106 (1.0929177810561006)
| > current_lr: 5e-06
| > step_time: 0.4136 (0.4089586284475502)
| > loader_time: 0.0095 (0.014400883620640001)
 --> TIME: 2024-11-16 06:12:52 -- STEP: 5350/6312 -- GLOBAL_STEP: 5350
| > loss_text_ce: 0.06979038566350937 (0.07181988641132274)
| > loss_mel_ce: 4.238277912139893 (4.298205198573184)
| > loss: 1.077017068862915 (1.0925062709322622)
| > current_lr: 5e-06
| > step_time: 0.4529 (0.40904885060319235)
| > loader_time: 0.0097 (0.014407132139829816)
 --> TIME: 2024-11-16 06:13:27 -- STEP: 5400/6312 -- GLOBAL_STEP: 5400
| > loss_text_ce: 0.0587737150490284 (0.07172015738845994)
| > loss_mel_ce: 4.24463415145874 (4.296289176543566)
| > loss: 1.0758519172668457 (1.0920023332039535)
| > current_lr: 5e-06
| > step_time: 0.3623 (0.40907804414078)
| > loader_time: 0.0243 (0.014398318882341722)
 --> TIME: 2024-11-16 06:13:59 -- STEP: 5450/6312 -- GLOBAL_STEP: 5450
| > loss_text_ce: 0.06310608983039856 (0.07162417379609506)
| > loss_mel_ce: 4.405558109283447 (4.294325209582628)
| > loss: 1.1171660423278809 (1.0914873455533218)
| > current_lr: 5e-06
| > step_time: 0.4332 (0.4089219508039838)
| > loader_time: 0.0095 (0.014387830331784854)
 --> TIME: 2024-11-16 06:14:34 -- STEP: 5500/6312 -- GLOBAL_STEP: 5500
| > loss_text_ce: 0.056705258786678314 (0.07154897228154282)
| > loss_mel_ce: 3.7631776332855225 (4.292801842039291)
| > loss: 0.9549707174301147 (1.0910877033363713)
| > current_lr: 5e-06
| > step_time: 0.3995 (0.4090799097147851)
| > loader_time: 0.0106 (0.014372171445326357)
 --> TIME: 2024-11-16 06:15:08 -- STEP: 5550/6312 -- GLOBAL_STEP: 5550
| > loss_text_ce: 0.06607840210199356 (0.0714620479176174)
| > loss_mel_ce: 4.160889625549316 (4.291189364398928)
| > loss: 1.0567419528961182 (1.0906628528801185)
| > current_lr: 5e-06
| > step_time: 0.3694 (0.40913208570566223)
| > loader_time: 0.0215 (0.014360815469209124)
 --> TIME: 2024-11-16 06:15:42 -- STEP: 5600/6312 -- GLOBAL_STEP: 5600
| > loss_text_ce: 0.06696689128875732 (0.07138614077054499)
| > loss_mel_ce: 3.8666470050811768 (4.289770946119514)
| > loss: 0.9834034442901611 (1.0902892715164607)
| > current_lr: 5e-06
| > step_time: 0.2754 (0.40908497640064745)
| > loader_time: 0.0092 (0.014345082087176171)
 --> TIME: 2024-11-16 06:16:16 -- STEP: 5650/6312 -- GLOBAL_STEP: 5650
| > loss_text_ce: 0.07208910584449768 (0.07131167631431493)
| > loss_mel_ce: 3.9800496101379395 (4.288743215957576)
| > loss: 1.013034701347351 (1.0900137228248417)
| > current_lr: 5e-06
| > step_time: 0.3638 (0.4091321178875134)
| > loader_time: 0.0087 (0.01432944652253546)
 --> TIME: 2024-11-16 06:16:51 -- STEP: 5700/6312 -- GLOBAL_STEP: 5700
| > loss_text_ce: 0.07122498005628586 (0.07123361219243524)
| > loss_mel_ce: 4.174830913543701 (4.28715633091174)
| > loss: 1.0615140199661255 (1.0895974855673967)
| > current_lr: 5e-06
| > step_time: 0.4473 (0.409220797346349)
| > loader_time: 0.0095 (0.014324130426373384)
 --> TIME: 2024-11-16 06:17:25 -- STEP: 5750/6312 -- GLOBAL_STEP: 5750
| > loss_text_ce: 0.06635512411594391 (0.07115068622970068)
| > loss_mel_ce: 4.19437837600708 (4.285292172349018)
| > loss: 1.065183401107788 (1.0891107145185073)
| > current_lr: 5e-06
| > step_time: 0.4611 (0.40924008920918303)
| > loader_time: 0.0094 (0.01432432365417479)
 --> TIME: 2024-11-16 06:18:00 -- STEP: 5800/6312 -- GLOBAL_STEP: 5800
| > loss_text_ce: 0.06358488649129868 (0.07107346350648287)
| > loss_mel_ce: 3.9966378211975098 (4.283827749531844)
| > loss: 1.0150556564331055 (1.0887253030929096)
| > current_lr: 5e-06
| > step_time: 0.4364 (0.4093877470082246)
| > loader_time: 0.0095 (0.014333589652488958)
 --> TIME: 2024-11-16 06:18:33 -- STEP: 5850/6312 -- GLOBAL_STEP: 5850
| > loss_text_ce: 0.05050116404891014 (0.07099425212822413)
| > loss_mel_ce: 3.5622377395629883 (4.2817744299489195)
| > loss: 0.903184711933136 (1.0881921703285669)
| > current_lr: 5e-06
| > step_time: 0.3603 (0.4092817269023663)
| > loader_time: 0.0092 (0.014323999046260461)
 --> TIME: 2024-11-16 06:19:07 -- STEP: 5900/6312 -- GLOBAL_STEP: 5900
| > loss_text_ce: 0.050284624099731445 (0.07092147023811693)
| > loss_mel_ce: 4.058730602264404 (4.280450124094042)
| > loss: 1.0272538661956787 (1.0878428984193511)
| > current_lr: 5e-06
| > step_time: 0.3532 (0.4092318143278862)
| > loader_time: 0.0087 (0.014308963508929223)
 --> TIME: 2024-11-16 06:19:41 -- STEP: 5950/6312 -- GLOBAL_STEP: 5950
| > loss_text_ce: 0.06671793013811111 (0.07084444974650866)
| > loss_mel_ce: 4.160628795623779 (4.278976248092009)
| > loss: 1.056836724281311 (1.0874551743018561)
| > current_lr: 5e-06
| > step_time: 0.3473 (0.40925679868008874)
| > loader_time: 0.0318 (0.014314976419721317)
 --> TIME: 2024-11-16 06:20:15 -- STEP: 6000/6312 -- GLOBAL_STEP: 6000
| > loss_text_ce: 0.05832890421152115 (0.07078928446086763)
| > loss_mel_ce: 4.321648597717285 (4.277603690147399)
| > loss: 1.0949944257736206 (1.087098243524635)
| > current_lr: 5e-06
| > step_time: 0.2866 (0.4092691661516822)
| > loader_time: 0.0253 (0.0143069442907969)
 --> TIME: 2024-11-16 06:20:49 -- STEP: 6050/6312 -- GLOBAL_STEP: 6050
| > loss_text_ce: 0.05493095889687538 (0.0707172002058386)
| > loss_mel_ce: 4.286261558532715 (4.276363732440414)
| > loss: 1.0852981805801392 (1.0867702330045501)
| > current_lr: 5e-06
| > step_time: 0.4312 (0.40927833407378356)
| > loader_time: 0.0268 (0.014302292618869736)
 --> TIME: 2024-11-16 06:21:22 -- STEP: 6100/6312 -- GLOBAL_STEP: 6100
| > loss_text_ce: 0.06370208412408829 (0.07064931101547425)
| > loss_mel_ce: 4.268980026245117 (4.274962920556308)
| > loss: 1.0831705331802368 (1.086403057751112)
| > current_lr: 5e-06
| > step_time: 0.3756 (0.40915461106378487)
| > loader_time: 0.0096 (0.014292235608960747)
 --> TIME: 2024-11-16 06:21:56 -- STEP: 6150/6312 -- GLOBAL_STEP: 6150
| > loss_text_ce: 0.05991522967815399 (0.07057992027724198)
| > loss_mel_ce: 4.08201789855957 (4.273300347638326)
| > loss: 1.0354832410812378 (1.0859700667955077)
| > current_lr: 5e-06
| > step_time: 0.3615 (0.40919754834679045)
| > loader_time: 0.0085 (0.014269081208764038)
 --> TIME: 2024-11-16 06:22:30 -- STEP: 6200/6312 -- GLOBAL_STEP: 6200
| > loss_text_ce: 0.06830406188964844 (0.07050453474564926)
| > loss_mel_ce: 4.103909492492676 (4.272036592345083)
| > loss: 1.043053388595581 (1.0856352816281825)
| > current_lr: 5e-06
| > step_time: 0.3865 (0.4092287628496843)
| > loader_time: 0.009 (0.014271417125578835)
 --> TIME: 2024-11-16 06:23:05 -- STEP: 6250/6312 -- GLOBAL_STEP: 6250
| > loss_text_ce: 0.06871412694454193 (0.07044121114611647)
| > loss_mel_ce: 3.809135675430298 (4.2705220735168465)
| > loss: 0.9694624543190002 (1.0852408210563707)
| > current_lr: 5e-06
| > step_time: 0.3568 (0.40932916477203335)
| > loader_time: 0.0089 (0.014272699623107897)
> CHECKPOINT : checkpoints/GPT_XTTS_FT-November-16-2024_05+12AM-4b65e59/checkpoint_6250.pth
 --> TIME: 2024-11-16 06:24:20 -- STEP: 6300/6312 -- GLOBAL_STEP: 6300
| > loss_text_ce: 0.0656561553478241 (0.07036574914578428)
| > loss_mel_ce: 4.26099157333374 (4.269376304225314)
| > loss: 1.0816619396209717 (1.0849355132523082)
| > current_lr: 5e-06
| > step_time: 0.3998 (0.40939678468401436)
| > loader_time: 0.0237 (0.014268293872712137)
 > EVALUATION 
--> EVAL PERFORMANCE
| > avg_loader_time: 0.4727956673194622 (+0)
| > avg_loss_text_ce: 0.059023432733341194 (+0)
| > avg_loss_mel_ce: 3.999843802945368 (+0)
| > avg_loss: 4.058867229812446 (+0)
 > EPOCH: 1/2
--> checkpoints/GPT_XTTS_FT-November-16-2024_05+12AM-4b65e59
 > TRAINING (2024-11-16 06:28:34) 
 --> TIME: 2024-11-16 06:29:01 -- STEP: 38/6312 -- GLOBAL_STEP: 6350
| > loss_text_ce: 0.06461291760206223 (0.060702322148963025)
| > loss_mel_ce: 3.875960111618042 (4.02208075397893)
| > loss: 0.98514324426651 (1.0206957647674961)
| > current_lr: 5e-06
| > step_time: 0.3673 (0.40007398630443375)
| > loader_time: 0.0094 (0.014848502058731882)
 --> TIME: 2024-11-16 06:29:35 -- STEP: 88/6312 -- GLOBAL_STEP: 6400
| > loss_text_ce: 0.05987628176808357 (0.060063213321634314)
| > loss_mel_ce: 4.10789680480957 (4.050400972366332)
| > loss: 1.0419433116912842 (1.0276160429824477)
| > current_lr: 5e-06
| > step_time: 0.2947 (0.4039698893373664)
| > loader_time: 0.0111 (0.014401760968295011)
 --> TIME: 2024-11-16 06:30:10 -- STEP: 138/6312 -- GLOBAL_STEP: 6450
| > loss_text_ce: 0.0664081797003746 (0.06102129034158112)
| > loss_mel_ce: 4.163527965545654 (4.078809123108354)
| > loss: 1.0574840307235718 (1.0349576015403301)
| > current_lr: 5e-06
| > step_time: 0.356 (0.40994463105132617)
| > loader_time: 0.0269 (0.013736412145089413)
 --> TIME: 2024-11-16 06:30:44 -- STEP: 188/6312 -- GLOBAL_STEP: 6500
| > loss_text_ce: 0.05301523953676224 (0.06157133226936802)
| > loss_mel_ce: 3.9721477031707764 (4.088198659267832)
| > loss: 1.0062907934188843 (1.0374424964823612)
| > current_lr: 5e-06
| > step_time: 0.3925 (0.41156817370272697)
| > loader_time: 0.0096 (0.013689381010988926)
 --> TIME: 2024-11-16 06:31:17 -- STEP: 238/6312 -- GLOBAL_STEP: 6550
| > loss_text_ce: 0.06394050270318985 (0.061334454117953276)
| > loss_mel_ce: 3.8712046146392822 (4.087316957842404)
| > loss: 0.9837862849235535 (1.0371628536396662)
| > current_lr: 5e-06
| > step_time: 0.4595 (0.4098687282129496)
| > loader_time: 0.013 (0.013522876410925087)
 --> TIME: 2024-11-16 06:31:51 -- STEP: 288/6312 -- GLOBAL_STEP: 6600
| > loss_text_ce: 0.05422750860452652 (0.06131800967786047)
| > loss_mel_ce: 4.28978967666626 (4.09240580101808)
| > loss: 1.0860042572021484 (1.0384309521565829)
| > current_lr: 5e-06
| > step_time: 0.3856 (0.4089812371465895)
| > loader_time: 0.0093 (0.013619658019807603)
 --> TIME: 2024-11-16 06:32:26 -- STEP: 338/6312 -- GLOBAL_STEP: 6650
| > loss_text_ce: 0.0535762645304203 (0.060919020865002324)
| > loss_mel_ce: 4.08420467376709 (4.081699268352355)
| > loss: 1.0344452857971191 (1.0356545716347776)
| > current_lr: 5e-06
| > step_time: 0.4715 (0.4113177724138519)
| > loader_time: 0.0089 (0.013641443478285208)
 --> TIME: 2024-11-16 06:32:59 -- STEP: 388/6312 -- GLOBAL_STEP: 6700
| > loss_text_ce: 0.05807321518659592 (0.06109079219324073)
| > loss_mel_ce: 4.171273231506348 (4.078862885224457)
| > loss: 1.0573365688323975 (1.034988418375094)
| > current_lr: 5e-06
| > step_time: 0.3798 (0.40995099802607116)
| > loader_time: 0.009 (0.013542834016465648)
 --> TIME: 2024-11-16 06:33:33 -- STEP: 438/6312 -- GLOBAL_STEP: 6750
| > loss_text_ce: 0.07202156633138657 (0.06105406962539235)
| > loss_mel_ce: 4.101592540740967 (4.078960503617387)
| > loss: 1.0434035062789917 (1.035003642513327)
| > current_lr: 5e-06
| > step_time: 0.357 (0.4094774886353375)
| > loader_time: 0.0092 (0.013421656334236877)
 --> TIME: 2024-11-16 06:34:07 -- STEP: 488/6312 -- GLOBAL_STEP: 6800
| > loss_text_ce: 0.06619283556938171 (0.061208236168642516)
| > loss_mel_ce: 4.200325965881348 (4.07613980037268)
| > loss: 1.0666296482086182 (1.034337008341413)
| > current_lr: 5e-06
| > step_time: 0.3579 (0.4097981697223226)
| > loader_time: 0.0103 (0.013362576727007256)
 --> TIME: 2024-11-16 06:34:41 -- STEP: 538/6312 -- GLOBAL_STEP: 6850
| > loss_text_ce: 0.06019800528883934 (0.06108222343027592)
| > loss_mel_ce: 4.0644307136535645 (4.075714612538937)
| > loss: 1.03115713596344 (1.034199208353531)
| > current_lr: 5e-06
| > step_time: 0.4224 (0.41029527949577815)
| > loader_time: 0.0239 (0.01347895936008723)
 --> TIME: 2024-11-16 06:35:14 -- STEP: 588/6312 -- GLOBAL_STEP: 6900
| > loss_text_ce: 0.06245969608426094 (0.060862943929220946)
| > loss_mel_ce: 3.93929386138916 (4.07212761754082)
| > loss: 1.0004383325576782 (1.0332476392286971)
| > current_lr: 5e-06
| > step_time: 0.282 (0.4093630407132259)
| > loader_time: 0.025 (0.01334590936193661)
 --> TIME: 2024-11-16 06:35:48 -- STEP: 638/6312 -- GLOBAL_STEP: 6950
| > loss_text_ce: 0.06438322365283966 (0.060806575409344)
| > loss_mel_ce: 4.161640644073486 (4.069837069810378)
| > loss: 1.0565059185028076 (1.032660910141505)
| > current_lr: 5e-06
| > step_time: 0.5132 (0.4093374959353742)
| > loader_time: 0.0243 (0.013343822246061229)
 --> TIME: 2024-11-16 06:36:23 -- STEP: 688/6312 -- GLOBAL_STEP: 7000
| > loss_text_ce: 0.06770768761634827 (0.06084919194041123)
| > loss_mel_ce: 4.280162334442139 (4.0721145265324195)
| > loss: 1.0869674682617188 (1.0332409285528712)
| > current_lr: 5e-06
| > step_time: 0.3551 (0.41032783236614484)
| > loader_time: 0.01 (0.013346098536668826)
 --> TIME: 2024-11-16 06:36:57 -- STEP: 738/6312 -- GLOBAL_STEP: 7050
| > loss_text_ce: 0.062325846403837204 (0.060812865356160054)
| > loss_mel_ce: 4.0534892082214355 (4.072953837027722)
| > loss: 1.0289537906646729 (1.0334416745800954)
| > current_lr: 5e-06
| > step_time: 0.354 (0.410306325449853)
| > loader_time: 0.0085 (0.013329376373187641)
 --> TIME: 2024-11-16 06:37:30 -- STEP: 788/6312 -- GLOBAL_STEP: 7100
| > loss_text_ce: 0.06013082340359688 (0.06074129811814291)
| > loss_mel_ce: 4.197961807250977 (4.072022781154232)
| > loss: 1.0645231008529663 (1.0331910186763948)
| > current_lr: 5e-06
| > step_time: 0.3727 (0.40980947441255977)
| > loader_time: 0.0091 (0.013393553380433682)
 --> TIME: 2024-11-16 06:38:03 -- STEP: 838/6312 -- GLOBAL_STEP: 7150
| > loss_text_ce: 0.06987680494785309 (0.060661213837012756)
| > loss_mel_ce: 3.555269718170166 (4.071283328789237)
| > loss: 0.9062866568565369 (1.0329861342195683)
| > current_lr: 5e-06
| > step_time: 0.3015 (0.40903820854951767)
| > loader_time: 0.0081 (0.013386468329691379)
 --> TIME: 2024-11-16 06:38:36 -- STEP: 888/6312 -- GLOBAL_STEP: 7200
| > loss_text_ce: 0.05093912407755852 (0.06057515958122708)
| > loss_mel_ce: 4.062649250030518 (4.070378622493232)
| > loss: 1.0283970832824707 (1.0327384441688245)
| > current_lr: 5e-06
| > step_time: 0.3569 (0.40796353151132386)
| > loader_time: 0.0212 (0.013525260729832697)
 --> TIME: 2024-11-16 06:39:10 -- STEP: 938/6312 -- GLOBAL_STEP: 7250
| > loss_text_ce: 0.06498738378286362 (0.060598134224031014)
| > loss_mel_ce: 4.062078475952148 (4.066221345970627)
| > loss: 1.031766414642334 (1.0317048685255832)
| > current_lr: 5e-06
| > step_time: 0.3449 (0.40785937039836884)
| > loader_time: 0.0099 (0.013593411394782162)
 --> TIME: 2024-11-16 06:39:45 -- STEP: 988/6312 -- GLOBAL_STEP: 7300
| > loss_text_ce: 0.05883764103055 (0.06059516251826214)
| > loss_mel_ce: 4.1269402503967285 (4.065927768284498)
| > loss: 1.0464444160461426 (1.0316307313649744)
| > current_lr: 5e-06
| > step_time: 0.4164 (0.4088122021814106)
| > loader_time: 0.0276 (0.013613617371933666)
 --> TIME: 2024-11-16 06:40:18 -- STEP: 1038/6312 -- GLOBAL_STEP: 7350
| > loss_text_ce: 0.06338584423065186 (0.06054397400016385)
| > loss_mel_ce: 4.006345748901367 (4.065699258980727)
| > loss: 1.0174329280853271 (1.031560807271728)
| > current_lr: 5e-06
| > step_time: 0.426 (0.4085977936526024)
| > loader_time: 0.0087 (0.013635747694555738)
 --> TIME: 2024-11-16 06:40:53 -- STEP: 1088/6312 -- GLOBAL_STEP: 7400
| > loss_text_ce: 0.05104463919997215 (0.06049035054028911)
| > loss_mel_ce: 4.189610004425049 (4.064654957722219)
| > loss: 1.0601636171340942 (1.0312863261274539)
| > current_lr: 5e-06
| > step_time: 0.3708 (0.40888750049121236)
| > loader_time: 0.0092 (0.01376271554652383)
 --> TIME: 2024-11-16 06:41:27 -- STEP: 1138/6312 -- GLOBAL_STEP: 7450
| > loss_text_ce: 0.055655501782894135 (0.060449371238767785)
| > loss_mel_ce: 4.213855743408203 (4.06341891171434)
| > loss: 1.0673778057098389 (1.0309670698454159)
| > current_lr: 5e-06
| > step_time: 0.3454 (0.40883079522104376)
| > loader_time: 0.0091 (0.013761988214114014)
 --> TIME: 2024-11-16 06:42:01 -- STEP: 1188/6312 -- GLOBAL_STEP: 7500
| > loss_text_ce: 0.057377204298973083 (0.06045542004508803)
| > loss_mel_ce: 4.132730960845947 (4.06164590638094)
| > loss: 1.0475270748138428 (1.0305253305419113)
| > current_lr: 5e-06
| > step_time: 0.2958 (0.40905740445711786)
| > loader_time: 0.0295 (0.013837169517170305)
 --> TIME: 2024-11-16 06:42:34 -- STEP: 1238/6312 -- GLOBAL_STEP: 7550
| > loss_text_ce: 0.07158641517162323 (0.06042795297005161)
| > loss_mel_ce: 4.151811599731445 (4.0606079839158005)
| > loss: 1.055849552154541 (1.0302589834217863)
| > current_lr: 5e-06
| > step_time: 0.3549 (0.4084074676325709)
| > loader_time: 0.0096 (0.013835818201351633)
 --> TIME: 2024-11-16 06:43:07 -- STEP: 1288/6312 -- GLOBAL_STEP: 7600
| > loss_text_ce: 0.05710970237851143 (0.060399995159404064)
| > loss_mel_ce: 3.98812198638916 (4.061185172625956)
| > loss: 1.011307954788208 (1.0303962909564457)
| > current_lr: 5e-06
| > step_time: 0.4316 (0.4084003915697889)
| > loader_time: 0.0093 (0.013819315974016373)
 --> TIME: 2024-11-16 06:43:43 -- STEP: 1338/6312 -- GLOBAL_STEP: 7650
| > loss_text_ce: 0.06700874865055084 (0.06034687396990879)
| > loss_mel_ce: 4.2809062004089355 (4.064070186985627)
| > loss: 1.086978793144226 (1.0311042643715973)
| > current_lr: 5e-06
| > step_time: 0.557 (0.40921953601152883)
| > loader_time: 0.0097 (0.01376199348090476)
 --> TIME: 2024-11-16 06:44:16 -- STEP: 1388/6312 -- GLOBAL_STEP: 7700
| > loss_text_ce: 0.05914709344506264 (0.06034186024819997)
| > loss_mel_ce: 4.139700412750244 (4.0624835060034385)
| > loss: 1.04971182346344 (1.0307063410536346)
| > current_lr: 5e-06
| > step_time: 0.2607 (0.40897392711309577)
| > loader_time: 0.0092 (0.013829185571038419)
 --> TIME: 2024-11-16 06:44:51 -- STEP: 1438/6312 -- GLOBAL_STEP: 7750
| > loss_text_ce: 0.05866221338510513 (0.060298624680337386)
| > loss_mel_ce: 4.008269786834717 (4.061800692111291)
| > loss: 1.0167330503463745 (1.0305248286694235)
| > current_lr: 5e-06
| > step_time: 0.4675 (0.4096152319861716)
| > loader_time: 0.009 (0.013833912092057659)
 --> TIME: 2024-11-16 06:45:26 -- STEP: 1488/6312 -- GLOBAL_STEP: 7800
| > loss_text_ce: 0.054083067923784256 (0.06026671903197884)
| > loss_mel_ce: 4.021108627319336 (4.061384140003108)
| > loss: 1.0187978744506836 (1.0304127144477038)
| > current_lr: 5e-06
| > step_time: 0.3726 (0.4097571959418633)
| > loader_time: 0.0089 (0.013780191060035463)
 --> TIME: 2024-11-16 06:46:00 -- STEP: 1538/6312 -- GLOBAL_STEP: 7850
| > loss_text_ce: 0.06187654659152031 (0.06025318605449869)
| > loss_mel_ce: 4.009779453277588 (4.0605478624683675)
| > loss: 1.017914056777954 (1.0302002620247435)
| > current_lr: 5e-06
| > step_time: 0.3683 (0.4099889979715928)
| > loader_time: 0.0088 (0.013777464511955672)
 --> TIME: 2024-11-16 06:46:34 -- STEP: 1588/6312 -- GLOBAL_STEP: 7900
| > loss_text_ce: 0.051662810146808624 (0.06024738635764614)
| > loss_mel_ce: 4.061514377593994 (4.059466955944034)
| > loss: 1.028294324874878 (1.0299285857067855)
| > current_lr: 5e-06
| > step_time: 0.3931 (0.4098440890648501)
| > loader_time: 0.01 (0.013768066086757097)
 --> TIME: 2024-11-16 06:47:06 -- STEP: 1638/6312 -- GLOBAL_STEP: 7950
| > loss_text_ce: 0.06114071235060692 (0.060203703732135384)
| > loss_mel_ce: 4.100691318511963 (4.059145502700616)
| > loss: 1.0404579639434814 (1.0298373019447795)
| > current_lr: 5e-06
| > step_time: 0.4767 (0.4093766650582988)
| > loader_time: 0.0263 (0.013765025342631548)
 --> TIME: 2024-11-16 06:47:40 -- STEP: 1688/6312 -- GLOBAL_STEP: 8000
| > loss_text_ce: 0.06431081146001816 (0.06016366682522952)
| > loss_mel_ce: 3.806918144226074 (4.057590306652662)
| > loss: 0.9678072333335876 (1.0294384936938872)
| > current_lr: 5e-06
| > step_time: 0.2845 (0.409354361877622)
| > loader_time: 0.009 (0.013794705907315444)
 --> TIME: 2024-11-16 06:48:15 -- STEP: 1738/6312 -- GLOBAL_STEP: 8050
| > loss_text_ce: 0.0711505338549614 (0.06017422407728344)
| > loss_mel_ce: 4.113681316375732 (4.05669700564394)
| > loss: 1.0462079048156738 (1.0292178078268375)
| > current_lr: 5e-06
| > step_time: 0.453 (0.4093872337538846)
| > loader_time: 0.009 (0.0138500592788152)
 --> TIME: 2024-11-16 06:48:49 -- STEP: 1788/6312 -- GLOBAL_STEP: 8100
| > loss_text_ce: 0.05997205525636673 (0.0601311431222614)
| > loss_mel_ce: 3.856219530105591 (4.057362357108646)
| > loss: 0.9790478944778442 (1.029373375628081)
| > current_lr: 5e-06
| > step_time: 0.2736 (0.4097771305899223)
| > loader_time: 0.0085 (0.013821219971249304)
 --> TIME: 2024-11-16 06:49:23 -- STEP: 1838/6312 -- GLOBAL_STEP: 8150
| > loss_text_ce: 0.07021951675415039 (0.060119147154059714)
| > loss_mel_ce: 3.786098003387451 (4.055093374942403)
| > loss: 0.9640793800354004 (1.0288031311042893)
| > current_lr: 5e-06
| > step_time: 0.3479 (0.4097467099232303)
| > loader_time: 0.0088 (0.01383630936241773)
 --> TIME: 2024-11-16 06:49:57 -- STEP: 1888/6312 -- GLOBAL_STEP: 8200
| > loss_text_ce: 0.05881601572036743 (0.06007208409955946)
| > loss_mel_ce: 4.314101696014404 (4.054135776298525)
| > loss: 1.0932294130325317 (1.0285519657309237)
| > current_lr: 5e-06
| > step_time: 0.3739 (0.4095858199111484)
| > loader_time: 0.0092 (0.013840392231941227)
 --> TIME: 2024-11-16 06:50:31 -- STEP: 1938/6312 -- GLOBAL_STEP: 8250
| > loss_text_ce: 0.06879647821187973 (0.06003144581036179)
| > loss_mel_ce: 4.289897441864014 (4.055029554755813)
| > loss: 1.0896735191345215 (1.0287652508085563)
| > current_lr: 5e-06
| > step_time: 0.3499 (0.40949468826730906)
| > loader_time: 0.0089 (0.013870236182237438)
 --> TIME: 2024-11-16 06:51:04 -- STEP: 1988/6312 -- GLOBAL_STEP: 8300
| > loss_text_ce: 0.054291412234306335 (0.060023677745230604)
| > loss_mel_ce: 3.6199328899383545 (4.054344044964551)
| > loss: 0.9185560941696167 (1.0285919314958638)
| > current_lr: 5e-06
| > step_time: 0.2677 (0.4094037359629117)
| > loader_time: 0.0216 (0.013850947620883078)
 --> TIME: 2024-11-16 06:51:39 -- STEP: 2038/6312 -- GLOBAL_STEP: 8350
| > loss_text_ce: 0.054777245968580246 (0.05999266742650964)
| > loss_mel_ce: 4.132874011993408 (4.054476842562036)
| > loss: 1.0469127893447876 (1.02861737827674)
| > current_lr: 5e-06
| > step_time: 0.5299 (0.409605167213904)
| > loader_time: 0.0214 (0.013840161549097426)
 --> TIME: 2024-11-16 06:52:13 -- STEP: 2088/6312 -- GLOBAL_STEP: 8400
| > loss_text_ce: 0.06440303474664688 (0.05995479061971221)
| > loss_mel_ce: 4.1905317306518555 (4.053660621474065)
| > loss: 1.063733696937561 (1.028403853947628)
| > current_lr: 5e-06
| > step_time: 0.376 (0.4097759833281066)
| > loader_time: 0.0104 (0.013796352906245384)
 --> TIME: 2024-11-16 06:52:46 -- STEP: 2138/6312 -- GLOBAL_STEP: 8450
| > loss_text_ce: 0.07034755498170853 (0.05992903007530684)
| > loss_mel_ce: 4.193737030029297 (4.053722397315371)
| > loss: 1.066021203994751 (1.0284128579388572)
| > current_lr: 5e-06
| > step_time: 0.4717 (0.4096192498425661)
| > loader_time: 0.0094 (0.013791540942535546)
 --> TIME: 2024-11-16 06:53:20 -- STEP: 2188/6312 -- GLOBAL_STEP: 8500
| > loss_text_ce: 0.05408677086234093 (0.059879033490879204)
| > loss_mel_ce: 4.149914264678955 (4.052397354865196)
| > loss: 1.0510002374649048 (1.028069098100362)
| > current_lr: 5e-06
| > step_time: 0.29 (0.4095512692845501)
| > loader_time: 0.0093 (0.013762257216835373)
 --> TIME: 2024-11-16 06:53:54 -- STEP: 2238/6312 -- GLOBAL_STEP: 8550
| > loss_text_ce: 0.06153767928481102 (0.05986889598556559)
| > loss_mel_ce: 4.256723880767822 (4.051919212822822)
| > loss: 1.079565405845642 (1.027947028211653)
| > current_lr: 5e-06
| > step_time: 0.429 (0.40945264790290387)
| > loader_time: 0.0093 (0.013750612575950318)
 --> TIME: 2024-11-16 06:54:28 -- STEP: 2288/6312 -- GLOBAL_STEP: 8600
| > loss_text_ce: 0.0621483251452446 (0.05987862697672227)
| > loss_mel_ce: 3.812927722930908 (4.051912293046494)
| > loss: 0.9687690138816833 (1.0279477308732192)
| > current_lr: 5e-06
| > step_time: 0.2534 (0.4096074111603359)
| > loader_time: 0.01 (0.013731470370626114)
 --> TIME: 2024-11-16 06:55:03 -- STEP: 2338/6312 -- GLOBAL_STEP: 8650
| > loss_text_ce: 0.06740766018629074 (0.05985245115296567)
| > loss_mel_ce: 3.977213144302368 (4.050928030185345)
| > loss: 1.0111552476882935 (1.0276951212929868)
| > current_lr: 5e-06
| > step_time: 0.4612 (0.4099254619371598)
| > loader_time: 0.0101 (0.01377462463770492)
 --> TIME: 2024-11-16 06:55:37 -- STEP: 2388/6312 -- GLOBAL_STEP: 8700
| > loss_text_ce: 0.0568319596350193 (0.05981659096114997)
| > loss_mel_ce: 3.9491488933563232 (4.049785528250856)
| > loss: 1.0014952421188354 (1.0274005309620862)
| > current_lr: 5e-06
| > step_time: 0.2945 (0.4098740187122596)
| > loader_time: 0.0094 (0.013788359748258863)
 --> TIME: 2024-11-16 06:56:11 -- STEP: 2438/6312 -- GLOBAL_STEP: 8750
| > loss_text_ce: 0.04898742213845253 (0.059774245171394354)
| > loss_mel_ce: 3.7578701972961426 (4.0487543570398605)
| > loss: 0.9517143964767456 (1.027132151628534)
| > current_lr: 5e-06
| > step_time: 0.4704 (0.41003728147605295)
| > loader_time: 0.0294 (0.013824922907440461)
 --> TIME: 2024-11-16 06:56:45 -- STEP: 2488/6312 -- GLOBAL_STEP: 8800
| > loss_text_ce: 0.05406889319419861 (0.05973573112590805)
| > loss_mel_ce: 3.953127861022949 (4.047984661205005)
| > loss: 1.00179922580719 (1.0269300992513741)
| > current_lr: 5e-06
| > step_time: 0.2745 (0.4101196884917293)
| > loader_time: 0.0247 (0.013813281078430618)
 --> TIME: 2024-11-16 06:57:20 -- STEP: 2538/6312 -- GLOBAL_STEP: 8850
| > loss_text_ce: 0.06322132796049118 (0.05968210284896868)
| > loss_mel_ce: 3.945927858352661 (4.046728228658447)
| > loss: 1.0022872686386108 (1.0266025841846687)
| > current_lr: 5e-06
| > step_time: 0.3435 (0.4102271203453659)
| > loader_time: 0.0088 (0.01385562310921479)
 --> TIME: 2024-11-16 06:57:55 -- STEP: 2588/6312 -- GLOBAL_STEP: 8900
| > loss_text_ce: 0.05226665735244751 (0.059674023578614205)
| > loss_mel_ce: 4.225100994110107 (4.046651196166559)
| > loss: 1.0693418979644775 (1.0265813062235194)
| > current_lr: 5e-06
| > step_time: 0.4338 (0.41044839078897305)
| > loader_time: 0.0096 (0.013852812315249553)
 --> TIME: 2024-11-16 06:58:29 -- STEP: 2638/6312 -- GLOBAL_STEP: 8950
| > loss_text_ce: 0.06292121857404709 (0.05960924186976095)
| > loss_mel_ce: 3.9877655506134033 (4.045198585428539)
| > loss: 1.012671709060669 (1.0262019582053197)
| > current_lr: 5e-06
| > step_time: 0.4535 (0.41049627321429827)
| > loader_time: 0.0096 (0.01386097157155139)
 --> TIME: 2024-11-16 06:59:03 -- STEP: 2688/6312 -- GLOBAL_STEP: 9000
| > loss_text_ce: 0.06369476765394211 (0.05959099710924504)
| > loss_mel_ce: 4.131172180175781 (4.044465573149766)
| > loss: 1.0487167835235596 (1.0260141439780213)
| > current_lr: 5e-06
| > step_time: 0.4911 (0.4105920066081342)
| > loader_time: 0.0107 (0.013879167625591868)
 --> TIME: 2024-11-16 06:59:37 -- STEP: 2738/6312 -- GLOBAL_STEP: 9050
| > loss_text_ce: 0.04925151169300079 (0.05955939117745597)
| > loss_mel_ce: 3.8036983013153076 (4.043317901469903)
| > loss: 0.963237464427948 (1.0257193244574605)
| > current_lr: 5e-06
| > step_time: 0.4493 (0.410574588486537)
| > loader_time: 0.0243 (0.013927162782043501)
 --> TIME: 2024-11-16 07:00:12 -- STEP: 2788/6312 -- GLOBAL_STEP: 9100
| > loss_text_ce: 0.05455918610095978 (0.059537755990857866)
| > loss_mel_ce: 3.7518341541290283 (4.042520471077559)
| > loss: 0.9515983462333679 (1.0255145580645422)
| > current_lr: 5e-06
| > step_time: 0.3577 (0.4107169279751169)
| > loader_time: 0.0093 (0.013967987027024608)
 --> TIME: 2024-11-16 07:00:46 -- STEP: 2838/6312 -- GLOBAL_STEP: 9150
| > loss_text_ce: 0.04679180681705475 (0.059487846667515366)
| > loss_mel_ce: 3.941854953765869 (4.041723719274933)
| > loss: 0.9971616864204407 (1.025302892780537)
| > current_lr: 5e-06
| > step_time: 0.4857 (0.41088029017660127)
| > loader_time: 0.0106 (0.014008672410455532)
 --> TIME: 2024-11-16 07:01:21 -- STEP: 2888/6312 -- GLOBAL_STEP: 9200
| > loss_text_ce: 0.06463929265737534 (0.05945338142411446)
| > loss_mel_ce: 4.276904106140137 (4.040766646657287)
| > loss: 1.085385799407959 (1.0250550082722165)
| > current_lr: 5e-06
| > step_time: 0.4477 (0.4109383028960294)
| > loader_time: 0.0091 (0.014022595086586442)
 --> TIME: 2024-11-16 07:01:54 -- STEP: 2938/6312 -- GLOBAL_STEP: 9250
| > loss_text_ce: 0.050770677626132965 (0.05945181911792211)
| > loss_mel_ce: 3.925560712814331 (4.040122042958465)
| > loss: 0.9940828680992126 (1.0248934667762852)
| > current_lr: 5e-06
| > step_time: 0.2936 (0.41086251691866277)
| > loader_time: 0.0088 (0.014012929125656636)
 --> TIME: 2024-11-16 07:02:29 -- STEP: 2988/6312 -- GLOBAL_STEP: 9300
| > loss_text_ce: 0.05795653164386749 (0.05942312789235606)
| > loss_mel_ce: 4.08872652053833 (4.038937414506345)
| > loss: 1.0366708040237427 (1.0245901368747878)
| > current_lr: 5e-06
| > step_time: 0.3984 (0.4108960007727545)
| > loader_time: 0.0235 (0.014006779535388052)
 --> TIME: 2024-11-16 07:03:02 -- STEP: 3038/6312 -- GLOBAL_STEP: 9350
| > loss_text_ce: 0.05675186589360237 (0.059402748513194256)
| > loss_mel_ce: 3.8648626804351807 (4.038147298343237)
| > loss: 0.9804036617279053 (1.0243875129863225)
| > current_lr: 5e-06
| > step_time: 0.4355 (0.4108370917654885)
| > loader_time: 0.0107 (0.013988239346090472)
 --> TIME: 2024-11-16 07:03:36 -- STEP: 3088/6312 -- GLOBAL_STEP: 9400
| > loss_text_ce: 0.05635987967252731 (0.05937609565564927)
| > loss_mel_ce: 4.344240665435791 (4.037069694057019)
| > loss: 1.1001501083374023 (1.0241114485989546)
| > current_lr: 5e-06
| > step_time: 0.2716 (0.4108069563776718)
| > loader_time: 0.0101 (0.013979903774557954)
 --> TIME: 2024-11-16 07:04:11 -- STEP: 3138/6312 -- GLOBAL_STEP: 9450
| > loss_text_ce: 0.05000467225909233 (0.05936141748876422)
| > loss_mel_ce: 4.271990776062012 (4.036430772662226)
| > loss: 1.0804988145828247 (1.0239480486943322)
| > current_lr: 5e-06
| > step_time: 0.4095 (0.410866473266621)
| > loader_time: 0.0238 (0.013976354140092644)
 --> TIME: 2024-11-16 07:04:44 -- STEP: 3188/6312 -- GLOBAL_STEP: 9500
| > loss_text_ce: 0.06751410663127899 (0.05936492357398364)
| > loss_mel_ce: 3.7811667919158936 (4.035900946019418)
| > loss: 0.9621702432632446 (1.0238164685563695)
| > current_lr: 5e-06
| > step_time: 0.2924 (0.4107274589604386)
| > loader_time: 0.0086 (0.013975165270203479)
 --> TIME: 2024-11-16 07:05:18 -- STEP: 3238/6312 -- GLOBAL_STEP: 9550
| > loss_text_ce: 0.061709415167570114 (0.059326992871227106)
| > loss_mel_ce: 4.151865005493164 (4.035102590357084)
| > loss: 1.0533936023712158 (1.0236073970021524)
| > current_lr: 5e-06
| > step_time: 0.4372 (0.41061205551755364)
| > loader_time: 0.0101 (0.01397937547284339)
 --> TIME: 2024-11-16 07:05:51 -- STEP: 3288/6312 -- GLOBAL_STEP: 9600
| > loss_text_ce: 0.057883601635694504 (0.05929976971049558)
| > loss_mel_ce: 3.9207894802093506 (4.034652139076044)
| > loss: 0.9946682453155518 (1.0234879784992987)
| > current_lr: 5e-06
| > step_time: 0.4263 (0.41054254298952664)
| > loader_time: 0.0096 (0.013957509666753801)
 --> TIME: 2024-11-16 07:06:25 -- STEP: 3338/6312 -- GLOBAL_STEP: 9650
| > loss_text_ce: 0.05580606311559677 (0.05927017712134167)
| > loss_mel_ce: 4.057142734527588 (4.03358083674119)
| > loss: 1.028237223625183 (1.0232127547406928)
| > current_lr: 5e-06
| > step_time: 0.4489 (0.41056389377244296)
| > loader_time: 0.0273 (0.013939904266663694)
 --> TIME: 2024-11-16 07:06:59 -- STEP: 3388/6312 -- GLOBAL_STEP: 9700
| > loss_text_ce: 0.05970244109630585 (0.059254505909064496)
| > loss_mel_ce: 3.7310147285461426 (4.032542098441102)
| > loss: 0.9476792812347412 (1.0229491523193188)
| > current_lr: 5e-06
| > step_time: 0.2973 (0.4104978274004237)
| > loader_time: 0.0222 (0.013918409733169898)
 --> TIME: 2024-11-16 07:07:33 -- STEP: 3438/6312 -- GLOBAL_STEP: 9750
| > loss_text_ce: 0.044777289032936096 (0.05923458883552927)
| > loss_mel_ce: 4.004416465759277 (4.031844322524682)
| > loss: 1.0122984647750854 (1.0227697290883195)
| > current_lr: 5e-06
| > step_time: 0.4509 (0.4104814838017746)
| > loader_time: 0.0092 (0.013901783949833681)
 --> TIME: 2024-11-16 07:08:07 -- STEP: 3488/6312 -- GLOBAL_STEP: 9800
| > loss_text_ce: 0.06293590366840363 (0.059240698443785496)
| > loss_mel_ce: 4.403448581695557 (4.031971572120784)
| > loss: 1.1165961027145386 (1.0228030689299905)
| > current_lr: 5e-06
| > step_time: 0.3912 (0.4104581627687183)
| > loader_time: 0.0101 (0.01388666348172984)
 --> TIME: 2024-11-16 07:08:41 -- STEP: 3538/6312 -- GLOBAL_STEP: 9850
| > loss_text_ce: 0.06208460032939911 (0.05922604597991235)
| > loss_mel_ce: 4.041192054748535 (4.031040378045449)
| > loss: 1.025819182395935 (1.0225666071514121)
| > current_lr: 5e-06
| > step_time: 0.4591 (0.410552611318667)
| > loader_time: 0.009 (0.013851198858969491)
 --> TIME: 2024-11-16 07:09:16 -- STEP: 3588/6312 -- GLOBAL_STEP: 9900
| > loss_text_ce: 0.04827294126152992 (0.05920090236147675)
| > loss_mel_ce: 3.806104898452759 (4.030434967615644)
| > loss: 0.9635944366455078 (1.0224089684959485)
| > current_lr: 5e-06
| > step_time: 0.2828 (0.4107458123661071)
| > loader_time: 0.0276 (0.013832193486799498)
 --> TIME: 2024-11-16 07:09:49 -- STEP: 3638/6312 -- GLOBAL_STEP: 9950
| > loss_text_ce: 0.05296103283762932 (0.059166911201999024)
| > loss_mel_ce: 3.9767801761627197 (4.029274758039539)
| > loss: 1.0074353218078613 (1.0221104183453904)
| > current_lr: 5e-06
| > step_time: 0.5299 (0.4106895392120638)
| > loader_time: 0.013 (0.013840565450772143)
 --> TIME: 2024-11-16 07:10:23 -- STEP: 3688/6312 -- GLOBAL_STEP: 10000
| > loss_text_ce: 0.0552692748606205 (0.05911699198428823)
| > loss_mel_ce: 3.9274134635925293 (4.0284547336447805)
| > loss: 0.9956706762313843 (1.0218929324032417)
| > current_lr: 5e-06
| > step_time: 0.3753 (0.4105937036800798)
| > loader_time: 0.0085 (0.0138198050009713)
 --> TIME: 2024-11-16 07:10:56 -- STEP: 3738/6312 -- GLOBAL_STEP: 10050
| > loss_text_ce: 0.054980434477329254 (0.059101277775962816)
| > loss_mel_ce: 3.8667824268341064 (4.028208617860431)
| > loss: 0.9804407358169556 (1.021827474850392)
| > current_lr: 5e-06
| > step_time: 0.3449 (0.4103811104199413)
| > loader_time: 0.0092 (0.013801278884299877)
 --> TIME: 2024-11-16 07:11:30 -- STEP: 3788/6312 -- GLOBAL_STEP: 10100
| > loss_text_ce: 0.058749131858348846 (0.05908652550689363)
| > loss_mel_ce: 4.2543816566467285 (4.027965969243772)
| > loss: 1.0782827138900757 (1.021763124681197)
| > current_lr: 5e-06
| > step_time: 0.2571 (0.4103838638368101)
| > loader_time: 0.0236 (0.013793404527803158)
 --> TIME: 2024-11-16 07:12:03 -- STEP: 3838/6312 -- GLOBAL_STEP: 10150
| > loss_text_ce: 0.05567106232047081 (0.059069924901951516)
| > loss_mel_ce: 4.014957427978516 (4.027781183300914)
| > loss: 1.0176571607589722 (1.0217127780264514)
| > current_lr: 5e-06
| > step_time: 0.3462 (0.41031032325452416)
| > loader_time: 0.0134 (0.013784495786554557)
 --> TIME: 2024-11-16 07:12:38 -- STEP: 3888/6312 -- GLOBAL_STEP: 10200
| > loss_text_ce: 0.05074901878833771 (0.05906555173274726)
| > loss_mel_ce: 3.779757261276245 (4.027610522300099)
| > loss: 0.9576265811920166 (1.0216690194582245)
| > current_lr: 5e-06
| > step_time: 0.2824 (0.4104211309564457)
| > loader_time: 0.0178 (0.013777440099559203)
 --> TIME: 2024-11-16 07:13:11 -- STEP: 3938/6312 -- GLOBAL_STEP: 10250
| > loss_text_ce: 0.06930238008499146 (0.05904615247965524)
| > loss_mel_ce: 3.8697803020477295 (4.026605135296222)
| > loss: 0.984770655632019 (1.0214128229840866)
| > current_lr: 5e-06
| > step_time: 0.2848 (0.4103538349475154)
| > loader_time: 0.0121 (0.013779871532794726)
 --> TIME: 2024-11-16 07:13:45 -- STEP: 3988/6312 -- GLOBAL_STEP: 10300
| > loss_text_ce: 0.05716247111558914 (0.059033954670183804)
| > loss_mel_ce: 4.264610290527344 (4.0259091350116085)
| > loss: 1.0804431438446045 (1.0212357734276745)
| > current_lr: 5e-06
| > step_time: 0.3726 (0.41027952687312275)
| > loader_time: 0.0087 (0.013767368097601826)
 --> TIME: 2024-11-16 07:14:18 -- STEP: 4038/6312 -- GLOBAL_STEP: 10350
| > loss_text_ce: 0.0495695061981678 (0.05900348912243679)
| > loss_mel_ce: 3.996046543121338 (4.025248895605435)
| > loss: 1.011404037475586 (1.021063097134977)
| > current_lr: 5e-06
| > step_time: 0.488 (0.4101907952461224)
| > loader_time: 0.009 (0.01377251544997738)
 --> TIME: 2024-11-16 07:14:52 -- STEP: 4088/6312 -- GLOBAL_STEP: 10400
| > loss_text_ce: 0.05985306575894356 (0.0589709191905688)
| > loss_mel_ce: 3.9249424934387207 (4.024317099040238)
| > loss: 0.9961988925933838 (1.0208220054521244)
| > current_lr: 5e-06
| > step_time: 0.3656 (0.41018722884342396)
| > loader_time: 0.0089 (0.013762042830368311)
 --> TIME: 2024-11-16 07:15:25 -- STEP: 4138/6312 -- GLOBAL_STEP: 10450
| > loss_text_ce: 0.06290588527917862 (0.05894735382591657)
| > loss_mel_ce: 3.970395565032959 (4.024252675419644)
| > loss: 1.0083253383636475 (1.02080000821031)
| > current_lr: 5e-06
| > step_time: 0.4595 (0.4100736356576668)
| > loader_time: 0.0117 (0.013767931010770361)
 --> TIME: 2024-11-16 07:16:00 -- STEP: 4188/6312 -- GLOBAL_STEP: 10500
| > loss_text_ce: 0.061401255428791046 (0.05893192866616084)
| > loss_mel_ce: 3.7292892932891846 (4.024019859923304)
| > loss: 0.9476726651191711 (1.0207379480686882)
| > current_lr: 5e-06
| > step_time: 0.336 (0.4101666343018569)
| > loader_time: 0.0318 (0.01375935651511381)
 --> TIME: 2024-11-16 07:16:34 -- STEP: 4238/6312 -- GLOBAL_STEP: 10550
| > loss_text_ce: 0.049485862255096436 (0.05890869445230329)
| > loss_mel_ce: 4.21941614151001 (4.02334662927325)
| > loss: 1.067225456237793 (1.0205638318508519)
| > current_lr: 5e-06
| > step_time: 0.3722 (0.410152092076743)
| > loader_time: 0.0108 (0.013762339436259683)
 --> TIME: 2024-11-16 07:17:07 -- STEP: 4288/6312 -- GLOBAL_STEP: 10600
| > loss_text_ce: 0.062349241226911545 (0.05887281193672467)
| > loss_mel_ce: 4.100584506988525 (4.022479145940531)
| > loss: 1.0407334566116333 (1.0203379904017302)
| > current_lr: 5e-06
| > step_time: 0.2965 (0.4100585917372312)
| > loader_time: 0.0255 (0.013773543787981148)
 --> TIME: 2024-11-16 07:17:41 -- STEP: 4338/6312 -- GLOBAL_STEP: 10650
| > loss_text_ce: 0.06389690935611725 (0.058854791682270174)
| > loss_mel_ce: 3.8782906532287598 (4.0220123918004544)
| > loss: 0.985546886920929 (1.0202167968614801)
| > current_lr: 5e-06
| > step_time: 0.4486 (0.4100110109855965)
| > loader_time: 0.0087 (0.013780175512866972)
 --> TIME: 2024-11-16 07:18:14 -- STEP: 4388/6312 -- GLOBAL_STEP: 10700
| > loss_text_ce: 0.05854211747646332 (0.0588163215999806)
| > loss_mel_ce: 3.7983341217041016 (4.0208518790894185)
| > loss: 0.9642190337181091 (1.0199170511015567)
| > current_lr: 5e-06
| > step_time: 0.3455 (0.4098104210256335)
| > loader_time: 0.0092 (0.013803951068693877)
 --> TIME: 2024-11-16 07:18:47 -- STEP: 4438/6312 -- GLOBAL_STEP: 10750
| > loss_text_ce: 0.05718289688229561 (0.05880145871035545)
| > loss_mel_ce: 4.210494041442871 (4.020278105922704)
| > loss: 1.066919207572937 (1.019769892103235)
| > current_lr: 5e-06
| > step_time: 0.4229 (0.4096843574217926)
| > loader_time: 0.0241 (0.01382307186487722)
 --> TIME: 2024-11-16 07:19:23 -- STEP: 4488/6312 -- GLOBAL_STEP: 10800
| > loss_text_ce: 0.059240277856588364 (0.058787808217333974)
| > loss_mel_ce: 4.180911540985107 (4.019722221794373)
| > loss: 1.0600379705429077 (1.019627508473269)
| > current_lr: 5e-06
| > step_time: 0.3681 (0.4100123577470663)
| > loader_time: 0.0093 (0.013816027770917902)
 --> TIME: 2024-11-16 07:19:57 -- STEP: 4538/6312 -- GLOBAL_STEP: 10850
| > loss_text_ce: 0.06382870674133301 (0.058776855228223276)
| > loss_mel_ce: 3.9748690128326416 (4.0187539728058495)
| > loss: 1.0096744298934937 (1.0193827079749418)
| > current_lr: 5e-06
| > step_time: 0.3563 (0.4101197533063189)
| > loader_time: 0.0247 (0.013797360298758622)
 --> TIME: 2024-11-16 07:20:31 -- STEP: 4588/6312 -- GLOBAL_STEP: 10900
| > loss_text_ce: 0.05853895843029022 (0.05875471142407278)
| > loss_mel_ce: 4.1447577476501465 (4.018682285350795)
| > loss: 1.0508241653442383 (1.0193592501575266)
| > current_lr: 5e-06
| > step_time: 0.3836 (0.41012373340576774)
| > loader_time: 0.0235 (0.01381379516621518)
 --> TIME: 2024-11-16 07:21:05 -- STEP: 4638/6312 -- GLOBAL_STEP: 10950
| > loss_text_ce: 0.05851801112294197 (0.05873311788595505)
| > loss_mel_ce: 3.9833264350891113 (4.018390352430885)
| > loss: 1.0104610919952393 (1.019280868570781)
| > current_lr: 5e-06
| > step_time: 0.4745 (0.41006452444042824)
| > loader_time: 0.0099 (0.013809717796444134)
 --> TIME: 2024-11-16 07:21:39 -- STEP: 4688/6312 -- GLOBAL_STEP: 11000
| > loss_text_ce: 0.06378674507141113 (0.058707356405263214)
| > loss_mel_ce: 3.833876132965088 (4.018039324838929)
| > loss: 0.9744157195091248 (1.0191866713401196)
| > current_lr: 5e-06
| > step_time: 0.2874 (0.41008276393185716)
| > loader_time: 0.0092 (0.013792215150370952)
 --> TIME: 2024-11-16 07:22:13 -- STEP: 4738/6312 -- GLOBAL_STEP: 11050
| > loss_text_ce: 0.059916820377111435 (0.0586723556852934)
| > loss_mel_ce: 4.096810340881348 (4.017760860512324)
| > loss: 1.0391818284988403 (1.019108305139952)
| > current_lr: 5e-06
| > step_time: 0.4586 (0.4100804382258198)
| > loader_time: 0.0087 (0.013798208717758428)
 --> TIME: 2024-11-16 07:22:47 -- STEP: 4788/6312 -- GLOBAL_STEP: 11100
| > loss_text_ce: 0.05626944825053215 (0.058646952759851045)
| > loss_mel_ce: 3.872283458709717 (4.017298521603124)
| > loss: 0.9821382164955139 (1.018986369658233)
| > current_lr: 5e-06
| > step_time: 0.3899 (0.4101231080746787)
| > loader_time: 0.0087 (0.013798897228543732)
 --> TIME: 2024-11-16 07:23:22 -- STEP: 4838/6312 -- GLOBAL_STEP: 11150
| > loss_text_ce: 0.05398077517747879 (0.05861273269026962)
| > loss_mel_ce: 4.250906467437744 (4.017032340618601)
| > loss: 1.0762218236923218 (1.0189112692935847)
| > current_lr: 5e-06
| > step_time: 0.4521 (0.41027893952073596)
| > loader_time: 0.009 (0.013800443721832754)
 --> TIME: 2024-11-16 07:23:56 -- STEP: 4888/6312 -- GLOBAL_STEP: 11200
| > loss_text_ce: 0.05338648706674576 (0.05858512338763022)
| > loss_mel_ce: 3.5620734691619873 (4.016477789504211)
| > loss: 0.9038649797439575 (1.0187657291462886)
| > current_lr: 5e-06
| > step_time: 0.2865 (0.4102531838339008)
| > loader_time: 0.011 (0.013790628426203762)
 --> TIME: 2024-11-16 07:24:30 -- STEP: 4938/6312 -- GLOBAL_STEP: 11250
| > loss_text_ce: 0.056189488619565964 (0.05855837866887234)
| > loss_mel_ce: 3.66269850730896 (4.015786350183772)
| > loss: 0.929722011089325 (1.0185861831109204)
| > current_lr: 5e-06
| > step_time: 0.3639 (0.41030535049986944)
| > loader_time: 0.0214 (0.013776267429458425)
 --> TIME: 2024-11-16 07:25:04 -- STEP: 4988/6312 -- GLOBAL_STEP: 11300
| > loss_text_ce: 0.052941568195819855 (0.05854161461455251)
| > loss_mel_ce: 4.239297866821289 (4.014998362364501)
| > loss: 1.0730599164962769 (1.0183849951675046)
| > current_lr: 5e-06
| > step_time: 0.2834 (0.41027581696521775)
| > loader_time: 0.0293 (0.01379498846548123)
 --> TIME: 2024-11-16 07:25:40 -- STEP: 5038/6312 -- GLOBAL_STEP: 11350
| > loss_text_ce: 0.05275028198957443 (0.058525998095060025)
| > loss_mel_ce: 3.8316643238067627 (4.0147574119692475)
| > loss: 0.9711036682128906 (1.018320853474026)
| > current_lr: 5e-06
| > step_time: 0.4482 (0.41055660694541796)
| > loader_time: 0.027 (0.01378794135721775)
 --> TIME: 2024-11-16 07:26:13 -- STEP: 5088/6312 -- GLOBAL_STEP: 11400
| > loss_text_ce: 0.0579095296561718 (0.05850495614035273)
| > loss_mel_ce: 3.976348876953125 (4.014259529094993)
| > loss: 1.0085645914077759 (1.0181911222800681)
| > current_lr: 5e-06
| > step_time: 0.2942 (0.4104244215589649)
| > loader_time: 0.0089 (0.013806618094069413)
 --> TIME: 2024-11-16 07:26:48 -- STEP: 5138/6312 -- GLOBAL_STEP: 11450
| > loss_text_ce: 0.05707567185163498 (0.05846731477979321)
| > loss_mel_ce: 4.1147050857543945 (4.013865671903963)
| > loss: 1.042945146560669 (1.0180832475783417)
| > current_lr: 5e-06
| > step_time: 0.4879 (0.41057518981875685)
| > loader_time: 0.0089 (0.013805919604897278)
 --> TIME: 2024-11-16 07:27:22 -- STEP: 5188/6312 -- GLOBAL_STEP: 11500
| > loss_text_ce: 0.05074966326355934 (0.05843212032430702)
| > loss_mel_ce: 4.154684543609619 (4.013384800510549)
| > loss: 1.0513585805892944 (1.0179542311242464)
| > current_lr: 5e-06
| > step_time: 0.4874 (0.4106214234896964)
| > loader_time: 0.0092 (0.013804514130538681)
 --> TIME: 2024-11-16 07:27:56 -- STEP: 5238/6312 -- GLOBAL_STEP: 11550
| > loss_text_ce: 0.050595611333847046 (0.05840772336314038)
| > loss_mel_ce: 4.2425360679626465 (4.01256853221433)
| > loss: 1.0732829570770264 (1.0177440648203628)
| > current_lr: 5e-06
| > step_time: 0.3529 (0.4106079214353878)
| > loader_time: 0.0288 (0.013802284227613742)
 --> TIME: 2024-11-16 07:28:31 -- STEP: 5288/6312 -- GLOBAL_STEP: 11600
| > loss_text_ce: 0.048710424453020096 (0.05839222400701416)
| > loss_mel_ce: 3.939770221710205 (4.012302499450432)
| > loss: 0.9971201419830322 (1.0176736817845953)
| > current_lr: 5e-06
| > step_time: 0.4108 (0.4106582271529759)
| > loader_time: 0.0098 (0.013803137548031147)
 --> TIME: 2024-11-16 07:29:04 -- STEP: 5338/6312 -- GLOBAL_STEP: 11650
| > loss_text_ce: 0.06049573794007301 (0.05837566024561346)
| > loss_mel_ce: 4.003974437713623 (4.011894629081091)
| > loss: 1.0161175727844238 (1.0175675732694602)
| > current_lr: 5e-06
| > step_time: 0.4688 (0.41065752323132054)
| > loader_time: 0.0094 (0.01379742453544049)
 --> TIME: 2024-11-16 07:29:39 -- STEP: 5388/6312 -- GLOBAL_STEP: 11700
| > loss_text_ce: 0.05829909071326256 (0.05836241467183129)
| > loss_mel_ce: 4.107619285583496 (4.011752370382623)
| > loss: 1.0414795875549316 (1.0175286971513835)
| > current_lr: 5e-06
| > step_time: 0.2968 (0.41076431879752995)
| > loader_time: 0.0092 (0.013791898488113425)
 --> TIME: 2024-11-16 07:30:14 -- STEP: 5438/6312 -- GLOBAL_STEP: 11750
| > loss_text_ce: 0.058009129017591476 (0.05835293694089309)
| > loss_mel_ce: 4.078938007354736 (4.01127332841352)
| > loss: 1.0342367887496948 (1.0174065671803618)
| > current_lr: 5e-06
| > step_time: 0.3104 (0.41086738172722637)
| > loader_time: 0.0157 (0.01378048768417062)
 --> TIME: 2024-11-16 07:30:48 -- STEP: 5488/6312 -- GLOBAL_STEP: 11800
| > loss_text_ce: 0.05709939822554588 (0.0583388931280865)
| > loss_mel_ce: 4.154338836669922 (4.010130630246726)
| > loss: 1.0528595447540283 (1.0171173816606514)
| > current_lr: 5e-06
| > step_time: 0.4024 (0.4108732487821718)
| > loader_time: 0.0103 (0.013785614824851133)
 --> TIME: 2024-11-16 07:31:21 -- STEP: 5538/6312 -- GLOBAL_STEP: 11850
| > loss_text_ce: 0.058982137590646744 (0.05831718708984832)
| > loss_mel_ce: 3.8713808059692383 (4.009040404631773)
| > loss: 0.9825907349586487 (1.0168393987014663)
| > current_lr: 5e-06
| > step_time: 0.36 (0.4108115788098491)
| > loader_time: 0.0089 (0.013797443280714273)
 --> TIME: 2024-11-16 07:31:55 -- STEP: 5588/6312 -- GLOBAL_STEP: 11900
| > loss_text_ce: 0.0507952906191349 (0.058306316222459485)
| > loss_mel_ce: 3.8884329795837402 (4.0084597597227765)
| > loss: 0.9848070740699768 (1.016691519786137)
| > current_lr: 5e-06
| > step_time: 0.3783 (0.41069966179179396)
| > loader_time: 0.0223 (0.013805615492351764)
 --> TIME: 2024-11-16 07:32:28 -- STEP: 5638/6312 -- GLOBAL_STEP: 11950
| > loss_text_ce: 0.06038466840982437 (0.05829100494027536)
| > loss_mel_ce: 4.027162551879883 (4.007462979885375)
| > loss: 1.0218868255615234 (1.0164384970050941)
| > current_lr: 5e-06
| > step_time: 0.4234 (0.4106813036156445)
| > loader_time: 0.0093 (0.013806500516068061)
 --> TIME: 2024-11-16 07:33:02 -- STEP: 5688/6312 -- GLOBAL_STEP: 12000
| > loss_text_ce: 0.056697066873311996 (0.05826091347458741)
| > loss_mel_ce: 3.849360704421997 (4.006760919437777)
| > loss: 0.976514458656311 (1.016255459044802)
| > current_lr: 5e-06
| > step_time: 0.2807 (0.4105964454333658)
| > loader_time: 0.0323 (0.013814455891124988)
 --> TIME: 2024-11-16 07:33:35 -- STEP: 5738/6312 -- GLOBAL_STEP: 12050
| > loss_text_ce: 0.05204227939248085 (0.05825942866867758)
| > loss_mel_ce: 3.9579384326934814 (4.006122983802503)
| > loss: 1.0024951696395874 (1.0160956039333948)
| > current_lr: 5e-06
| > step_time: 0.3486 (0.410525723923038)
| > loader_time: 0.0097 (0.013810704252203827)
 --> TIME: 2024-11-16 07:34:10 -- STEP: 5788/6312 -- GLOBAL_STEP: 12100
| > loss_text_ce: 0.049697499722242355 (0.05823658913716351)
| > loss_mel_ce: 4.2517924308776855 (4.005454409509654)
| > loss: 1.0753724575042725 (1.0159227504203763)
| > current_lr: 5e-06
| > step_time: 0.4845 (0.4106328804161768)
| > loader_time: 0.0094 (0.01380907175206447)
 --> TIME: 2024-11-16 07:34:44 -- STEP: 5838/6312 -- GLOBAL_STEP: 12150
| > loss_text_ce: 0.0536055788397789 (0.058208476963895486)
| > loss_mel_ce: 3.994285821914673 (4.004660323124365)
| > loss: 1.0119729042053223 (1.015717200799604)
| > current_lr: 5e-06
| > step_time: 0.5676 (0.4105943763202982)
| > loader_time: 0.0092 (0.013808757321970977)
 --> TIME: 2024-11-16 07:35:18 -- STEP: 5888/6312 -- GLOBAL_STEP: 12200
| > loss_text_ce: 0.04857238754630089 (0.05818636288914481)
| > loss_mel_ce: 4.068244457244873 (4.004073279018942)
| > loss: 1.029204249382019 (1.015564911300313)
| > current_lr: 5e-06
| > step_time: 0.3653 (0.41054160933455697)
| > loader_time: 0.0088 (0.01382541458081941)
 --> TIME: 2024-11-16 07:35:51 -- STEP: 5938/6312 -- GLOBAL_STEP: 12250
| > loss_text_ce: 0.053571976721286774 (0.05816797556154208)
| > loss_mel_ce: 3.998563051223755 (4.0032839372768985)
| > loss: 1.0130337476730347 (1.015362978985347)
| > current_lr: 5e-06
| > step_time: 0.4496 (0.4105043701548976)
| > loader_time: 0.0089 (0.013824497701825427)
 --> TIME: 2024-11-16 07:36:24 -- STEP: 5988/6312 -- GLOBAL_STEP: 12300
| > loss_text_ce: 0.05345112830400467 (0.05814874009151473)
| > loss_mel_ce: 4.085923194885254 (4.002494778008785)
| > loss: 1.0348435640335083 (1.0151608802512522)
| > current_lr: 5e-06
| > step_time: 0.4065 (0.4103937765400491)
| > loader_time: 0.0283 (0.01382815869712958)
 --> TIME: 2024-11-16 07:36:59 -- STEP: 6038/6312 -- GLOBAL_STEP: 12350
| > loss_text_ce: 0.05839666351675987 (0.05811350147784957)
| > loss_mel_ce: 3.793591022491455 (4.001863627411646)
| > loss: 0.9629969000816345 (1.0149942829380647)
| > current_lr: 5e-06
| > step_time: 0.2933 (0.4104814607207054)
| > loader_time: 0.0281 (0.013828487023489103)
 --> TIME: 2024-11-16 07:37:33 -- STEP: 6088/6312 -- GLOBAL_STEP: 12400
| > loss_text_ce: 0.05447766184806824 (0.0581067755124464)
| > loss_mel_ce: 3.791832685470581 (4.001433893352841)
| > loss: 0.9615775942802429 (1.0148851678715234)
| > current_lr: 5e-06
| > step_time: 0.2601 (0.41048672611234066)
| > loader_time: 0.0255 (0.013816627190398176)
 --> TIME: 2024-11-16 07:38:07 -- STEP: 6138/6312 -- GLOBAL_STEP: 12450
| > loss_text_ce: 0.053448982536792755 (0.058088236062645716)
| > loss_mel_ce: 4.274224758148193 (4.0008808252826)
| > loss: 1.081918478012085 (1.0147422659898164)
| > current_lr: 5e-06
| > step_time: 0.3617 (0.41047337708546233)
| > loader_time: 0.009 (0.013812006471521431)
 --> TIME: 2024-11-16 07:38:40 -- STEP: 6188/6312 -- GLOBAL_STEP: 12500
| > loss_text_ce: 0.062232211232185364 (0.058065060349139115)
| > loss_mel_ce: 3.999107599258423 (4.000377380508853)
| > loss: 1.015334963798523 (1.0146106108878574)
| > current_lr: 5e-06
| > step_time: 0.2753 (0.4104526710648959)
| > loader_time: 0.0141 (0.013798853814717495)
> CHECKPOINT : checkpoints/GPT_XTTS_FT-November-16-2024_05+12AM-4b65e59/checkpoint_12500.pth
 --> TIME: 2024-11-16 07:40:00 -- STEP: 6238/6312 -- GLOBAL_STEP: 12550
| > loss_text_ce: 0.04739612340927124 (0.05804410937883293)
| > loss_mel_ce: 3.715331792831421 (4.000103949430295)
| > loss: 0.9406819939613342 (1.0145370154168263)
| > current_lr: 5e-06
| > step_time: 0.328 (0.4104116754907949)
| > loader_time: 0.0088 (0.013807578016527582)
 --> TIME: 2024-11-16 07:40:34 -- STEP: 6288/6312 -- GLOBAL_STEP: 12600
| > loss_text_ce: 0.05123625695705414 (0.05802164425061533)
| > loss_mel_ce: 4.075647830963135 (3.999731126442793)
| > loss: 1.0317209959030151 (1.014438193424131)
| > current_lr: 5e-06
| > step_time: 0.3741 (0.4104879065096833)
| > loader_time: 0.0105 (0.013805110832207085)
 > EVALUATION 
--> EVAL PERFORMANCE
| > avg_loader_time: 0.5487493482129325 (+0.07595368089347032)
| > avg_loss_text_ce: 0.05374100585949832 (-0.005282426873842873)
| > avg_loss_mel_ce: 3.8636940073692934 (-0.13614979557607443)
| > avg_loss: 3.9174350239764686 (-0.14143220583597715)
> BEST MODEL : checkpoints/GPT_XTTS_FT-November-16-2024_05+12AM-4b65e59/best_model_12624.pth