“The requested GPU duration (240s) is larger than the maximum allowed retry in -1 day, 23:59:59”?

#118
by John6666 - opened

I have heard two reports on the forum of apparently buggy error messages.
It seems to have occurred in the space below and another Zero GPU space I don't know where.
Note that this problem is not 100% reproducible.
It may be related to the fact that Spaces requires 240 seconds for duration.

Forum

https://discuss.huggingface.co/t/the-requested-gpu-duration-240s-is-larger-than-the-maximum-allowed-retry-in-1-day-2359/106988

Space

https://huggingface.co/spaces/KwaiVGI/LivePortrait

Screenshot

SS1
SS2

ZeroGPU Explorers org

I've had such problems with the CogVideo-5b spaces and other spaces setting the duration parameter beyond 140 seconds, sadly there isn't much to do except pray HF implements a duration increase.

When I tried that, I got an error saying that I requested 240 seconds but only 173 seconds remained, and -1 day did not show up.
I thought 120 seconds was the max, but I guess that's not really true.
Maybe this is a problem that sometimes occurs with any space that requests a duration longer than 120 seconds...?

I'm even more confused by the fact that there are cases where it doesn't occur as it did for me. Well, bugs are like that.

ZeroGPU Explorers org

Maybe this is a problem that sometimes occurs with any space that requests a duration longer than 120 seconds...?

Yeah it seems like it is

I'm even more confused by the fact that there are cases where it doesn't occur as it did for me. Well, bugs are like that.

Yeah I get the same stuff every now and then, hopefully someone at HF can look into this and hopefully get the duration parameter increased if possible.

I saw the code for the spaces library on KingNish's Space the other day.
Well, it's Python to begin with, so I could see all the client-side code if I wanted to...

Remembering that, I think that when you query the server side for Quota, the response you get back is indefinite if the number is over 120. Or maybe the function for calculating Quota punishment is exponential, for example, and it's causing an overflow, or something like that.
Anyway, it's a server-side processing problem.

We'd like to see support for 240 seconds as well, since 120 seconds is too tight for video-related inferences. Even if the price is a penalty, like consuming two Zero GPU space slots, it's better than no option itself.
Also, although we can support it on our own, it would be useful to have a decorator that allows dynamic specification of duration, and a decorator that prohibits multi-process/multi-threading in wrapping functions.
There are several libraries that are no longer working in any way with the current Zero GPU space implementation.
Even more so for generative AI-related stuff, since the dependencies tend to get too complex...

Sign up or log in to comment