You can use a wattage meter between your outlet and computer. I’ve tried that, and the usage is around the same as a graphically intensive videogame while it is generating.
Sure, but without actually knowing what kind of hardware the servers are running, what kind of software too, and what their service backend looks like we can’t say whether it is going to be higher or lower.
I think we can assume it’s nvidia H200 which peaks at 700W what what I saw on Google. Multiply that by the turnaround time from your prompt to full response and you have a ceiling value.
There’s probably some queueing and other delays so in reality the time GPU spends on your query will be much less. If you use the API, it may include the timing information.
You can use a wattage meter between your outlet and computer. I’ve tried that, and the usage is around the same as a graphically intensive videogame while it is generating.
How does a wattage meter on my computer measure power used by an LLM server someplace else per prompt?
I have a LLM server on my computer, so I can tell how much electricity it is using this way. It’s not somewhere else is how
Cool but that is not the question I asked is it then mate.
You asked if there’s a way to tell how much power LLMs use, you didn’t specify LLMs on a server you don’t have physical access to.
It says LLMs meaning in general so you can take it any way you like.
There’s a huge difference between a model you can run locally and a chatgpt model.
Sure, but without actually knowing what kind of hardware the servers are running, what kind of software too, and what their service backend looks like we can’t say whether it is going to be higher or lower.
I think we can assume it’s nvidia H200 which peaks at 700W what what I saw on Google. Multiply that by the turnaround time from your prompt to full response and you have a ceiling value. There’s probably some queueing and other delays so in reality the time GPU spends on your query will be much less. If you use the API, it may include the timing information.