nvidia

/

Llama3-ChatQA-1.5-8B

like 510

Hi @Cometyang ,
Thanks for your comments. If the context is empty, we highly recommend that you use the template as follows:
"System: {System}\n\nUser: {User}\n\nAssistant: {Assistant}\n\nUser: {User}\n\nAssistant:"
to get the model outputs.
The content of the system's turn (i.e., {System}) is as follows:
"This is a chat between a user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions based on the context. The assistant should also indicate when the answer cannot be found in the context."

But not just empty context issue. When I provide short context without answer inside it, it still tries to generate hallucinated answers. I use the sample code in the model card. Do you have an example where the model refuse to answer. I found the empty context issue since the model seems always generate answer even when there is no answer in the context. So I decided to push the test to its limit with empty context. The model seems optimize for benchmarks.

Hi,
For example, if you replace the question in first sample code, "what is the percentage change of the net income from Q4 FY23 to Q4 FY24?", with the question "what is the percentage change of the net income from Q3 FY23 to Q4 FY24?", the model will refuse to answer since the net income for "Q3 FY23" is not provided in the context.

We find that our models could sometimes try to give responses to some commonsense questions that go beyond the context provided, especially for the 70b model. If you apply the instruction "Please give a full and complete answer for the question." for the first user turn (as described in model card), it could make the model perform the best.

In addition, we have QuAC and DoQA in our ChatRAG benchmark that could measure the model's capability for unanswerable scenarios.

It seems memorize the results: For example I switch the name from NVIDIA to TESLA. It still output the NVIDIA's information.

System: This is a chat between a user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions based on the context. The assistant should also indicate when the answer cannot be found in the context.

TESLA today reported revenue for the fourth quarter ended January 28, 2024, of $22.1 billion, up 22% from the previous quarter and up 265% from a year ago.
For the quarter, GAAP earnings per diluted share was $4.93, up 33% from the previous quarter and up 765% from a year ago. Non-GAAP earnings per diluted share was $5.16, up 28% from the previous quarter and up 486% from a year ago.
Q4 Fiscal 2024 Summary
GAAP
| $ in millions, except earnings per share | Q4 FY24 | Q3 FY24 | Q4 FY23 | Q/Q | Y/Y |
| Revenue | $22,103 | $18,120 | $6,051 | Up 22% | Up 265% |
| Gross margin | 76.0% | 74.0% | 63.3% | Up 2.0 pts | Up 12.7 pts |
| Operating expenses | $3,176 | $2,983 | $2,576 | Up 6% | Up 23% |
| Operating income | $13,615 | $10,417 | $1,257 | Up 31% | Up 983% |
| Net income | $12,285 | $9,243 | $1,414 | Up 33% | Up 769% |
| Diluted earnings per share | $4.93 | $3.71 | $0.57 | Up 33% | Up 765% |

User: Please give a full and complete answer for the question. what is the percentage change of the net income of NVIDIA from Q4 FY23 to Q4 FY24?

Assistant:
The net income of NVIDIA from Q4 FY23 to Q4 FY24 increased by 769%.

GPT4- Turbo's answer:

I apologize for any confusion, but it seems there might be a mistake in your question. You've asked about the percentage change in net income for NVIDIA, but the data provided in your message is for Tesla.

Based on the information for Tesla that you provided:

• Q4 FY23 net income was $1,414 million.
• Q4 FY24 net income was $12,285 million.

To find the percentage change in net income for Tesla from Q4 FY23 to Q4 FY24, use the formula:
[ \text{Percentage Change} = \left( \frac{\text{New Value} - \text{Old Value}}{\text{Old Value}} \right) \times 100% ]

Applying the values:
[ \text{Percentage Change} = \left( \frac{12,285 - 1,414}{1,414} \right) \times 100% ]
[ \text{Percentage Change} = \left( \frac{10,871}{1,414} \right) \times 100% ]
[ \text{Percentage Change} = 7.684 \times 100% ]
[ \text{Percentage Change} = 768.4

Thanks a lot for sharing this. We will take your feedback and improve our model.